From brett at python.org  Sat Jun  1 00:37:13 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 31 May 2013 18:37:13 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Add a reset_name
 argument to importlib.util.module_to_load in order to
In-Reply-To: <3bMfy630Xkz7LpQ@mail.python.org>
References: <3bMfy630Xkz7LpQ@mail.python.org>
Message-ID: <CAP1=2W7saV5G3RNyW4pBQcGOqZFySF=_J2oLcxgUwZnMyG48gw@mail.gmail.com>

I realize this broke the buildbots. Missed part of a diff in the commit.
I'm trying to split a massive CL into reasonable commit sizes, so please be
patient.


On Fri, May 31, 2013 at 6:11 PM, brett.cannon <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/39cc1b04713e
> changeset:   83998:39cc1b04713e
> user:        Brett Cannon <brett at python.org>
> date:        Fri May 31 18:11:17 2013 -0400
> summary:
>   Add a reset_name argument to importlib.util.module_to_load in order to
> control whether to reset the module's __name__ attribute in case a
> reload is being done.
>
> files:
>   Doc/library/importlib.rst            |   6 +++++-
>   Lib/importlib/_bootstrap.py          |  14 +++++++++++++-
>   Lib/test/test_importlib/test_util.py |  12 ++++++++++++
>   3 files changed, 30 insertions(+), 2 deletions(-)
>
>
> diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
> --- a/Doc/library/importlib.rst
> +++ b/Doc/library/importlib.rst
> @@ -788,7 +788,7 @@
>
>     .. versionadded:: 3.3
>
> -.. function:: module_to_load(name)
> +.. function:: module_to_load(name, *, reset_name=True)
>
>      Returns a :term:`context manager` which provides the module to load.
> The
>      module will either come from :attr:`sys.modules` in the case of
> reloading or
> @@ -796,6 +796,10 @@
>      :attr:`sys.modules` occurs if the module was new and an exception was
>      raised.
>
> +    If **reset_name** is true and the module requested is being reloaded
> then
> +    the module's :attr:`__name__` attribute will
> +    be reset to **name**, else it will be left untouched.
> +
>      .. versionadded:: 3.4
>
>  .. decorator:: module_for_loader
> diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
> --- a/Lib/importlib/_bootstrap.py
> +++ b/Lib/importlib/_bootstrap.py
> @@ -493,8 +493,14 @@
>
>      """
>
> -    def __init__(self, name):
> +    def __init__(self, name, *, reset_name=True):
> +        """Prepare the context manager.
> +
> +        The reset_name argument specifies whether to unconditionally reset
> +        the __name__ attribute if the module is found to be a reload.
> +        """
>          self._name = name
> +        self._reset_name = reset_name
>
>      def __enter__(self):
>          self._module = sys.modules.get(self._name)
> @@ -508,6 +514,12 @@
>              # (otherwise an optimization shortcut in import.c becomes
> wrong)
>              self._module.__initializing__ = True
>              sys.modules[self._name] = self._module
> +        elif self._reset_name:
> +            try:
> +                self._module.__name__ = self._name
> +            except AttributeError:
> +                pass
> +
>          return self._module
>
>      def __exit__(self, *args):
> diff --git a/Lib/test/test_importlib/test_util.py
> b/Lib/test/test_importlib/test_util.py
> --- a/Lib/test/test_importlib/test_util.py
> +++ b/Lib/test/test_importlib/test_util.py
> @@ -55,6 +55,18 @@
>          else:
>              self.fail('importlib.util.module_to_load swallowed an
> exception')
>
> +    def test_reset_name(self):
> +        # If reset_name is true then module.__name__ = name, else leave
> it be.
> +        odd_name = 'not your typical name'
> +        created_module = imp.new_module(self.module_name)
> +        created_module.__name__ = odd_name
> +        sys.modules[self.module_name] = created_module
> +        with util.module_to_load(self.module_name) as module:
> +            self.assertEqual(module.__name__, self.module_name)
> +        created_module.__name__ = odd_name
> +        with util.module_to_load(self.module_name, reset_name=False) as
> module:
> +            self.assertEqual(module.__name__, odd_name)
> +
>
>  class ModuleForLoaderTests(unittest.TestCase):
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130531/c1bf41a0/attachment.html>

From martin at v.loewis.de  Sun Jun  2 12:37:32 2013
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Sun, 02 Jun 2013 12:37:32 +0200
Subject: [Python-Dev] Problem building Python 2.7.5 with separate sysroot
In-Reply-To: <1370000129.4119.53.camel@homebase>
References: <1369986770.4119.43.camel@homebase>
	<nad-05E5FB.01214031052013@news.gmane.org>
	<1370000129.4119.53.camel@homebase>
Message-ID: <20130602123732.Horde.ShEJBOEliraK4rCETOXT9w9@webmail.df.eu>


Quoting Paul Smith <paul at mad-scientist.net>:

> My case may be unusual but even in a
> more formal cross-compilation environment it's not good to
> add /usr/include/..., or base such a decision on the behavior of the
> _build_ system.

The offical procedure to cover "unusual" cases is to edit Modules/Setup.
If you are not happy with the way in which modules are build, you can
override all flags on a per-module basis there.

> It seems to me (keeping with the theme of this mailing list) that the
> add_multiarch_paths() function in setup.py is not right.

Well, ..., yes. For the last two decades, the build process of Python
was *always* wrong, and it always ever will be, in the sense that it
doesn't support all cases that people come up with. The only way to
deal with this, unfortunately, is to patch the build process for each
new use case discovered.

As Ned says: if you come up with a patch, don't hesitate to post it
to the tracker.

Regards,
Martin



From koobs.freebsd at gmail.com  Sun Jun  2 07:12:43 2013
From: koobs.freebsd at gmail.com (Kubilay Kocak)
Date: Sun, 02 Jun 2013 15:12:43 +1000
Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
Message-ID: <51AAD44B.4090802@FreeBSD.org>

Afternoon (UTC+10),

I'd like to request a new user/pass for a FreeBSD 10.0-CURRENT VM guest
I've just setup as a dedicated buildbot slave to complement my other
koobs-freebsd slaves.

Also, with this and future additions to the FreeBSD buildslave set in
mind, I think it might also be prudent for a rename to take place:

koobs-freebsd9-amd64
koobs-freebsd9-amd64-clang (CC=clang)
koobs-freebsd10-amd64 (clang is default here)

Convention being: koobs-freebsdX[Y]-arch[-config] (Happy for feedback here)

If there are any permutations or additions you'd like to specifically
see for FreeBSD to increase coverage just let me know (Hint: I have a
PandaBoard arm board here i'm getting up and running)

I have ZFS & DTrace to work with among other things, and the long term
plan is to have FreeBSD buildbots running the full gamut of versions,
from -RELEASE through -STABLE to HEAD branches.

I'm on #python-dev IRC if someone needs to discuss.

--
Regards,

Koobs

From carlosnepomuceno at outlook.com  Sun Jun  2 21:51:24 2013
From: carlosnepomuceno at outlook.com (Carlos Nepomuceno)
Date: Sun, 2 Jun 2013 22:51:24 +0300
Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
In-Reply-To: <51AAD44B.4090802@FreeBSD.org>
References: <51AAD44B.4090802@FreeBSD.org>
Message-ID: <BLU176-W474C7D3F83824AD5E5F015D79C0@phx.gbl>

----------------------------------------
> Date: Sun, 2 Jun 2013 15:12:43 +1000
> From: koobs.freebsd at gmail.com
> To: python-dev at python.org
> Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
[...]
> koobs-freebsd10-amd64 (clang is default here)


Does CPython code compiled with clang runs faster than gcc?

Why did you chose clang? Any benchmarks? Any benefits? 		 	   		  

From drsalists at gmail.com  Sun Jun  2 22:43:24 2013
From: drsalists at gmail.com (Dan Stromberg)
Date: Sun, 2 Jun 2013 13:43:24 -0700
Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
In-Reply-To: <BLU176-W474C7D3F83824AD5E5F015D79C0@phx.gbl>
References: <51AAD44B.4090802@FreeBSD.org>
	<BLU176-W474C7D3F83824AD5E5F015D79C0@phx.gbl>
Message-ID: <CAGGBd_o9+7MdUveT3AdQqQsUcL167GK_P886fdEiHqVjEhFBvg@mail.gmail.com>

On Sun, Jun 2, 2013 at 12:51 PM, Carlos Nepomuceno <
carlosnepomuceno at outlook.com> wrote:

> ----------------------------------------
> > Date: Sun, 2 Jun 2013 15:12:43 +1000
> > From: koobs.freebsd at gmail.com
> > To: python-dev at python.org
> > Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
> [...]
> > koobs-freebsd10-amd64 (clang is default here)
>
>
> Does CPython code compiled with clang runs faster than gcc?
>
> Why did you chose clang? Any benchmarks? Any benefits?
>

Clang is sometimes favored over gcc for its non-GPL license; I believe the
FreeBSD project sees this as an important issue.

In general, the more C compilers a piece of C code compiles on, the fewer
bugs are left in it, because different C compilers tend to catch different
problems - even with cranked up warnings.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130602/0104c45d/attachment.html>

From benjamin at python.org  Mon Jun  3 04:16:17 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 2 Jun 2013 19:16:17 -0700
Subject: [Python-Dev] PEP 442: Safe object finalization
In-Reply-To: <20130518163315.6a21a0cd@fsol>
References: <20130518105910.1eecfa5f@fsol>
	<CAMSv6X39grvMWsAEkXBUdwHdYr-5y6Mwt_MdFdN3KjCs-EC3_w@mail.gmail.com>
	<20130518154552.2a879bf6@fsol>
	<CAMSv6X1J9JaKsLJBhMU+nZOa+M0eV8ZQawWe9iEE_eL0bhmMVw@mail.gmail.com>
	<20130518163315.6a21a0cd@fsol>
Message-ID: <CAPZV6o_-E-8O4cQGyDK5kmW9ZY+BiZu6kuXrp=2LF0GydmSEkA@mail.gmail.com>

2013/5/18 Antoine Pitrou <solipsis at pitrou.net>:
> Calling finalizers only once is fine with me, but it would be a change
> in behaviour; I don't know if it may break existing code.

I agree with Armin that this is better behavior. (Mostly significantly
consistent with weakrefs.)

>
> (for example, say someone is using __del__ to manage a freelist)

Do you know if it breaks any of the projects you tested it with?


--
Regards,
Benjamin

From benjamin at python.org  Mon Jun  3 04:27:49 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 2 Jun 2013 19:27:49 -0700
Subject: [Python-Dev] PEP 442: Safe object finalization
In-Reply-To: <20130518105910.1eecfa5f@fsol>
References: <20130518105910.1eecfa5f@fsol>
Message-ID: <CAPZV6o-uyL21JwaXCfmn__psQdEoTYzBaoRaofgtBc3M3SW-9Q@mail.gmail.com>

2013/5/18 Antoine Pitrou <solipsis at pitrou.net>:
>
> Hello,
>
> I would like to submit the following PEP for discussion and evaluation.

Will the API of the gc module be at all affected? I assume nothing
will just be printed for DEBUG_UNCOLLECTABLE. Maybe there should be a
way to discover when a cycle is resurrected?


--
Regards,
Benjamin

From donald at stufft.io  Mon Jun  3 07:20:42 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 01:20:42 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle
	in CPython)
Message-ID: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>

As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available.

So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does").

Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/69d7d439/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/69d7d439/attachment.pgp>

From clp2 at rebertia.com  Mon Jun  3 07:57:15 2013
From: clp2 at rebertia.com (Chris Rebert)
Date: Sun, 2 Jun 2013 22:57:15 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <CAMZYqRStomPbCfySD9FRT1XNOH8Qfj9ehNK6PKGh1HQ7cCUsOw@mail.gmail.com>

On Jun 2, 2013 10:22 PM, "Donald Stufft" <donald at stufft.io> wrote:
>
> As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available.

Relevant: http://bugs.python.org/issue13647

> So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does").
>
> Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value.

There's an existing request for this:
http://bugs.python.org/issue13655

Cheers,
Chris

From benjamin at python.org  Mon Jun  3 07:58:12 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 2 Jun 2013 22:58:12 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>

2013/6/2 Donald Stufft <donald at stufft.io>:
> As of right now, as far as I can tell, Python does not validate HTTPS
> certificates by default. As far as I can tell this is because there is no
> guaranteed certificates available.
>
> So I would like to propose that CPython adopt the Mozilla SSL certificate
> list and include it in core, and switch over the API's so that they verify
> HTTPS by default.

+1

>
> Ideally this would take the shape of attempting to locate the system
> certificate store if possible, and if that doesn't work falling back to the
> bundled certificates. That way the various Linux distros can easily have
> their copies of Python depend soley on their built in certs, but Windows,
> OSX, Source compiles etc will all still have a fallback value.

My preference would be actually be for the included certificates file
to be used by default. This would provide a consistent experience
across platforms. We could provide options to look for system cert
repositories if desired.



--
Regards,
Benjamin

From donald at stufft.io  Mon Jun  3 09:12:05 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 03:12:05 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
Message-ID: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>


On Jun 3, 2013, at 1:58 AM, Benjamin Peterson <benjamin at python.org> wrote:

> 2013/6/2 Donald Stufft <donald at stufft.io>:
>> As of right now, as far as I can tell, Python does not validate HTTPS
>> certificates by default. As far as I can tell this is because there is no
>> guaranteed certificates available.
>> 
>> So I would like to propose that CPython adopt the Mozilla SSL certificate
>> list and include it in core, and switch over the API's so that they verify
>> HTTPS by default.
> 
> +1
> 
>> 
>> Ideally this would take the shape of attempting to locate the system
>> certificate store if possible, and if that doesn't work falling back to the
>> bundled certificates. That way the various Linux distros can easily have
>> their copies of Python depend soley on their built in certs, but Windows,
>> OSX, Source compiles etc will all still have a fallback value.
> 
> My preference would be actually be for the included certificates file
> to be used by default. This would provide a consistent experience
> across platforms. We could provide options to look for system cert
> repositories if desired.

That's fine with me too. My only reason for wanting to use the system certs first is so
if someone has modified their system certs (say to include a corporate cert) that it
would ideally take affect for Python as well.

But honestly the Linux distros will probably modify things to use system certs anyways
and non Linux (esp Windows) probably doesn't have a way to get those system certs
into OpenSSL.

> 
> 
> 
> --
> Regards,
> Benjamin


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/ce3d1dee/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/ce3d1dee/attachment.pgp>

From benjamin at python.org  Mon Jun  3 09:14:39 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 3 Jun 2013 00:14:39 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
Message-ID: <CAPZV6o8RPsTEcYDfazJtFm2iXr4HVCdRpEVvH=NB9Yz7nfNUZw@mail.gmail.com>

2013/6/3 Donald Stufft <donald at stufft.io>:
>
> On Jun 3, 2013, at 1:58 AM, Benjamin Peterson <benjamin at python.org> wrote:
>
> 2013/6/2 Donald Stufft <donald at stufft.io>:
>
> As of right now, as far as I can tell, Python does not validate HTTPS
> certificates by default. As far as I can tell this is because there is no
> guaranteed certificates available.
>
> So I would like to propose that CPython adopt the Mozilla SSL certificate
> list and include it in core, and switch over the API's so that they verify
> HTTPS by default.
>
>
> +1
>
>
> Ideally this would take the shape of attempting to locate the system
> certificate store if possible, and if that doesn't work falling back to the
> bundled certificates. That way the various Linux distros can easily have
> their copies of Python depend soley on their built in certs, but Windows,
> OSX, Source compiles etc will all still have a fallback value.
>
>
> My preference would be actually be for the included certificates file
> to be used by default. This would provide a consistent experience
> across platforms. We could provide options to look for system cert
> repositories if desired.
>
>
> That's fine with me too. My only reason for wanting to use the system certs
> first is so
> if someone has modified their system certs (say to include a corporate cert)
> that it
> would ideally take affect for Python as well.

I don't think users should be able to modify stdlib behaviors (in this
case could be unintentionally) without application consent.

>
> But honestly the Linux distros will probably modify things to use system
> certs anyways
> and non Linux (esp Windows) probably doesn't have a way to get those system
> certs
> into OpenSSL.

Yes, I'm happy to let them figure it out.



--
Regards,
Benjamin

From benhoyt at gmail.com  Mon Jun  3 09:38:45 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 3 Jun 2013 19:38:45 +1200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
Message-ID: <CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>

Love this idea. Some third-party HTTP libraries turn this on by
default in any case (eg: requests, and I think others), so this would
mean Python would get their "safe-by-default" behaviour in its stdlib.

> > Ideally this would take the shape of attempting to locate the system
> > certificate store if possible, and if that doesn't work falling back to the
> > bundled certificates. That way the various Linux distros can easily have
> > their copies of Python depend soley on their built in certs, but Windows,
> > OSX, Source compiles etc will all still have a fallback value.
>
> My preference would be actually be for the included certificates file
> to be used by default. This would provide a consistent experience
> across platforms. We could provide options to look for system cert
> repositories if desired.

Very much agreed. When the Windows version of the mimetypes module
tried to use Windows' system mimetype mappings by default, chaos and
bugs ensued (for example, http://bugs.python.org/issue15207 and
http://bugs.python.org/issue10551).

-Ben

From solipsis at pitrou.net  Mon Jun  3 11:07:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 11:07:43 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <20130603110743.23886e58@fsol>

On Mon, 3 Jun 2013 01:20:42 -0400
Donald Stufft <donald at stufft.io> wrote:
> As of right now, as far as I can tell, Python does not validate HTTPS
> certificates by default. As far as I can tell this is because there is
> no guaranteed certificates available.

Also for backwards compatibility.

Regards

Antoine.



From solipsis at pitrou.net  Mon Jun  3 11:22:05 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 11:22:05 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAMZYqRStomPbCfySD9FRT1XNOH8Qfj9ehNK6PKGh1HQ7cCUsOw@mail.gmail.com>
Message-ID: <20130603112205.4f76913e@fsol>

On Sun, 2 Jun 2013 22:57:15 -0700
Chris Rebert <clp2 at rebertia.com> wrote:
> On Jun 2, 2013 10:22 PM, "Donald Stufft" <donald at stufft.io> wrote:
> >
> > As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available.
> 
> Relevant: http://bugs.python.org/issue13647
> 
> > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does").
> >
> > Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value.
> 
> There's an existing request for this:
> http://bugs.python.org/issue13655

See also http://bugs.python.org/issue17134

Regards

Antoine.



From rdmurray at bitdance.com  Mon Jun  3 11:28:51 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 03 Jun 2013 05:28:51 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
Message-ID: <20130603092852.2A6532500B9@webabinitio.net>

On Mon, 03 Jun 2013 19:38:45 +1200, Ben Hoyt <benhoyt at gmail.com> wrote:
> Love this idea. Some third-party HTTP libraries turn this on by
> default in any case (eg: requests, and I think others), so this would
> mean Python would get their "safe-by-default" behaviour in its stdlib.
> 
> > > Ideally this would take the shape of attempting to locate the system
> > > certificate store if possible, and if that doesn't work falling back to the
> > > bundled certificates. That way the various Linux distros can easily have
> > > their copies of Python depend soley on their built in certs, but Windows,
> > > OSX, Source compiles etc will all still have a fallback value.
> >
> > My preference would be actually be for the included certificates file
> > to be used by default. This would provide a consistent experience
> > across platforms. We could provide options to look for system cert
> > repositories if desired.
> 
> Very much agreed. When the Windows version of the mimetypes module
> tried to use Windows' system mimetype mappings by default, chaos and
> bugs ensued (for example, http://bugs.python.org/issue15207 and
> http://bugs.python.org/issue10551).

On the other hand, the fact that it uses /etc/mimetypes on unix
does *not* cause chaos, quite the opposite:  it means Python recognizes
new mime types provided by the distro automatically, which provides
for a more consistent experience for the Python application user.

The situation for certs is probably fairly parallel:  on unix, it would
probably be an advantage as Python would automatically follow distro
decisions about cert chains, while on windows trying to use a system
cert store would probably be a disaster.

An application can choose to explicitly ignore the system mimetypes
file, by the way.

That said, I don't feel strongly either way about the cert store case.

--David

From benhoyt at gmail.com  Mon Jun  3 11:37:10 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 3 Jun 2013 21:37:10 +1200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <20130603092852.2A6532500B9@webabinitio.net>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
Message-ID: <CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>

> The situation for certs is probably fairly parallel:  on unix, it would
> probably be an advantage as Python would automatically follow distro
> decisions about cert chains, while on windows trying to use a system
> cert store would probably be a disaster.

Yeah, fair enough. If it's stable and just works on Linux, this would
be an advantage.

> An application can choose to explicitly ignore the system mimetypes
> file, by the way.

My main concern is that it could be broken out of the box on Windows
(mimetypes currently is), and callers have to go out of their way to
find this workaround.

I'm not familiar with Unix/Linux, but on Windows, if it's anything
like mimetypes it'll be really hard to get consistent behaviour across
different boxes/versions from the registry, or wherever certs might be
stored on Windows. I'd much rather have a slightly outdated but
consistent experience by default.

-Ben

From christian at python.org  Mon Jun  3 11:45:10 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 03 Jun 2013 11:45:10 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <51AC65A6.80602@python.org>

Am 03.06.2013 07:20, schrieb Donald Stufft:
> Ideally this would take the shape of attempting to locate the system
> certificate store if possible, and if that doesn't work falling back to
> the bundled certificates. That way the various Linux distros can easily
> have their copies of Python depend soley on their built in certs, but
> Windows, OSX, Source compiles etc will all still have a fallback value.

On Windows it's rather easy to access the cert storage with a couple of
calls to crypt32.dll. I have created a smallish ctypes interface:
https://bitbucket.org/tiran/wincertstore/overview . setuptools 0.7 is
using it to for its ssl support. I'm going to write a patch for Python
3.4, too. See http://bugs.python.org/issue17134

I welcome a cert bundle in CPython. But how are we suppose to handle
updates and invalidation of CAs? A couple of months ago there was an
intense discussion about a copy of the Olson time zone database in
Python's stdlib. People argued against it as they feared outdated tz
information. IMHO outdated CA certs are much more severe than tz ...

I'd like to see OCSP support, too. Unfortunately OpenSSL's OCSP API is
far from trivial. I had a look at it once.

Christian

From solipsis at pitrou.net  Mon Jun  3 11:51:30 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 11:51:30 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
Message-ID: <20130603115130.299b2504@fsol>

On Mon, 3 Jun 2013 21:37:10 +1200
Ben Hoyt <benhoyt at gmail.com> wrote:
> 
> I'm not familiar with Unix/Linux, but on Windows, if it's anything
> like mimetypes it'll be really hard to get consistent behaviour across
> different boxes/versions from the registry, or wherever certs might be
> stored on Windows. I'd much rather have a slightly outdated but
> consistent experience by default.

The problem with a "slightly outdated" CA store is that it can be a
security risk.

Regards

Antoine.



From solipsis at pitrou.net  Mon Jun  3 11:52:52 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 11:52:52 +0200
Subject: [Python-Dev] PEP 442: Safe object finalization
In-Reply-To: <CAPZV6o-uyL21JwaXCfmn__psQdEoTYzBaoRaofgtBc3M3SW-9Q@mail.gmail.com>
References: <20130518105910.1eecfa5f@fsol>
	<CAPZV6o-uyL21JwaXCfmn__psQdEoTYzBaoRaofgtBc3M3SW-9Q@mail.gmail.com>
Message-ID: <20130603115252.6c07ff38@fsol>

On Sun, 2 Jun 2013 19:27:49 -0700
Benjamin Peterson <benjamin at python.org> wrote:
> 2013/5/18 Antoine Pitrou <solipsis at pitrou.net>:
> >
> > Hello,
> >
> > I would like to submit the following PEP for discussion and evaluation.
> 
> Will the API of the gc module be at all affected? I assume nothing
> will just be printed for DEBUG_UNCOLLECTABLE.

Objects with tp_del may still exist (third-party extensions perhaps).

> Maybe there should be a
> way to discover when a cycle is resurrected?

Is it more important than discovering when a non-cycle is resurrected?

Regards

Antoine.

From solipsis at pitrou.net  Mon Jun  3 11:53:55 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 11:53:55 +0200
Subject: [Python-Dev] PEP 442: Safe object finalization
In-Reply-To: <CAPZV6o_-E-8O4cQGyDK5kmW9ZY+BiZu6kuXrp=2LF0GydmSEkA@mail.gmail.com>
References: <20130518105910.1eecfa5f@fsol>
	<CAMSv6X39grvMWsAEkXBUdwHdYr-5y6Mwt_MdFdN3KjCs-EC3_w@mail.gmail.com>
	<20130518154552.2a879bf6@fsol>
	<CAMSv6X1J9JaKsLJBhMU+nZOa+M0eV8ZQawWe9iEE_eL0bhmMVw@mail.gmail.com>
	<20130518163315.6a21a0cd@fsol>
	<CAPZV6o_-E-8O4cQGyDK5kmW9ZY+BiZu6kuXrp=2LF0GydmSEkA@mail.gmail.com>
Message-ID: <20130603115355.356bc4e4@fsol>

On Sun, 2 Jun 2013 19:16:17 -0700
Benjamin Peterson <benjamin at python.org> wrote:
> 2013/5/18 Antoine Pitrou <solipsis at pitrou.net>:
> > Calling finalizers only once is fine with me, but it would be a change
> > in behaviour; I don't know if it may break existing code.
> 
> I agree with Armin that this is better behavior. (Mostly significantly
> consistent with weakrefs.)

Keep in mind that it is a limitation of weakrefs, not a feature: you
can't "unclear" a weakref.

> > (for example, say someone is using __del__ to manage a freelist)
> 
> Do you know if it breaks any of the projects you tested it with?

I haven't tested it (yet).

Regards

Antoine.

From benhoyt at gmail.com  Mon Jun  3 11:57:10 2013
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 3 Jun 2013 21:57:10 +1200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <20130603115130.299b2504@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
Message-ID: <CAL9jXCH0maadkcGL90rzC9hJbsMHK9RtRSrALRSSd0k5Rymy-g@mail.gmail.com>

>> I'm not familiar with Unix/Linux, but on Windows, if it's anything
>> like mimetypes it'll be really hard to get consistent behaviour across
>> different boxes/versions from the registry, or wherever certs might be
>> stored on Windows. I'd much rather have a slightly outdated but
>> consistent experience by default.
>
> The problem with a "slightly outdated" CA store is that it can be a
> security risk.

True. This is different from mimetypes in that respect.

Also, with Christian's post about Windows cert store, it does look
like Windows has a stable/documented way to get this from the system.
I wonder, are the crypt32.dll Cert* functions what IE uses?

-Ben

From ben at bendarnell.com  Mon Jun  3 15:05:14 2013
From: ben at bendarnell.com (Ben Darnell)
Date: Mon, 3 Jun 2013 09:05:14 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>

On Mon, Jun 3, 2013 at 1:20 AM, Donald Stufft <donald at stufft.io> wrote:

> As of right now, as far as I can tell, Python does not validate HTTPS
> certificates by default. As far as I can tell this is because there is no
> guaranteed certificates available.
>
> So I would like to propose that CPython adopt the Mozilla SSL certificate
> list and include it in core, and switch over the API's so that they verify
> HTTPS by default. This is what most people are going to expect when using a
> https url (Especially after learning that Python 2.x doesn't verify TLS,
> but Python 3.x "does").
>
> Ideally this would take the shape of attempting to locate the system
> certificate store if possible, and if that doesn't work falling back to the
> bundled certificates. That way the various Linux distros can easily have
> their copies of Python depend soley on their built in certs, but Windows,
> OSX, Source compiles etc will all still have a fallback value.
>

+1.  I bundle a copy of the Mozilla CA list with Tornado, but would love to
access the system's CA roots and/or use a Python-provided copy.  I'd prefer
to use the certificates from the operating system when possible, as that
list is most likely to receive timely security updates (or be updated with
a local corporate CA, for example).  It's better to aim for consistency
with the user's browser than consistency of Python applications across
different installations.

The data is analogous to the time zone database (PEP 431) in that it may
need to be updated independently of Python's own release schedule, so we
may want to use similar techniques to manage both.  Also see certifi (
https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list
in a pip-installable form.

-Ben


>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/95d2c9fc/attachment.html>

From ethan at stoneleaf.us  Mon Jun  3 17:27:24 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 03 Jun 2013 08:27:24 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <51ACB5DC.3040902@stoneleaf.us>

On 06/02/2013 10:20 PM, Donald Stufft wrote:
>
> So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over
> the API's so that they verify HTTPS by default.

+1

Also, +1 on using system certs when available.

--
~Ethan~

From donald at stufft.io  Mon Jun  3 18:43:32 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 12:43:32 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603115130.299b2504@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
Message-ID: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>


On Jun 3, 2013, at 5:51 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 3 Jun 2013 21:37:10 +1200
> Ben Hoyt <benhoyt at gmail.com> wrote:
>> 
>> I'm not familiar with Unix/Linux, but on Windows, if it's anything
>> like mimetypes it'll be really hard to get consistent behaviour across
>> different boxes/versions from the registry, or wherever certs might be
>> stored on Windows. I'd much rather have a slightly outdated but
>> consistent experience by default.
> 
> The problem with a "slightly outdated" CA store is that it can be a
> security risk.
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/956e6fc3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/956e6fc3/attachment.pgp>

From solipsis at pitrou.net  Mon Jun  3 18:56:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 18:56:08 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
	<95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
Message-ID: <20130603185608.4cbfae0f@fsol>

On Mon, 3 Jun 2013 12:43:32 -0400
Donald Stufft <donald at stufft.io> wrote:
> 
> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > On Mon, 3 Jun 2013 21:37:10 +1200
> > Ben Hoyt <benhoyt at gmail.com> wrote:
> >> 
> >> I'm not familiar with Unix/Linux, but on Windows, if it's anything
> >> like mimetypes it'll be really hard to get consistent behaviour across
> >> different boxes/versions from the registry, or wherever certs might be
> >> stored on Windows. I'd much rather have a slightly outdated but
> >> consistent experience by default.
> > 
> > The problem with a "slightly outdated" CA store is that it can be a
> > security risk.
> > 
> > Regards
> > 
> > Antoine.
> > 
> > 
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io
> 
> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update.

Let's see if our security release managers want to do that job.

Regards

Antoine.

From ethan at stoneleaf.us  Mon Jun  3 18:52:27 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 03 Jun 2013 09:52:27 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
	<95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
Message-ID: <51ACC9CB.1090407@stoneleaf.us>

On 06/03/2013 09:43 AM, Donald Stufft wrote:
> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote:
>>
>> The problem with a "slightly outdated" CA store is that it can be a
>> security risk.
>
> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just
> need to watch them for blacklisting certs and roll that into a security update.

Personally, I'm not interested in waiting six months for an update.  And why can't I have the new additions?

Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), 
as well as having the bundle (black lists as well as new additions) in the regular updates.

--
~Ethan~

From donald at stufft.io  Mon Jun  3 19:55:16 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 13:55:16 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <51ACC9CB.1090407@stoneleaf.us>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
	<95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
	<51ACC9CB.1090407@stoneleaf.us>
Message-ID: <6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io>


On Jun 3, 2013, at 12:52 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 06/03/2013 09:43 AM, Donald Stufft wrote:
>> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote:
>>> 
>>> The problem with a "slightly outdated" CA store is that it can be a
>>> security risk.
>> 
>> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just
>> need to watch them for blacklisting certs and roll that into a security update.
> 
> Personally, I'm not interested in waiting six months for an update.  And why can't I have the new additions?
> 
> Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates.
> 
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just replace the default certificates. The goal here is to get secure by default without any work on the part of the developers. If a developer knows to update the certs they can know to use certifi.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/5a203665/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/5a203665/attachment.pgp>

From barry at python.org  Mon Jun  3 18:36:36 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 Jun 2013 12:36:36 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
Message-ID: <20130603123636.7a44dd9c@anarchist>

On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote:

>So I would like to propose that CPython adopt the Mozilla SSL certificate
>list and include it in core, and switch over the API's so that they verify
>HTTPS by default. This is what most people are going to expect when using a
>https url (Especially after learning that Python 2.x doesn't verify TLS, but
>Python 3.x "does").

For the "verify HTTPS by default", do you mean specifically changing the
cadefault argument to urllib.request.urlopen() to True?  Note that I recently
closed a bug related to this:

http://bugs.python.org/issue17977

+1 for changing the default to True.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/8fb02bfe/attachment.pgp>

From barry at python.org  Mon Jun  3 18:48:36 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 Jun 2013 12:48:36 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
Message-ID: <20130603124836.23a83360@anarchist>

On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote:

>The data is analogous to the time zone database (PEP 431) in that it may
>need to be updated independently of Python's own release schedule, so we
>may want to use similar techniques to manage both.  Also see certifi (
>https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list
>in a pip-installable form.

Right, this is very much analogous, except with the additional twist that
out-of-date certificates can pose a significant security risk.

I'm fairly certain that Debian and Ubuntu would explicitly not use any
certificates shipped with Python, for two main reasons: 1) our security teams
already manage the certificate store distro-wide and we want to make sure that
one update fixes everything; 2) we don't want to duplicate code in multiple
packages[1].

So *if* Python decides to do this (and I'm -0, but from a decidedly
Linux-distro bias), it must be easily disabled.  I generally like the way PEP
431 handles the tzdata, so I think we should do the same here.

-Barry

[1] This gives us headaches in upstreams like coverage caused by bundling
externally available JavaScript libraries, or like urllib3 bundling chardet
and urllib3, not to mention their own certificates yet again. :(

From donald at stufft.io  Mon Jun  3 20:12:34 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 14:12:34 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603123636.7a44dd9c@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
Message-ID: <B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>


On Jun 3, 2013, at 12:36 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote:
> 
>> So I would like to propose that CPython adopt the Mozilla SSL certificate
>> list and include it in core, and switch over the API's so that they verify
>> HTTPS by default. This is what most people are going to expect when using a
>> https url (Especially after learning that Python 2.x doesn't verify TLS, but
>> Python 3.x "does").
> 
> For the "verify HTTPS by default", do you mean specifically changing the
> cadefault argument to urllib.request.urlopen() to True?  Note that I recently
> closed a bug related to this:
> 
> http://bugs.python.org/issue17977
> 
> +1 for changing the default to True.
> 
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

If that's all it takes to make sure that people have to opt out of an invalid certificate throwing
an error than yes :)

My goal here is to make it that when someone accesses a TLS secured network resource (I said HTTP, but if there are other things, xmlrpclib, or what not where people can reasonably expect valid TLS certificates those too) they are protected by a MITM attack by default. I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe".

As with most things security the outcome of "failing" to do it right when the default is insecure is well nothing until someone attacks you. So Hopefully we make things secure by default for folks :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/39ff8f6a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/39ff8f6a/attachment-0001.pgp>

From barry at python.org  Mon Jun  3 18:52:00 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 Jun 2013 12:52:00 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
Message-ID: <20130603125200.69de2e13@anarchist>

On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote:

>That's fine with me too. My only reason for wanting to use the system certs
>first is so if someone has modified their system certs (say to include a
>corporate cert) that it would ideally take affect for Python as well.

This reminds me of one other thing.  We have to make sure that the APIs
(e.g urlopen()) continue to allow us to use self-signed certificates, if for
no other reason than for testing purposes.  OTOH, taking this away would be a
backward incompatible change in API so probably wouldn't happen anyway.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/d930edbe/attachment.pgp>

From donald at stufft.io  Mon Jun  3 20:17:48 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 14:17:48 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603124836.23a83360@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
	<20130603124836.23a83360@anarchist>
Message-ID: <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io>


On Jun 3, 2013, at 12:48 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote:
> 
>> The data is analogous to the time zone database (PEP 431) in that it may
>> need to be updated independently of Python's own release schedule, so we
>> may want to use similar techniques to manage both.  Also see certifi (
>> https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list
>> in a pip-installable form.
> 
> Right, this is very much analogous, except with the additional twist that
> out-of-date certificates can pose a significant security risk.
> 
> I'm fairly certain that Debian and Ubuntu would explicitly not use any
> certificates shipped with Python, for two main reasons: 1) our security teams
> already manage the certificate store distro-wide and we want to make sure that
> one update fixes everything; 2) we don't want to duplicate code in multiple
> packages[1].
> 
> So *if* Python decides to do this (and I'm -0, but from a decidedly
> Linux-distro bias), it must be easily disabled.  I generally like the way PEP
> 431 handles the tzdata, so I think we should do the same here.

I'd actually prefer for Linux to not use the bundled certs when installed from a package
manager because it should use the system certs, but people can't depend on certs
being there if they are only there on linux.

Adding them into Python means people _can_ depend on them being there, and Windows
and other systems without system integrators to modify it to use the system store will still
get certs and Ubuntu can make it just work(?).

> 
> -Barry
> 
> [1] This gives us headaches in upstreams like coverage caused by bundling
> externally available JavaScript libraries, or like urllib3 bundling chardet
> and urllib3, not to mention their own certificates yet again. :(

This would probably (eventually) make the bundling of certificates better too.

Meaning that once it's been in long enough people are willing to depend on it, they
won't need to bundle their own certs and ubuntu/debian can just modify the one
location instead of needing to modify it for every package that does it.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/c0cd530c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/c0cd530c/attachment.pgp>

From donald at stufft.io  Mon Jun  3 20:21:06 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 14:21:06 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603125200.69de2e13@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
	<20130603125200.69de2e13@anarchist>
Message-ID: <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io>


On Jun 3, 2013, at 12:52 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote:
> 
>> That's fine with me too. My only reason for wanting to use the system certs
>> first is so if someone has modified their system certs (say to include a
>> corporate cert) that it would ideally take affect for Python as well.
> 
> This reminds me of one other thing.  We have to make sure that the APIs
> (e.g urlopen()) continue to allow us to use self-signed certificates, if for
> no other reason than for testing purposes.  OTOH, taking this away would be a
> backward incompatible change in API so probably wouldn't happen anyway.
> 
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this "error" shouldn't be silent by default.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/f0f65aef/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/f0f65aef/attachment.pgp>

From barry at python.org  Mon Jun  3 19:04:07 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 Jun 2013 13:04:07 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
	<20130603125200.69de2e13@anarchist>
	<706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io>
Message-ID: <20130603130407.43d1cd0d@anarchist>

On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote:

>The other additional comment I'd like to throw in here is that if we don't
>bundle SSL certs I think we should still verify by default (which means HTTPS
>urls will throw an error by default if we can't locate a certificate store)
>because I think the risk to people unknowingly thinking that their HTTPS urls
>are protected are significant enough that this "error" shouldn't be silent by
>default.

+1, especially if we ensure that the APIs are available to not verify, as is
currently the case with urlopen().  I don't think people will want to do that
in production, but it will be useful for testing (e.g. guess how I found
issues 17977 :).

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/3f5d198e/attachment.pgp>

From barry at python.org  Mon Jun  3 19:07:00 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 Jun 2013 13:07:00 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
	<20130603124836.23a83360@anarchist>
	<1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io>
Message-ID: <20130603130700.4482ae2a@anarchist>

On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote:

>I'd actually prefer for Linux to not use the bundled certs when installed
>from a package manager because it should use the system certs, but people
>can't depend on certs being there if they are only there on linux.

I think we agree on that.

>Adding them into Python means people _can_ depend on them being there, and
>Windows and other systems without system integrators to modify it to use the
>system store will still get certs and Ubuntu can make it just work(?).

Again, I think PEP 431 provides a pretty good model for how this should be
done.  Maybe it's worth factoring out this specific part of PEP 431 into an
informational PEP?

>This would probably (eventually) make the bundling of certificates better
>too.
>
>Meaning that once it's been in long enough people are willing to depend on
>it, they won't need to bundle their own certs and ubuntu/debian can just
>modify the one location instead of needing to modify it for every package
>that does it.

Can we do the same for the JavaScript libraries? :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/24178cfd/attachment.pgp>

From solipsis at pitrou.net  Mon Jun  3 20:34:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 20:34:08 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
Message-ID: <20130603203408.2c800172@fsol>

On Mon, 3 Jun 2013 14:12:34 -0400
Donald Stufft <donald at stufft.io> wrote:
> 
> I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe".

Yet there's quite a visible warning in the docs:
http://docs.python.org/dev/library/urllib.request.html

Regards

Antoine.



From ethan at stoneleaf.us  Mon Jun  3 20:15:51 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 03 Jun 2013 11:15:51 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<CAL9jXCF7TQ-=e5uxZhdXxGLUuaRidVh8_jNz4V-Laep4sbXaxw@mail.gmail.com>
	<20130603092852.2A6532500B9@webabinitio.net>
	<CAL9jXCHmEAb8nwCm=Vnp8xFUiK0=Q2EJg-+VBMk7nN0gB0nzrg@mail.gmail.com>
	<20130603115130.299b2504@fsol>
	<95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io>
	<51ACC9CB.1090407@stoneleaf.us>
	<6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io>
Message-ID: <51ACDD57.10808@stoneleaf.us>

On 06/03/2013 10:55 AM, Donald Stufft wrote:
> On Jun 3, 2013, at 12:52 PM, Ethan Furman wrote:
>> On 06/03/2013 09:43 AM, Donald Stufft wrote:
>>> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote:
>>>>
>>>> The problem with a "slightly outdated" CA store is that it can be a
>>>> security risk.
>>>
>>> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just
>>> need to watch them for blacklisting certs and roll that into a security update.
>>
>> Personally, I'm not interested in waiting six months for an update.  And why can't I have the new additions?
>>
>> Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call
>> them), as well as having the bundle (black lists as well as new additions) in the regular updates.
>
> Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just
> replace the default certificates. The goal here is to get secure by default without any work on the part of the
> developers. If a developer knows to update the certs they can know to use certifi.

Let's pretend I want to do something with https.  I'm pretty much a complete novice in that area, but I do know about 
these certificate things and that I should use them.  (Which will probably happen sooner rather than later. ;)

 From my POV, having the bundled certs in the stdlib is great, but also having something in the stdlib to query for and 
download updates would also be great.  I just looked at certifi and the only thing it has on there is `where` which 
tells me where the certs are on the system.  This on only half the problem.  If we're going to solve it, why not solve 
the whole problem?

--
~Ethan~

From donald at stufft.io  Mon Jun  3 21:06:56 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 15:06:56 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603130700.4482ae2a@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
	<20130603124836.23a83360@anarchist>
	<1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io>
	<20130603130700.4482ae2a@anarchist>
Message-ID: <084A63CF-6466-4EBB-8554-3D3DB5C4B17F@stufft.io>


On Jun 3, 2013, at 1:07 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote:
> 
>> I'd actually prefer for Linux to not use the bundled certs when installed
>> from a package manager because it should use the system certs, but people
>> can't depend on certs being there if they are only there on linux.
> 
> I think we agree on that.
> 
>> Adding them into Python means people _can_ depend on them being there, and
>> Windows and other systems without system integrators to modify it to use the
>> system store will still get certs and Ubuntu can make it just work(?).
> 
> Again, I think PEP 431 provides a pretty good model for how this should be
> done.  Maybe it's worth factoring out this specific part of PEP 431 into an
> informational PEP?

Looks fine to me minus the not updating in security releases (but that's just
a difference in the type of data).

> 
>> This would probably (eventually) make the bundling of certificates better
>> too.
>> 
>> Meaning that once it's been in long enough people are willing to depend on
>> it, they won't need to bundle their own certs and ubuntu/debian can just
>> modify the one location instead of needing to modify it for every package
>> that does it.
> 
> Can we do the same for the JavaScript libraries? :)
> 
> -Barry


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/77894bd5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/77894bd5/attachment-0001.pgp>

From brett at python.org  Mon Jun  3 21:20:27 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 3 Jun 2013 15:20:27 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <20130603130407.43d1cd0d@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
	<0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io>
	<20130603125200.69de2e13@anarchist>
	<706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io>
	<20130603130407.43d1cd0d@anarchist>
Message-ID: <CAP1=2W7Zrp4O=5PnO4n975u6duRREqTCPSFoOya87q=_6ssrQg@mail.gmail.com>

On Mon, Jun 3, 2013 at 1:04 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote:
>
> >The other additional comment I'd like to throw in here is that if we don't
> >bundle SSL certs I think we should still verify by default (which means
> HTTPS
> >urls will throw an error by default if we can't locate a certificate
> store)
> >because I think the risk to people unknowingly thinking that their HTTPS
> urls
> >are protected are significant enough that this "error" shouldn't be
> silent by
> >default.
>
> +1, especially if we ensure that the APIs are available to not verify, as
> is
> currently the case with urlopen().  I don't think people will want to do
> that
> in production, but it will be useful for testing (e.g. guess how I found
> issues 17977 :).
>

+1 from me as well. Whether we bundle or simply provide a command to
download the certs I think making this default is the bare-minimum,
especially if setting nothing more than cadefault=True is all that is
needed to get this behaviour since that's backwards-compatible to Python
3.3.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/1631237a/attachment.html>

From ethan at stoneleaf.us  Mon Jun  3 20:56:45 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 03 Jun 2013 11:56:45 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <20130603203408.2c800172@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol>
Message-ID: <51ACE6ED.7000200@stoneleaf.us>

On 06/03/2013 11:34 AM, Antoine Pitrou wrote:
> On Mon, 3 Jun 2013 14:12:34 -0400
> Donald Stufft wrote:
>>
>> I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe".
>
> Yet there's quite a visible warning in the docs:
> http://docs.python.org/dev/library/urllib.request.html

As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. 
  The dangerous/unexpected behavior should not be the default.

--
~Ethan~

From solipsis at pitrou.net  Mon Jun  3 21:52:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 21:52:44 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
Message-ID: <20130603215244.576896b2@fsol>

On Mon, 03 Jun 2013 11:56:45 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/03/2013 11:34 AM, Antoine Pitrou wrote:
> > On Mon, 3 Jun 2013 14:12:34 -0400
> > Donald Stufft wrote:
> >>
> >> I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe".
> >
> > Yet there's quite a visible warning in the docs:
> > http://docs.python.org/dev/library/urllib.request.html
> 
> As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. 
>   The dangerous/unexpected behavior should not be the default.

cadefault=True will probably be fail if the system certs are not
properly configured in OpenSSL, e.g. under Windows or with a hand-made
OpenSSL build.
And, because of the way the OpenSSL API works, there's no way of
knowing if it is the case or not:
http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths

While I'm not saying we shouldn't try to verify certs by default, doing
so is a little more delicate than "setting cadefault to True",
unfortunately. Bundling our own CA cert store is an option, assuming
Donald or someone else wants to take responsibility for it.

Regards

Antoine.



From rosuav at gmail.com  Mon Jun  3 22:05:01 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 4 Jun 2013 06:05:01 +1000
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <51ACE6ED.7000200@stoneleaf.us>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
Message-ID: <CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>

On Tue, Jun 4, 2013 at 4:56 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/03/2013 11:34 AM, Antoine Pitrou wrote:
>>
>> On Mon, 3 Jun 2013 14:12:34 -0400
>> Donald Stufft wrote:
>>>
>>>
>>> I worry with the current situation people will just use TLS connections
>>> without realizing it's not being verified and thinking they are "safe".
>>
>>
>> Yet there's quite a visible warning in the docs:
>> http://docs.python.org/dev/library/urllib.request.html
>
>
> As has been been mentioned elsewhere, at the very least we should change
> cadefault to True for secure-type connections.  The dangerous/unexpected
> behavior should not be the default.

+1 for having the default be safe, but this will have to be very
loudly announced ("when migrating from 3.3 to 3.4, TLS connections
will cease to work if blah blah"). Some legit sites with proper
certificates still manage to muck something up administratively
(developer.quicksales.com.au has a cert from RapidSSL but doesn't
bundle the intermediates, and I've told their devs about it, but all I
can do is disable cert checking). This will break code in ways that
will surprise people greatly. But I'd still rather the default be
True.

ChrisA

From dmalcolm at redhat.com  Mon Jun  3 22:07:05 2013
From: dmalcolm at redhat.com (David Malcolm)
Date: Mon, 03 Jun 2013 16:07:05 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <20130603124836.23a83360@anarchist>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAFkYKJ6LCFtcsJW8M-Ne52NdThk7gz27-CqYZiJx6gByAuJ19g@mail.gmail.com>
	<20130603124836.23a83360@anarchist>
Message-ID: <1370290025.22768.24.camel@surprise>

On Mon, 2013-06-03 at 12:48 -0400, Barry Warsaw wrote:
> On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote:
> 
> >The data is analogous to the time zone database (PEP 431) in that it may
> >need to be updated independently of Python's own release schedule, so we
> >may want to use similar techniques to manage both.  Also see certifi (
> >https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list
> >in a pip-installable form.
> 
> Right, this is very much analogous, except with the additional twist that
> out-of-date certificates can pose a significant security risk.
> 
> I'm fairly certain that Debian and Ubuntu would explicitly not use any
> certificates shipped with Python, for two main reasons: 1) our security teams
> already manage the certificate store distro-wide and we want to make sure that
> one update fixes everything; 2) we don't want to duplicate code in multiple
> packages[1].

Fedora/RHEL are in a similar position; I expect we'd rip out the bundled
certs in our builds shortly after unzipping the tarball, and use a
system-wide cert store (I "rm -rf" bundled libraries in our builds, to
make sure we're not using them).

[...snip...]



From christian at python.org  Mon Jun  3 22:19:26 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 03 Jun 2013 22:19:26 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603215244.576896b2@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<20130603215244.576896b2@fsol>
Message-ID: <51ACFA4E.6050501@python.org>

Am 03.06.2013 21:52, schrieb Antoine Pitrou:
> cadefault=True will probably be fail if the system certs are not
> properly configured in OpenSSL, e.g. under Windows or with a hand-made
> OpenSSL build.
> And, because of the way the OpenSSL API works, there's no way of
> knowing if it is the case or not:
> http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths

I only see an issue for uncommon Linux distributions and exotic Unices.

For Windows an interface to crypt32 API solves the CA issue as shown in
my wincertstore module. It gives the user the same SSL experience as
Internet Explorer.

Most Linux and BSD-ish operating systems have SSL certs at some standard
location.
https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15
lists a couple of standard locations.

Under which conditions do we need to ship a CA cert file?

Christian


From donald at stufft.io  Mon Jun  3 22:22:41 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 16:22:41 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <51ACFA4E.6050501@python.org>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<20130603215244.576896b2@fsol> <51ACFA4E.6050501@python.org>
Message-ID: <699D9AB5-26E0-4CCE-89C5-8E92C3305BDE@stufft.io>


On Jun 3, 2013, at 4:19 PM, Christian Heimes <christian at python.org> wrote:

> Am 03.06.2013 21:52, schrieb Antoine Pitrou:
>> cadefault=True will probably be fail if the system certs are not
>> properly configured in OpenSSL, e.g. under Windows or with a hand-made
>> OpenSSL build.
>> And, because of the way the OpenSSL API works, there's no way of
>> knowing if it is the case or not:
>> http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths
> 
> I only see an issue for uncommon Linux distributions and exotic Unices.
> 
> For Windows an interface to crypt32 API solves the CA issue as shown in
> my wincertstore module. It gives the user the same SSL experience as
> Internet Explorer.
> 
> Most Linux and BSD-ish operating systems have SSL certs at some standard
> location.
> https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15
> lists a couple of standard locations.
> 
> Under which conditions do we need to ship a CA cert file?
> 
> Christian
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


What about OSX?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/0b218f15/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/0b218f15/attachment.pgp>

From nad at acm.org  Mon Jun  3 23:24:59 2013
From: nad at acm.org (Ned Deily)
Date: Mon, 03 Jun 2013 14:24:59 -0700
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<20130603215244.576896b2@fsol> <51ACFA4E.6050501@python.org>
	<699D9AB5-26E0-4CCE-89C5-8E92C3305BDE@stufft.io>
Message-ID: <nad-76FAE3.14245903062013@news.gmane.org>

In article <699D9AB5-26E0-4CCE-89C5-8E92C3305BDE at stufft.io>,
 Donald Stufft <donald at stufft.io> wrote:
> On Jun 3, 2013, at 4:19 PM, Christian Heimes <christian at python.org> wrote: 
> > Most Linux and BSD-ish operating systems have SSL certs at some standard
> > location.
> > https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976c
> > fc38cd/setuptools/ssl_support.py?at=default#cl-15
> > lists a couple of standard locations.
> > Under which conditions do we need to ship a CA cert file?
> What about OSX?

OS X has an actively managed set of CA certs, by Apple and with 
provision for users to tailor their security policies (add/modify/delete 
certs and their policies via either a gui or cli).  For python.org OS X 
installers, we currently depend on the system-provided versions of 
libssl and libcrypto which use the system/user cert infrastructure.  But 
Apple has deprecated the OpenSSL-compatible APIs and we have an open 
issue (Issue17128) to move to supplying our own copy of OpenSSL libs.  
One hangup has been what to do about the certs.  One solution would be 
to use a Python distribution of them as discussed here.  Another, 
probably better for users, solution is to continue to use the existing 
system/user cert infrastructure; recently, we've seen a new approach to 
that which we need to look into: 
http://bugs.python.org/issue17128#msg189244.  Third-party distributors 
of Python on OS X may use other solutions; for example, MacPorts already 
ships and links with its own version of OpenSSL.

-- 
 Ned Deily,
 nad at acm.org


From p.f.moore at gmail.com  Mon Jun  3 23:31:40 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 3 Jun 2013 22:31:40 +0100
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
Message-ID: <CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>

On 3 June 2013 21:05, Chris Angelico <rosuav at gmail.com> wrote:

> +1 for having the default be safe, but this will have to be very
> loudly announced ("when migrating from 3.3 to 3.4, TLS connections
> will cease to work if blah blah").
>

+1 on the default being safe, certainly. But with the proviso that the same
code should work in 3.3 and 3.4, with no user impact (other than that the
connection is safer, but that's not user-visible unless there's an attack
:-))

In other words, that "will cease to work" clause should not exist - but see
below...


> Some legit sites with proper
> certificates still manage to muck something up administratively
> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
> bundle the intermediates, and I've told their devs about it, but all I
> can do is disable cert checking). This will break code in ways that
> will surprise people greatly. But I'd still rather the default be
> True.
>

I'm happy if the "will cease to work" clause only says "some sites with
broken security configurations may stop working" with a clear explanation
that it is *their* fault, not Python's. I'd also expect that the same sites
would fail in browsers - if not, we should also be able to make them work
(or face cries of "well, Internet Explorer/Firefox doesn't have a problem
with my site, why does Python?").

Also, we should consider the issue for application users. Suppose I'm using
a Python application that downloads something from the web. I upgrade to
3.4, and the app stops working because of a "will cease to work" case. As
an end user, how can I get the app working again? Having to patch the
sources isn't an option, and reverting to 3.3 provokes the reaction "Python
broke my app".

Summary - I'm +1 as long as either the "will cease to work" list is empty,
or we have a *very* good story for (legitimate) sites which do cease to
work.

Paul.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/0435465e/attachment.html>

From solipsis at pitrou.net  Mon Jun  3 23:41:19 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 23:41:19 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
Message-ID: <20130603234119.7cdeed4b@fsol>

On Mon, 3 Jun 2013 22:31:40 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> 
> > Some legit sites with proper
> > certificates still manage to muck something up administratively
> > (developer.quicksales.com.au has a cert from RapidSSL but doesn't
> > bundle the intermediates, and I've told their devs about it, but all I
> > can do is disable cert checking). This will break code in ways that
> > will surprise people greatly. But I'd still rather the default be
> > True.
> >
> 
> I'm happy if the "will cease to work" clause only says "some sites with
> broken security configurations may stop working" with a clear explanation
> that it is *their* fault, not Python's. I'd also expect that the same sites
> would fail in browsers - if not, we should also be able to make them work
> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem
> with my site, why does Python?").

Keep in mind that not every HTTPS service is a Web site that is meant
to be readable with a browser. Some are Web services, possibly internal,
possibly without a domain name (and, therefore, probably a non-matching
certificate subject name).

Regards

Antoine.



From rosuav at gmail.com  Mon Jun  3 23:46:18 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 4 Jun 2013 07:46:18 +1000
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
Message-ID: <CAPTjJmpcTOYAAHBvT2Rffs4LYbyNaZ6Hhr4_wQP_+BKA0imQkQ@mail.gmail.com>

On Tue, Jun 4, 2013 at 7:31 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> On 3 June 2013 21:05, Chris Angelico <rosuav at gmail.com> wrote:
>> Some legit sites with proper
>> certificates still manage to muck something up administratively
>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
>> bundle the intermediates, and I've told their devs about it, but all I
>> can do is disable cert checking). This will break code in ways that
>> will surprise people greatly. But I'd still rather the default be
>> True.
>
> I'm happy if the "will cease to work" clause only says "some sites with
> broken security configurations may stop working" with a clear explanation
> that it is *their* fault, not Python's. I'd also expect that the same sites
> would fail in browsers - if not, we should also be able to make them work
> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem
> with my site, why does Python?").

In this specific case, I get no warning from Google Chrome on either
Windows or Linux, but I do get a warning from Iceweasel on the same
Linux (it's Debian Wheezy, fwiw). So there probably will be cries of
"well, Chrome doesn't etc etc".

> Also, we should consider the issue for application users. Suppose I'm using
> a Python application that downloads something from the web. I upgrade to
> 3.4, and the app stops working because of a "will cease to work" case. As an
> end user, how can I get the app working again? Having to patch the sources
> isn't an option, and reverting to 3.3 provokes the reaction "Python broke my
> app".

Right. The instance I'm citing comes from what I'm doing at work; I
had to disable certificate checking in order to get a non-interactive
application to work. If I'd never thought to make sure cert checking
was happening, the app could easily have been unexpectedly broken by
this change.

But you're absolutely right. It's a failing at the remote end, or...

On Tue, Jun 4, 2013 at 7:41 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Keep in mind that not every HTTPS service is a Web site that is meant
> to be readable with a browser. Some are Web services, possibly internal,
> possibly without a domain name (and, therefore, probably a non-matching
> certificate subject name).

... this. This isn't a problem, as checking can easily be disabled,
but I'd be inclined to recommend that this one be solved without
changing code by explicitly importing that certificate. That would
resolve everything, but would require administrative action.

ChrisA

From donald at stufft.io  Mon Jun  3 23:46:52 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 17:46:52 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
Message-ID: <EE38CDBB-327D-4DB5-83AF-0BCAD81B586F@stufft.io>


On Jun 3, 2013, at 5:31 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> 
> On 3 June 2013 21:05, Chris Angelico <rosuav at gmail.com> wrote:
> +1 for having the default be safe, but this will have to be very
> loudly announced ("when migrating from 3.3 to 3.4, TLS connections
> will cease to work if blah blah").
> 
> +1 on the default being safe, certainly. But with the proviso that the same code should work in 3.3 and 3.4, with no user impact (other than that the connection is safer, but that's not user-visible unless there's an attack :-))

If we bundle certs that will be the case sans connections where it doesn't validate.

> 
> In other words, that "will cease to work" clause should not exist - but see below...
>  
> Some legit sites with proper
> certificates still manage to muck something up administratively
> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
> bundle the intermediates, and I've told their devs about it, but all I
> can do is disable cert checking). This will break code in ways that
> will surprise people greatly. But I'd still rather the default be
> True.
> 
> I'm happy if the "will cease to work" clause only says "some sites with broken security configurations may stop working" with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of "well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?").

Browsers tend to download intermediate certs while I don't think Python does.

> 
> Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a "will cease to work" case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction "Python broke my app".

Supply a SSL vert using the environment variable?

> 
> Summary - I'm +1 as long as either the "will cease to work" list is empty, or we have a *very* good story for (legitimate) sites which do cease to work.
> 
> Paul.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/d1a815db/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/d1a815db/attachment.pgp>

From donald at stufft.io  Mon Jun  3 23:47:31 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 17:47:31 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603234119.7cdeed4b@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<20130603234119.7cdeed4b@fsol>
Message-ID: <E2627BC6-899F-4CD3-9FB4-626FC469D78D@stufft.io>


On Jun 3, 2013, at 5:41 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 3 Jun 2013 22:31:40 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
>> 
>>> Some legit sites with proper
>>> certificates still manage to muck something up administratively
>>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
>>> bundle the intermediates, and I've told their devs about it, but all I
>>> can do is disable cert checking). This will break code in ways that
>>> will surprise people greatly. But I'd still rather the default be
>>> True.
>>> 
>> 
>> I'm happy if the "will cease to work" clause only says "some sites with
>> broken security configurations may stop working" with a clear explanation
>> that it is *their* fault, not Python's. I'd also expect that the same sites
>> would fail in browsers - if not, we should also be able to make them work
>> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem
>> with my site, why does Python?").
> 
> Keep in mind that not every HTTPS service is a Web site that is meant
> to be readable with a browser. Some are Web services, possibly internal,
> possibly without a domain name (and, therefore, probably a non-matching
> certificate subject name).

They should need to explicitly opt in to disabling the checks that allow that to work.

> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/7e2483be/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/7e2483be/attachment.pgp>

From solipsis at pitrou.net  Mon Jun  3 23:51:30 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 Jun 2013 23:51:30 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <E2627BC6-899F-4CD3-9FB4-626FC469D78D@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<20130603234119.7cdeed4b@fsol>
	<E2627BC6-899F-4CD3-9FB4-626FC469D78D@stufft.io>
Message-ID: <20130603235130.3a65fd02@fsol>

On Mon, 3 Jun 2013 17:47:31 -0400
Donald Stufft <donald at stufft.io> wrote:
> 
> On Jun 3, 2013, at 5:41 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > On Mon, 3 Jun 2013 22:31:40 +0100
> > Paul Moore <p.f.moore at gmail.com> wrote:
> >> 
> >>> Some legit sites with proper
> >>> certificates still manage to muck something up administratively
> >>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
> >>> bundle the intermediates, and I've told their devs about it, but all I
> >>> can do is disable cert checking). This will break code in ways that
> >>> will surprise people greatly. But I'd still rather the default be
> >>> True.
> >>> 
> >> 
> >> I'm happy if the "will cease to work" clause only says "some sites with
> >> broken security configurations may stop working" with a clear explanation
> >> that it is *their* fault, not Python's. I'd also expect that the same sites
> >> would fail in browsers - if not, we should also be able to make them work
> >> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem
> >> with my site, why does Python?").
> > 
> > Keep in mind that not every HTTPS service is a Web site that is meant
> > to be readable with a browser. Some are Web services, possibly internal,
> > possibly without a domain name (and, therefore, probably a non-matching
> > certificate subject name).
> 
> They should need to explicitly opt in to disabling the checks that allow that to work.

Obviously, which means compatibility is broken with existing code.

Regards

Antoine.

From donald at stufft.io  Mon Jun  3 23:56:48 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 17:56:48 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <20130603235130.3a65fd02@fsol>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<20130603234119.7cdeed4b@fsol>
	<E2627BC6-899F-4CD3-9FB4-626FC469D78D@stufft.io>
	<20130603235130.3a65fd02@fsol>
Message-ID: <ED63B9FB-A8CB-44E4-A633-7D27CC03C1BF@stufft.io>


On Jun 3, 2013, at 5:51 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 3 Jun 2013 17:47:31 -0400
> Donald Stufft <donald at stufft.io> wrote:
>> 
>> On Jun 3, 2013, at 5:41 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> 
>>> On Mon, 3 Jun 2013 22:31:40 +0100
>>> Paul Moore <p.f.moore at gmail.com> wrote:
>>>> 
>>>>> Some legit sites with proper
>>>>> certificates still manage to muck something up administratively
>>>>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't
>>>>> bundle the intermediates, and I've told their devs about it, but all I
>>>>> can do is disable cert checking). This will break code in ways that
>>>>> will surprise people greatly. But I'd still rather the default be
>>>>> True.
>>>>> 
>>>> 
>>>> I'm happy if the "will cease to work" clause only says "some sites with
>>>> broken security configurations may stop working" with a clear explanation
>>>> that it is *their* fault, not Python's. I'd also expect that the same sites
>>>> would fail in browsers - if not, we should also be able to make them work
>>>> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem
>>>> with my site, why does Python?").
>>> 
>>> Keep in mind that not every HTTPS service is a Web site that is meant
>>> to be readable with a browser. Some are Web services, possibly internal,
>>> possibly without a domain name (and, therefore, probably a non-matching
>>> certificate subject name).
>> 
>> They should need to explicitly opt in to disabling the checks that allow that to work.
> 
> Obviously, which means compatibility is broken with existing code.
> 
> Regards
> 
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


Yes in that case compat will be broken and they'll need to either specify a cert that can be used to validate the connection or disable the protection. I think it's very surprising for people that they need to *enable* secure mode when most tools have that on by default. It's handing users a security foot gun, and like most things security related "broken" is silent until it's too late.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/b2465bd5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/b2465bd5/attachment.pgp>

From p.f.moore at gmail.com  Tue Jun  4 00:01:22 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 3 Jun 2013 23:01:22 +0100
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <EE38CDBB-327D-4DB5-83AF-0BCAD81B586F@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<EE38CDBB-327D-4DB5-83AF-0BCAD81B586F@stufft.io>
Message-ID: <CACac1F_vJ0_QkcnsDfXpQ_HSe4K7RxO2aGtKBJjPaKcrkEbBJQ@mail.gmail.com>

On 3 June 2013 22:46, Donald Stufft <donald at stufft.io> wrote:

> Also, we should consider the issue for application users. Suppose I'm
> using a Python application that downloads something from the web. I upgrade
> to 3.4, and the app stops working because of a "will cease to work" case.
> As an end user, how can I get the app working again? Having to patch the
> sources isn't an option, and reverting to 3.3 provokes the reaction "Python
> broke my app".
>
>
> Supply a SSL vert using the environment variable?
>

Hmm, that would be acceptable, I guess, for many users (although Windows
users are somewhat more environment-variable-averse than Unix users). But
you say that as if it's obvious how to do that (or where to get a cert).
It's certainly not obvious to me, and if "it works in Internet Explorer",
I'd have no idea where to get a cert from that I could use in an
environment variable.

Just to repeat - I agree with the principle, but in many environments,
users are pretty much clueless about security and actively object to being
educated "for their own safety". These users will disable all security
quite happily if it stops the internal app failing, and will blame Python
for "making things harder" and breaking backward compatibility. On the
other hand, I suspect we're talking about an extremely low percentage of
cases, so let's not blow the issue out of proportion :-)

Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/2474bd83/attachment-0001.html>

From donald at stufft.io  Tue Jun  4 01:15:36 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 3 Jun 2013 19:15:36 -0400
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <CACac1F_vJ0_QkcnsDfXpQ_HSe4K7RxO2aGtKBJjPaKcrkEbBJQ@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<EE38CDBB-327D-4DB5-83AF-0BCAD81B586F@stufft.io>
	<CACac1F_vJ0_QkcnsDfXpQ_HSe4K7RxO2aGtKBJjPaKcrkEbBJQ@mail.gmail.com>
Message-ID: <2593421E-0413-492C-8C28-A3982B7B960E@stufft.io>


On Jun 3, 2013, at 6:01 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> 
> On 3 June 2013 22:46, Donald Stufft <donald at stufft.io> wrote:
>> Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a "will cease to work" case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction "Python broke my app".
> 
> Supply a SSL vert using the environment variable?
> 
> Hmm, that would be acceptable, I guess, for many users (although Windows users are somewhat more environment-variable-averse than Unix users). But you say that as if it's obvious how to do that (or where to get a cert). It's certainly not obvious to me, and if "it works in Internet Explorer", I'd have no idea where to get a cert from that I could use in an environment variable.
> 
> Just to repeat - I agree with the principle, but in many environments, users are pretty much clueless about security and actively object to being educated "for their own safety". These users will disable all security quite happily if it stops the internal app failing, and will blame Python for "making things harder" and breaking backward compatibility. On the other hand, I suspect we're talking about an extremely low percentage of cases, so let's not blow the issue out of proportion :-)
> 
> Paul

Let me make myself a bit more clear because maybe I haven't been.

If someone is relying on the insecure by default behavior and Python 3.4 gets this change, their code will break and their options will be:

    Client side:
        - Disable Verification, essentially reverting back to the old behavior.
        - Setup their Python install to be able to validate that certificate
           - This may be importing it into their OS certificate store
           - This may be using an env var to specify
           - This may be passing it directly somewhere

    Server side:
        - Make the certificate valid
          - Purchase a Valid by default certificate
          - Serve immedaries etc.


Generally any of these will be completely valid options, even disabling the checks. The idea behind my proposal is that people generally only use TLS for a reason and that reason is they want to protect against the kinds of attacks that TLS protects against. You really only get the bulk of those protections if you validate the certificates. So for the vast bulk of people validation is the option they want. It also happens that validation on by default is "secure" by default which aims a fairly large foot gun away from peoples feet.

To expand on something else, I'm completely willing to do whatever I can to hep make this happen. I don't really know what the process is. If I need to write a PEP I'll do that. If I need to volunteer to manage the certificates, I'll do that. If we need docs or tooling to help the people who this change might break I'll do that too.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/0d4a9e92/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/0d4a9e92/attachment.pgp>

From fijall at gmail.com  Tue Jun  4 03:56:55 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 4 Jun 2013 09:56:55 +0800
Subject: [Python-Dev] PEP 442: Safe object finalization
In-Reply-To: <20130518163315.6a21a0cd@fsol>
References: <20130518105910.1eecfa5f@fsol>
	<CAMSv6X39grvMWsAEkXBUdwHdYr-5y6Mwt_MdFdN3KjCs-EC3_w@mail.gmail.com>
	<20130518154552.2a879bf6@fsol>
	<CAMSv6X1J9JaKsLJBhMU+nZOa+M0eV8ZQawWe9iEE_eL0bhmMVw@mail.gmail.com>
	<20130518163315.6a21a0cd@fsol>
Message-ID: <CAK5idxRtNReB1C=RECfaGbZV5GQM2s3yUK2N_qNuu=rKVOtztA@mail.gmail.com>

On Sat, May 18, 2013 at 10:33 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 18 May 2013 16:22:55 +0200
> Armin Rigo <arigo at tunes.org> wrote:
>> Hi Antoine,
>>
>> On Sat, May 18, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> >> How is this done?  I don't see a clear way to determine it by looking
>> >> only at the objects in the CI, given that arbitrary modifications of
>> >> the object graph may have occurred.
>> >
>> > The same way a generation is traversed, but restricted to the CI.
>> >
>> > First the gc_refs field of each CI object is initialized to its
>> > ob_refcnt (again).
>> >
>> > Then, tp_traverse is called on each CI object, and each visited
>> > CI object has its gc_refs decremented. This substracts CI-internal
>> > references from the gc_refs fields.
>> >
>> > At the end of the traversal, if all CI objects have their gc_refs equal
>> > to 0, then the CI has no external reference to it and can be cleared.
>> > If at least one CI object has non-zero gc_refs, the CI cannot be
>> > cleared.
>>
>> Ok, indeed.  Then you really should call finalizers only once: in case
>> one of the finalizers in a cycle did a trivial change like I
>> described, the algorithm above will conservatively assume the cycle
>> should be kept alive.  At the next GC collection we must not call the
>> finalizer again, because it's likely to just do a similar trivial
>> change.
>
> Well, the finalizer will only be called if the resurrected object is
> dereferenced again; otherwise the object won't be considered by the GC.
> So, this will only happen if someone keeps trying to destroy a
> resurrected object.
>
> Calling finalizers only once is fine with me, but it would be a change
> in behaviour; I don't know if it may break existing code.
>
> (for example, say someone is using __del__ to manage a freelist)
>
> Regards
>
> Antoine.

PyPy already ever calls finalizers once. If you resurrect an object,
it'll be alive, but it's finalizer will not be called again. We
discussed a few changes a while ago and we decided (I think even
debated on python-dev) than even such behavior is correct:

* you have a reference cycle A <-> B, C references A. C references itself.

* you collect the stuff. We do topological order, so C finalizer is
called first (they're only undefined inside a cycle)

* then A and B finalizers are called in undefined order, even if C
finalizer would resurrect C.

* no more finalizers for those objects are called

I'm not sure if it's cool for CPython or not to do such changes

Cheers,
fijal

From tanqing.zjj at taobao.com  Tue Jun  4 04:43:59 2013
From: tanqing.zjj at taobao.com (=?gb2312?B?zL3H5w==?=)
Date: Tue, 4 Jun 2013 10:43:59 +0800
Subject: [Python-Dev]  Let's get rid of unbound methods
Message-ID: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com>

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130604/c09e13ee/attachment.html>

From raymond.hettinger at gmail.com  Tue Jun  4 06:44:05 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 3 Jun 2013 21:44:05 -0700
Subject: [Python-Dev] Putting the Mac Build in the Apple App Store
Message-ID: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>

Does anyone know what we would need to do to get Python in the Apple application store as a free App?

The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store).   A number of my students are having difficulty getting around it will help.

If we were in the app store, installation and upgrade would be a piece of cake.


Raymond



From fijall at gmail.com  Tue Jun  4 08:00:45 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 4 Jun 2013 14:00:45 +0800
Subject: [Python-Dev] Putting the Mac Build in the Apple App Store
In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
Message-ID: <CAK5idxQ3z6OGeSAbKmtogW91TMk_ptdQLO3agUgRg8x201ChCw@mail.gmail.com>

On Tue, Jun 4, 2013 at 12:44 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Does anyone know what we would need to do to get Python in the Apple application store as a free App?
>
> The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store).   A number of my students are having difficulty getting around it will help.
>
> If we were in the app store, installation and upgrade would be a piece of cake.
>
>
> Raymond
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

doesn't it break apple's rules? generally you should not be able to
download content from the internet and run it (like say javascript in
the browser). I suppose python interpreter counts as that but hey, who
knows.

cheers,
fijal

From clp2 at rebertia.com  Tue Jun  4 08:49:02 2013
From: clp2 at rebertia.com (Chris Rebert)
Date: Mon, 3 Jun 2013 23:49:02 -0700
Subject: [Python-Dev] Putting the Mac Build in the Apple App Store
In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
Message-ID: <CAMZYqRS2dLJgT-jgzCMTmvEmDtsMdYf7ygaoz-LN+u0dzYHe0g@mail.gmail.com>

On Jun 3, 2013 9:45 PM, "Raymond Hettinger" <raymond.hettinger at gmail.com>
wrote:
>
> Does anyone know what we would need to do to get Python in the Apple
application store as a free App?
>
> The default security settings on OS X 10.8 block the installation of the
DMG (or any software downloaded outside the app store).   A number of my
students are having difficulty getting around it will help.
>
> If we were in the app store, installation and upgrade would be a piece of
cake.

If you need another option, you could try installing Python via Homebrew (
http://brew.sh ). Homebrew itself is also easy to install.
(Although admittedly I haven't tried Homebrew's Python, but my experience
with their other packages has been good.)

Cheers,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130603/243c7531/attachment.html>

From rosuav at gmail.com  Tue Jun  4 09:55:03 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 4 Jun 2013 17:55:03 +1000
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
 Bundle in CPython)
In-Reply-To: <2593421E-0413-492C-8C28-A3982B7B960E@stufft.io>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<20130603123636.7a44dd9c@anarchist>
	<B18CAD66-9BE1-4203-A0D7-D30175EC98DA@stufft.io>
	<20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us>
	<CAPTjJmqCAvHZu19m4dhjcJdB86Md23zwGM=3GDUidu6kxWBaBw@mail.gmail.com>
	<CACac1F8vpR-XvmYFngJafOojOLvekXEENXKbyN_t=FTy29NdkQ@mail.gmail.com>
	<EE38CDBB-327D-4DB5-83AF-0BCAD81B586F@stufft.io>
	<CACac1F_vJ0_QkcnsDfXpQ_HSe4K7RxO2aGtKBJjPaKcrkEbBJQ@mail.gmail.com>
	<2593421E-0413-492C-8C28-A3982B7B960E@stufft.io>
Message-ID: <CAPTjJmpz7VmnGnPD0Pnn3qUcvE07jhtH0hUVvbQCFBH9pvPHhg@mail.gmail.com>

On Tue, Jun 4, 2013 at 9:15 AM, Donald Stufft <donald at stufft.io> wrote:
> Generally any of these will be completely valid options, even disabling the
> checks. The idea behind my proposal is that people generally only use TLS
> for a reason and that reason is they want to protect against the kinds of
> attacks that TLS protects against. You really only get the bulk of those
> protections if you validate the certificates. So for the vast bulk of people
> validation is the option they want. It also happens that validation on by
> default is "secure" by default which aims a fairly large foot gun away from
> peoples feet.

Yep. I absolutely agree, and if Python had been like this ever since
3.0, there would be no issue whatsoever. The only problem is that it's
going to trip up people who have otherwise-working code that gets
broken by a change of Python version. The problem isn't even in their
code, it's in the remote end. I support making the change, but all of
the solutions you propose require being aware of the problem. Somehow
people will need to be told, preferably before it breaks something,
that the default has changed.

ChrisA

From nad at acm.org  Tue Jun  4 10:19:36 2013
From: nad at acm.org (Ned Deily)
Date: Tue, 04 Jun 2013 01:19:36 -0700
Subject: [Python-Dev] Putting the Mac Build in the Apple App Store
References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
Message-ID: <nad-B835C9.01193604062013@news.gmane.org>

In article <770E1B33-4174-4880-9DA1-4E15E1818808 at gmail.com>,
 Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

> Does anyone know what we would need to do to get Python in the Apple 
> application store as a free App?
> 
> The default security settings on OS X 10.8 block the installation of the DMG 
> (or any software downloaded outside the app store).   A number of my students 
> are having difficulty getting around it will help.

The ReadMe file included in the current Python OS X installer DMGs 
explains exactly how to get around the problem:

"If you are attempting to install on an OS X 10.8 system, you may
see a message that Python can't be installed because it is from an
unidentified developer.  This is because this Python installer
package is not yet compatible with the Gatekeeper security feature
introduced in OS X 10.8.  To allow Python to be installed, you
can override the Gatekeeper policy for this install.  In the Finder,
instead of double-clicking, control-click or right click the "Python"
installer package icon.  Then select "Open using ... Installer" from
the contextual menu that appears."

There are a number of other installers that haven't been updated yet for 
Gatekeeper, including the currant ActiveState Tcl installers.  The same 
technique works for all of them.  This is basic knowledge needed to use 
OS X 10.8.  Even so, the Python installer will get updated.
 
> If we were in the app store, installation and upgrade would be a piece of 
> cake.

Sorry but, under the current Apple app store rules, Python would never 
be accepted.  Among other things, it would need to be "sandboxed" 
meaning it could only use a very limited set of OS APIs and would have 
very limited access to the user's file system.  Such a Python would not 
be very usable.

-- 
 Ned Deily,
 nad at acm.org


From ronaldoussoren at mac.com  Tue Jun  4 12:15:38 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 4 Jun 2013 12:15:38 +0200
Subject: [Python-Dev] Putting the Mac Build in the Apple App Store
In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com>
Message-ID: <50E1F5FD-D245-4998-9B48-3415A19B38FA@mac.com>


On 4 Jun, 2013, at 6:44, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

> Does anyone know what we would need to do to get Python in the Apple application store as a free App?
> 
> The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store).   A number of my students are having difficulty getting around it will help.
> 
> If we were in the app store, installation and upgrade would be a piece of cake.

A problem with the app store is that the Python installation should then be an app (for example IDLE.app), and that the application must be sandboxed. The latter is showstopper, as scripts run with the interpreter would be sandboxed as well and hence couldn't access most of the system.

A better solution for the problem with OSX 10.8's security settings it sign the installer with a developer ID. It can then be opened by double clicking because the app is provided by an "identified developer".  A problem with signing the installer is that this requires changes to the installer, we're currently using an ancient installer format that cannot be signed. That should be changed some time in the future anyway and signing the installer could be a good reason to work on that.

BTW. There is a workaround that makes it possible to install without signing the installer: right-click on the installer and select "open" (instead of double clicking the installer). The system will then give a scary warning, but will allow installation anyway.

Ronald
> 
> 
> Raymond
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From ronaldoussoren at mac.com  Tue Jun  4 12:20:35 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 4 Jun 2013 12:20:35 +0200
Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert
	Bundle in CPython)
In-Reply-To: <CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
References: <F12C493D-12E1-48A1-BA7B-E2971455C9C0@stufft.io>
	<CAPZV6o8bBmnk95LCsRGP69jEzmfLGGrGk2vhxOxFL+XemAnSuA@mail.gmail.com>
Message-ID: <25E65E5D-B8A2-4D7C-B866-AFC4CEF915BD@mac.com>


On 3 Jun, 2013, at 7:58, Benjamin Peterson <benjamin at python.org> wrote:

> 2013/6/2 Donald Stufft <donald at stufft.io>:
>> As of right now, as far as I can tell, Python does not validate HTTPS
>> certificates by default. As far as I can tell this is because there is no
>> guaranteed certificates available.
>> 
>> So I would like to propose that CPython adopt the Mozilla SSL certificate
>> list and include it in core, and switch over the API's so that they verify
>> HTTPS by default.
> 
> +1
> 
>> 
>> Ideally this would take the shape of attempting to locate the system
>> certificate store if possible, and if that doesn't work falling back to the
>> bundled certificates. That way the various Linux distros can easily have
>> their copies of Python depend soley on their built in certs, but Windows,
>> OSX, Source compiles etc will all still have a fallback value.
> 
> My preference would be actually be for the included certificates file
> to be used by default. This would provide a consistent experience
> across platforms. We could provide options to look for system cert
> repositories if desired.

I'd prefer to use the system CA list when that's available. I've had to hunt down the CA list for a number of application when a custom CA for internal use and that's not fun, using the system list is much friendlier to users. 

Ronald


From steve at pearwood.info  Tue Jun  4 13:57:44 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 04 Jun 2013 21:57:44 +1000
Subject: [Python-Dev] Let's get rid of unbound methods
In-Reply-To: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com>
References: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com>
Message-ID: <51ADD638.9020001@pearwood.info>

On 04/06/13 12:43, ?? wrote:
<nothing>


Your email appears to be blank, except for a large chunk of HTML code. Did you have something specific to say other than the subject line?


As for unbound methods, Guido's time machine strikes again. They're already gone in Python 3.


py> class X:
...     def spam(self):
...             pass
...
py> X.spam
<function spam at 0xb7bd2dac>
py> X().spam
<bound method X.spam of <__main__.X object at 0xb7bd0e0c>>



-- 
Steven

From secalert at redhat.com  Tue Jun  4 13:44:08 2013
From: secalert at redhat.com (Red Hat Security Response Team)
Date: Tue, 4 Jun 2013 07:44:08 -0400
Subject: [Python-Dev] [engineering.redhat.com #216557] [Fwd: Validating SSL
	By Default (aka Including a Cert Bundle in CPython)]
In-Reply-To: <1370289371.22768.17.camel@surprise>
References: <RT-Ticket-216557@engineering.redhat.com>
	<1370289371.22768.17.camel@surprise>
Message-ID: <rt-4.0.8-402-1370346248-1861.216557-6-0@engineering.redhat.com>

Hello David, Donald,

  David, thank you for sharing these intentions with us.

On Mon Jun 03 15:56:09 2013, dmalcolm at redhat.com wrote:
> 
> As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this
> is because there is no guaranteed certificates available.
>
> So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over
> the API's so that they verify HTTPS by default.

Donald, we would only welcome this enhancement / proposal. To mention
some examples - urllib2 and httplib modules:
  http://docs.python.org/2/library/urllib2.html
  http://docs.python.org/2/library/httplib.html

are documented upstream not to perform SSL certificate verification by default
(and due this fact there has been couple of CVE identifiers assigned in the past
for applications that incorrectly assumed certificates would be validated when
using these modules).

So any enhancement, which can upstream done in this area, would be only welcome.

> This is what most people are going to expect when using a https url 
> (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does").
> 
> Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't
> work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python
> depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value.

AFAWCT that proposal looks reasonable.

Thank you && Regards, Jan.
--
Jan iankko Lieskovsky / Red Hat Security Response Team

>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 

From carlosnepomuceno at outlook.com  Wed Jun  5 01:53:21 2013
From: carlosnepomuceno at outlook.com (Carlos Nepomuceno)
Date: Wed, 5 Jun 2013 02:53:21 +0300
Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)?
Message-ID: <BLU176-W1F35788C5EEAC4B30AB73D79E0@phx.gbl>

Do you consider Python a 4GL? Why (not)?
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/fc2ec887/attachment.html>

From drsalists at gmail.com  Wed Jun  5 03:17:33 2013
From: drsalists at gmail.com (Dan Stromberg)
Date: Tue, 4 Jun 2013 18:17:33 -0700
Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)?
In-Reply-To: <BLU176-W1F35788C5EEAC4B30AB73D79E0@phx.gbl>
References: <BLU176-W1F35788C5EEAC4B30AB73D79E0@phx.gbl>
Message-ID: <CAGGBd_rihU4iD2Cx75Ax0u-f6M+CUHQYcznzOTyYYR386hAQ-w@mail.gmail.com>

On Tue, Jun 4, 2013 at 4:53 PM, Carlos Nepomuceno <
carlosnepomuceno at outlook.com> wrote:

> Do you consider Python a 4GL? Why (not)?
>

By the wikipedia definition of 4GL and 5GL, I'd say Python is neither.  And
it's not a VHLL either, again according to the wikipedia definition.  But
IMO it is too high level to be a traditional 3GL too.

Perhaps "Scripting language" is the best general category we have that
Python fits into.  But I hope not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130604/63195ce5/attachment.html>

From rosuav at gmail.com  Wed Jun  5 03:21:50 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 5 Jun 2013 11:21:50 +1000
Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)?
In-Reply-To: <CAGGBd_rihU4iD2Cx75Ax0u-f6M+CUHQYcznzOTyYYR386hAQ-w@mail.gmail.com>
References: <BLU176-W1F35788C5EEAC4B30AB73D79E0@phx.gbl>
	<CAGGBd_rihU4iD2Cx75Ax0u-f6M+CUHQYcznzOTyYYR386hAQ-w@mail.gmail.com>
Message-ID: <CAPTjJmqpCpUBrxajvofNJqHpr=ptABsNCb2uHe6JiLRzz08gdw@mail.gmail.com>

On Wed, Jun 5, 2013 at 11:17 AM, Dan Stromberg <drsalists at gmail.com> wrote:
>
> On Tue, Jun 4, 2013 at 4:53 PM, Carlos Nepomuceno
> <carlosnepomuceno at outlook.com> wrote:
>>
>> Do you consider Python a 4GL? Why (not)?
>
>
> By the wikipedia definition of 4GL and 5GL, I'd say Python is neither.  And
> it's not a VHLL either, again according to the wikipedia definition.  But
> IMO it is too high level to be a traditional 3GL too.
>
> Perhaps "Scripting language" is the best general category we have that
> Python fits into.  But I hope not.

I think this would be a fun discussion for python-list - just what IS Python?

Here's some stuff on programming vs scripting:
http://www.perl.com/pub/2007/12/06/soto-11.html

(including an awesome line about actors and audience)

ChrisA

From guido at python.org  Wed Jun  5 03:32:18 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jun 2013 18:32:18 -0700
Subject: [Python-Dev] PEP 443 Accepted
Message-ID: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>

?ukasz,

Congratulations! I've accepted PEP 443. I've already marked it as
Accepted in the repo. I've also applied some very minor edits in order
to make the text flow a little better in a few places. I think this is
a great PEP -- it's simple, doesn't overreach, and you've managed the
bikeshedding admirably. Thank you for your great contribution to
Python!

-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Wed Jun  5 03:47:21 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 05 Jun 2013 02:47:21 +0100
Subject: [Python-Dev] PyPI upload error
Message-ID: <51AE98A9.60902@mrabarnett.plus.com>

I've just tried to upload to PyPI using setup.py and got this error:

     Upload failed (503): backend write error

Can anyone tell me what it means?

From ethan at stoneleaf.us  Wed Jun  5 03:54:12 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 04 Jun 2013 18:54:12 -0700
Subject: [Python-Dev] PEP 443 Accepted
In-Reply-To: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
References: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
Message-ID: <51AE9A44.70209@stoneleaf.us>

Congratulations, ?ukasz!

--
~Ethan~

From donald at stufft.io  Wed Jun  5 04:07:50 2013
From: donald at stufft.io (Donald Stufft)
Date: Tue, 4 Jun 2013 22:07:50 -0400
Subject: [Python-Dev] PyPI upload error
In-Reply-To: <51AE98A9.60902@mrabarnett.plus.com>
References: <51AE98A9.60902@mrabarnett.plus.com>
Message-ID: <2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io>


On Jun 4, 2013, at 9:47 PM, MRAB <python at mrabarnett.plus.com> wrote:

> I've just tried to upload to PyPI using setup.py and got this error:
> 
>    Upload failed (503): backend write error
> 
> Can anyone tell me what it means?
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

This is probably more appropriate for distutils-sig, but does it happen every time? or did it just happen once?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130604/d359dfe6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130604/d359dfe6/attachment.pgp>

From python at mrabarnett.plus.com  Wed Jun  5 04:25:36 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 05 Jun 2013 03:25:36 +0100
Subject: [Python-Dev] PyPI upload error
In-Reply-To: <2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io>
References: <51AE98A9.60902@mrabarnett.plus.com>
	<2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io>
Message-ID: <51AEA1A0.3020302@mrabarnett.plus.com>

On 05/06/2013 03:07, Donald Stufft wrote:
>
> On Jun 4, 2013, at 9:47 PM, MRAB <python at mrabarnett.plus.com
> <mailto:python at mrabarnett.plus.com>> wrote:
>
>> I've just tried to upload to PyPI using setup.py and got this error:
>>
>>    Upload failed (503): backend write error
>>
>> Can anyone tell me what it means?
>>
> This is probably more appropriate for distutils-sig, but does it happen
> every time? or did it just happen once?
>
It happened a couple of times, but worked some time later.

I'll try to remember to ask distutils-sig next time.

From p.f.moore at gmail.com  Wed Jun  5 09:07:54 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 5 Jun 2013 08:07:54 +0100
Subject: [Python-Dev] PEP 443 Accepted
In-Reply-To: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
References: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
Message-ID: <CACac1F-xTfkGDa7FBdG96P=cD-CUaeXtYwCgte=bjQ_VveZHAw@mail.gmail.com>

On 5 June 2013 02:32, Guido van Rossum <guido at python.org> wrote:

> ?ukasz,
>
> Congratulations! I've accepted PEP 443. I've already marked it as
> Accepted in the repo. I've also applied some very minor edits in order
> to make the text flow a little better in a few places. I think this is
> a great PEP -- it's simple, doesn't overreach, and you've managed the
> bikeshedding admirably. Thank you for your great contribution to
> Python!
>

Excellent news! Congratulations.
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/1adaeb24/attachment-0001.html>

From markus at unterwaditzer.net  Wed Jun  5 09:29:14 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Wed, 05 Jun 2013 09:29:14 +0200
Subject: [Python-Dev] PEP 443 Accepted
In-Reply-To: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
References: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
Message-ID: <e202f19d-6cd9-4d41-b8bc-804adec9bb8c@email.android.com>

As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs.

E.g if

    issubclass(Apple, Fruit)

And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere.

-- Markus (from phone)

Guido van Rossum <guido at python.org> wrote:
>?ukasz,
>
>Congratulations! I've accepted PEP 443. I've already marked it as
>Accepted in the repo. I've also applied some very minor edits in order
>to make the text flow a little better in a few places. I think this is
>a great PEP -- it's simple, doesn't overreach, and you've managed the
>bikeshedding admirably. Thank you for your great contribution to
>Python!
>
>-- 
>--Guido van Rossum (python.org/~guido)
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net


From ncoghlan at gmail.com  Wed Jun  5 09:47:35 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 5 Jun 2013 17:47:35 +1000
Subject: [Python-Dev] PEP 443 Accepted
In-Reply-To: <e202f19d-6cd9-4d41-b8bc-804adec9bb8c@email.android.com>
References: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
	<e202f19d-6cd9-4d41-b8bc-804adec9bb8c@email.android.com>
Message-ID: <CADiSq7ey8uOabugRYGAZwOvY9igKgA-Qbzn-YnDvtxm6mJAWyQ@mail.gmail.com>

On Wed, Jun 5, 2013 at 5:29 PM, Markus Unterwaditzer
<markus at unterwaditzer.net> wrote:
> As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs.
>
> E.g if
>
>     issubclass(Apple, Fruit)
>
> And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere.

That's covered by walking the MRO the same way method dispatch does
it, so it didn't really get discussed much (the question never came
up). The ABC handling is described explicitly as the generic dispatch
implementation that the PEP was based on didn't handle them, and
because it required a fair bit more thought than just walking the MRO.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From lukasz at langa.pl  Wed Jun  5 09:56:08 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 5 Jun 2013 09:56:08 +0200
Subject: [Python-Dev] PEP 443 Accepted
In-Reply-To: <e202f19d-6cd9-4d41-b8bc-804adec9bb8c@email.android.com>
References: <CAP7+vJKp1k3dUf4f=3LX6Rq-VSTUS7Ld9ATJxYcp0efQnTikQw@mail.gmail.com>
	<e202f19d-6cd9-4d41-b8bc-804adec9bb8c@email.android.com>
Message-ID: <F94D0E9D-4671-4A08-82B5-F2021169AE0C@langa.pl>

On 5 cze 2013, at 09:29, Markus Unterwaditzer <markus at unterwaditzer.net> wrote:

> As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs.
> 
> E.g if
> 
>    issubclass(Apple, Fruit)
> 
> And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere.

Yes, this is the supported behaviour. The PEP briefly explains that "Where there is no registered implementation for a specific type, its method resolution order is used to find a more generic implementation."

As Nick noted, the reason the description is so brief is that this works just like calling methods.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/d954bbea/attachment.html>

From koobs.freebsd at gmail.com  Wed Jun  5 11:22:44 2013
From: koobs.freebsd at gmail.com (Kubilay Kocak)
Date: Wed, 05 Jun 2013 19:22:44 +1000
Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot
In-Reply-To: <51AAD44B.4090802@FreeBSD.org>
References: <51AAD44B.4090802@FreeBSD.org>
Message-ID: <51AF0364.7060504@FreeBSD.org>

On 2/06/2013 3:12 PM, Kubilay Kocak wrote:
> Afternoon (UTC+10),
> 
> I'd like to request a new user/pass for a FreeBSD 10.0-CURRENT VM guest
> I've just setup as a dedicated buildbot slave to complement my other
> koobs-freebsd slaves.
> 
> Also, with this and future additions to the FreeBSD buildslave set in
> mind, I think it might also be prudent for a rename to take place:
> 
> koobs-freebsd9-amd64
> koobs-freebsd9-amd64-clang (CC=clang)
> koobs-freebsd10-amd64 (clang is default here)
> 
> Convention being: koobs-freebsdX[Y]-arch[-config] (Happy for feedback here)
> 
> If there are any permutations or additions you'd like to specifically
> see for FreeBSD to increase coverage just let me know (Hint: I have a
> PandaBoard arm board here i'm getting up and running)
> 
> I have ZFS & DTrace to work with among other things, and the long term
> plan is to have FreeBSD buildbots running the full gamut of versions,
> from -RELEASE through -STABLE to HEAD branches.
> 
> I'm on #python-dev IRC if someone needs to discuss.
> 
> --
> Regards,
> 
> Koobs
> 

Ping :)

From brett at python.org  Wed Jun  5 16:31:04 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jun 2013 10:31:04 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
	implementation for PEP 443
In-Reply-To: <3bQQxs6YX7zRbZ@mail.python.org>
References: <3bQQxs6YX7zRbZ@mail.python.org>
Message-ID: <CAP1=2W5NNM0V-98FMaXKErdLwtvU-6_mJcKWJ_jm2vnb3BYBrA@mail.gmail.com>

Any chance you could move your definitions for "generic function" and
"single dispatch" to the glossary and just link to them here?


On Wed, Jun 5, 2013 at 6:20 AM, lukasz.langa <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/dfcb64f51f7b
> changeset:   84039:dfcb64f51f7b
> user:        ?ukasz Langa <lukasz at langa.pl>
> date:        Wed Jun 05 12:20:24 2013 +0200
> summary:
>   Add reference implementation for PEP 443
>
> PEP accepted:
> http://mail.python.org/pipermail/python-dev/2013-June/126734.html
>
> files:
>   Doc/library/functools.rst  |  110 +++++++
>   Lib/functools.py           |  128 ++++++++-
>   Lib/pkgutil.py             |   52 +---
>   Lib/test/test_functools.py |  374 ++++++++++++++++++++++++-
>   Misc/NEWS                  |    3 +
>   Modules/Setup.dist         |    2 +-
>   6 files changed, 614 insertions(+), 55 deletions(-)
>
>
> diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst
> --- a/Doc/library/functools.rst
> +++ b/Doc/library/functools.rst
> @@ -6,6 +6,7 @@
>  .. moduleauthor:: Peter Harris <scav at blueyonder.co.uk>
>  .. moduleauthor:: Raymond Hettinger <python at rcn.com>
>  .. moduleauthor:: Nick Coghlan <ncoghlan at gmail.com>
> +.. moduleauthor:: ?ukasz Langa <lukasz at langa.pl>
>  .. sectionauthor:: Peter Harris <scav at blueyonder.co.uk>
>
>  **Source code:** :source:`Lib/functools.py`
> @@ -186,6 +187,115 @@
>     *sequence* contains only one item, the first item is returned.
>
>
> +.. decorator:: singledispatch(default)
> +
> +   Transforms a function into a single-dispatch generic function.  A
> **generic
> +   function** is composed of multiple functions implementing the same
> operation
> +   for different types. Which implementation should be used during a call
> is
> +   determined by the dispatch algorithm.  When the implementation is
> chosen
> +   based on the type of a single argument, this is known as **single
> +   dispatch**.
> +
> +   To define a generic function, decorate it with the ``@singledispatch``
> +   decorator. Note that the dispatch happens on the type of the first
> argument,
> +   create your function accordingly::
> +
> +     >>> from functools import singledispatch
> +     >>> @singledispatch
> +     ... def fun(arg, verbose=False):
> +     ...     if verbose:
> +     ...         print("Let me just say,", end=" ")
> +     ...     print(arg)
> +
> +   To add overloaded implementations to the function, use the
> :func:`register`
> +   attribute of the generic function.  It is a decorator, taking a type
> +   parameter and decorating a function implementing the operation for that
> +   type::
> +
> +     >>> @fun.register(int)
> +     ... def _(arg, verbose=False):
> +     ...     if verbose:
> +     ...         print("Strength in numbers, eh?", end=" ")
> +     ...     print(arg)
> +     ...
> +     >>> @fun.register(list)
> +     ... def _(arg, verbose=False):
> +     ...     if verbose:
> +     ...         print("Enumerate this:")
> +     ...     for i, elem in enumerate(arg):
> +     ...         print(i, elem)
> +
> +   To enable registering lambdas and pre-existing functions, the
> +   :func:`register` attribute can be used in a functional form::
> +
> +     >>> def nothing(arg, verbose=False):
> +     ...     print("Nothing.")
> +     ...
> +     >>> fun.register(type(None), nothing)
> +
> +   The :func:`register` attribute returns the undecorated function which
> +   enables decorator stacking, pickling, as well as creating unit tests
> for
> +   each variant independently::
> +
> +     >>> @fun.register(float)
> +     ... @fun.register(Decimal)
> +     ... def fun_num(arg, verbose=False):
> +     ...     if verbose:
> +     ...         print("Half of your number:", end=" ")
> +     ...     print(arg / 2)
> +     ...
> +     >>> fun_num is fun
> +     False
> +
> +   When called, the generic function dispatches on the type of the first
> +   argument::
> +
> +     >>> fun("Hello, world.")
> +     Hello, world.
> +     >>> fun("test.", verbose=True)
> +     Let me just say, test.
> +     >>> fun(42, verbose=True)
> +     Strength in numbers, eh? 42
> +     >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
> +     Enumerate this:
> +     0 spam
> +     1 spam
> +     2 eggs
> +     3 spam
> +     >>> fun(None)
> +     Nothing.
> +     >>> fun(1.23)
> +     0.615
> +
> +   Where there is no registered implementation for a specific type, its
> +   method resolution order is used to find a more generic implementation.
> +   The original function decorated with ``@singledispatch`` is registered
> +   for the base ``object`` type, which means it is used if no better
> +   implementation is found.
> +
> +   To check which implementation will the generic function choose for
> +   a given type, use the ``dispatch()`` attribute::
> +
> +     >>> fun.dispatch(float)
> +     <function fun_num at 0x1035a2840>
> +     >>> fun.dispatch(dict)    # note: default implementation
> +     <function fun at 0x103fe0000>
> +
> +   To access all registered implementations, use the read-only
> ``registry``
> +   attribute::
> +
> +    >>> fun.registry.keys()
> +    dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
> +              <class 'decimal.Decimal'>, <class 'list'>,
> +              <class 'float'>])
> +    >>> fun.registry[float]
> +    <function fun_num at 0x1035a2840>
> +    >>> fun.registry[object]
> +    <function fun at 0x103fe0000>
> +
> +   .. versionadded:: 3.4
> +
> +
>  .. function:: update_wrapper(wrapper, wrapped,
> assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
>
>     Update a *wrapper* function to look like the *wrapped* function. The
> optional
> diff --git a/Lib/functools.py b/Lib/functools.py
> --- a/Lib/functools.py
> +++ b/Lib/functools.py
> @@ -3,19 +3,24 @@
>  # Python module wrapper for _functools C module
>  # to allow utilities written in Python to be added
>  # to the functools module.
> -# Written by Nick Coghlan <ncoghlan at gmail.com>
> -# and Raymond Hettinger <python at rcn.com>
> -#   Copyright (C) 2006-2010 Python Software Foundation.
> +# Written by Nick Coghlan <ncoghlan at gmail.com>,
> +# Raymond Hettinger <python at rcn.com>,
> +# and ?ukasz Langa <lukasz at langa.pl>.
> +#   Copyright (C) 2006-2013 Python Software Foundation.
>  # See C source code for _functools credits/copyright
>
>  __all__ = ['update_wrapper', 'wraps', 'WRAPPER_ASSIGNMENTS',
> 'WRAPPER_UPDATES',
> -           'total_ordering', 'cmp_to_key', 'lru_cache', 'reduce',
> 'partial']
> +           'total_ordering', 'cmp_to_key', 'lru_cache', 'reduce',
> 'partial',
> +           'singledispatch']
>
>  try:
>      from _functools import reduce
>  except ImportError:
>      pass
> +from abc import get_cache_token
>  from collections import namedtuple
> +from types import MappingProxyType
> +from weakref import WeakKeyDictionary
>  try:
>      from _thread import RLock
>  except:
> @@ -354,3 +359,118 @@
>          return update_wrapper(wrapper, user_function)
>
>      return decorating_function
> +
> +
>
> +################################################################################
> +### singledispatch() - single-dispatch generic function decorator
>
> +################################################################################
> +
> +def _compose_mro(cls, haystack):
> +    """Calculates the MRO for a given class `cls`, including relevant
> abstract
> +    base classes from `haystack`.
> +
> +    """
> +    bases = set(cls.__mro__)
> +    mro = list(cls.__mro__)
> +    for needle in haystack:
> +        if (needle in bases or not hasattr(needle, '__mro__')
> +                            or not issubclass(cls, needle)):
> +            continue   # either present in the __mro__ already or
> unrelated
> +        for index, base in enumerate(mro):
> +            if not issubclass(base, needle):
> +                break
> +        if base in bases and not issubclass(needle, base):
> +            # Conflict resolution: put classes present in __mro__ and
> their
> +            # subclasses first. See test_mro_conflicts() in
> test_functools.py
> +            # for examples.
> +            index += 1
> +        mro.insert(index, needle)
> +    return mro
> +
> +def _find_impl(cls, registry):
> +    """Returns the best matching implementation for the given class `cls`
> in
> +    `registry`. Where there is no registered implementation for a specific
> +    type, its method resolution order is used to find a more generic
> +    implementation.
> +
> +    Note: if `registry` does not contain an implementation for the base
> +    `object` type, this function may return None.
> +
> +    """
> +    mro = _compose_mro(cls, registry.keys())
> +    match = None
> +    for t in mro:
> +        if match is not None:
> +            # If `match` is an ABC but there is another unrelated, equally
> +            # matching ABC. Refuse the temptation to guess.
> +            if (t in registry and not issubclass(match, t)
> +                              and match not in cls.__mro__):
> +                raise RuntimeError("Ambiguous dispatch: {} or {}".format(
> +                    match, t))
> +            break
> +        if t in registry:
> +            match = t
> +    return registry.get(match)
> +
> +def singledispatch(func):
> +    """Single-dispatch generic function decorator.
> +
> +    Transforms a function into a generic function, which can have
> different
> +    behaviours depending upon the type of its first argument. The
> decorated
> +    function acts as the default implementation, and additional
> +    implementations can be registered using the 'register()' attribute of
> +    the generic function.
> +
> +    """
> +    registry = {}
> +    dispatch_cache = WeakKeyDictionary()
> +    cache_token = None
> +
> +    def dispatch(typ):
> +        """generic_func.dispatch(type) -> <function implementation>
> +
> +        Runs the dispatch algorithm to return the best available
> implementation
> +        for the given `type` registered on `generic_func`.
> +
> +        """
> +        nonlocal cache_token
> +        if cache_token is not None:
> +            current_token = get_cache_token()
> +            if cache_token != current_token:
> +                dispatch_cache.clear()
> +                cache_token = current_token
> +        try:
> +            impl = dispatch_cache[typ]
> +        except KeyError:
> +            try:
> +                impl = registry[typ]
> +            except KeyError:
> +                impl = _find_impl(typ, registry)
> +            dispatch_cache[typ] = impl
> +        return impl
> +
> +    def register(typ, func=None):
> +        """generic_func.register(type, func) -> func
> +
> +        Registers a new implementation for the given `type` on a
> `generic_func`.
> +
> +        """
> +        nonlocal cache_token
> +        if func is None:
> +            return lambda f: register(typ, f)
> +        registry[typ] = func
> +        if cache_token is None and hasattr(typ, '__abstractmethods__'):
> +            cache_token = get_cache_token()
> +        dispatch_cache.clear()
> +        return func
> +
> +    def wrapper(*args, **kw):
> +        return dispatch(args[0].__class__)(*args, **kw)
> +
> +    registry[object] = func
> +    wrapper.register = register
> +    wrapper.dispatch = dispatch
> +    wrapper.registry = MappingProxyType(registry)
> +    wrapper._clear_cache = dispatch_cache.clear
> +    update_wrapper(wrapper, func)
> +    return wrapper
> diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py
> --- a/Lib/pkgutil.py
> +++ b/Lib/pkgutil.py
> @@ -1,12 +1,13 @@
>  """Utilities to support packages."""
>
> +from functools import singledispatch as simplegeneric
> +import imp
> +import importlib
>  import os
> +import os.path
>  import sys
> -import importlib
> -import imp
> -import os.path
> +from types import ModuleType
>  from warnings import warn
> -from types import ModuleType
>
>  __all__ = [
>      'get_importer', 'iter_importers', 'get_loader', 'find_loader',
> @@ -27,46 +28,6 @@
>      return marshal.load(stream)
>
>
> -def simplegeneric(func):
> -    """Make a trivial single-dispatch generic function"""
> -    registry = {}
> -    def wrapper(*args, **kw):
> -        ob = args[0]
> -        try:
> -            cls = ob.__class__
> -        except AttributeError:
> -            cls = type(ob)
> -        try:
> -            mro = cls.__mro__
> -        except AttributeError:
> -            try:
> -                class cls(cls, object):
> -                    pass
> -                mro = cls.__mro__[1:]
> -            except TypeError:
> -                mro = object,   # must be an ExtensionClass or some such
>  :(
> -        for t in mro:
> -            if t in registry:
> -                return registry[t](*args, **kw)
> -        else:
> -            return func(*args, **kw)
> -    try:
> -        wrapper.__name__ = func.__name__
> -    except (TypeError, AttributeError):
> -        pass    # Python 2.3 doesn't allow functions to be renamed
> -
> -    def register(typ, func=None):
> -        if func is None:
> -            return lambda f: register(typ, f)
> -        registry[typ] = func
> -        return func
> -
> -    wrapper.__dict__ = func.__dict__
> -    wrapper.__doc__ = func.__doc__
> -    wrapper.register = register
> -    return wrapper
> -
> -
>  def walk_packages(path=None, prefix='', onerror=None):
>      """Yields (module_loader, name, ispkg) for all modules recursively
>      on path, or, if path is None, all accessible modules.
> @@ -148,13 +109,12 @@
>                  yield i, name, ispkg
>
>
> -#@simplegeneric
> + at simplegeneric
>  def iter_importer_modules(importer, prefix=''):
>      if not hasattr(importer, 'iter_modules'):
>          return []
>      return importer.iter_modules(prefix)
>
> -iter_importer_modules = simplegeneric(iter_importer_modules)
>
>  # Implement a file walker for the normal importlib path hook
>  def _iter_file_finder_modules(importer, prefix=''):
> diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py
> --- a/Lib/test/test_functools.py
> +++ b/Lib/test/test_functools.py
> @@ -1,24 +1,30 @@
>  import collections
> -import sys
> -import unittest
> -from test import support
> -from weakref import proxy
> +from itertools import permutations
>  import pickle
>  from random import choice
> +import sys
> +from test import support
> +import unittest
> +from weakref import proxy
>
>  import functools
>
>  py_functools = support.import_fresh_module('functools',
> blocked=['_functools'])
>  c_functools = support.import_fresh_module('functools',
> fresh=['_functools'])
>
> +decimal = support.import_fresh_module('decimal', fresh=['_decimal'])
> +
> +
>  def capture(*args, **kw):
>      """capture all positional and keyword arguments"""
>      return args, kw
>
> +
>  def signature(part):
>      """ return the signature of a partial object """
>      return (part.func, part.args, part.keywords, part.__dict__)
>
> +
>  class TestPartial:
>
>      def test_basic_examples(self):
> @@ -138,6 +144,7 @@
>          join = self.partial(''.join)
>          self.assertEqual(join(data), '0123456789')
>
> +
>  @unittest.skipUnless(c_functools, 'requires the C _functools module')
>  class TestPartialC(TestPartial, unittest.TestCase):
>      if c_functools:
> @@ -194,18 +201,22 @@
>                  "new style getargs format but argument is not a tuple",
>                  f.__setstate__, BadSequence())
>
> +
>  class TestPartialPy(TestPartial, unittest.TestCase):
>      partial = staticmethod(py_functools.partial)
>
> +
>  if c_functools:
>      class PartialSubclass(c_functools.partial):
>          pass
>
> +
>  @unittest.skipUnless(c_functools, 'requires the C _functools module')
>  class TestPartialCSubclass(TestPartialC):
>      if c_functools:
>          partial = PartialSubclass
>
> +
>  class TestUpdateWrapper(unittest.TestCase):
>
>      def check_wrapper(self, wrapper, wrapped,
> @@ -312,6 +323,7 @@
>          self.assertTrue(wrapper.__doc__.startswith('max('))
>          self.assertEqual(wrapper.__annotations__, {})
>
> +
>  class TestWraps(TestUpdateWrapper):
>
>      def _default_update(self):
> @@ -372,6 +384,7 @@
>          self.assertEqual(wrapper.attr, 'This is a different test')
>          self.assertEqual(wrapper.dict_attr, f.dict_attr)
>
> +
>  class TestReduce(unittest.TestCase):
>      func = functools.reduce
>
> @@ -452,6 +465,7 @@
>          d = {"one": 1, "two": 2, "three": 3}
>          self.assertEqual(self.func(add, d), "".join(d.keys()))
>
> +
>  class TestCmpToKey:
>
>      def test_cmp_to_key(self):
> @@ -534,14 +548,17 @@
>          self.assertRaises(TypeError, hash, k)
>          self.assertNotIsInstance(k, collections.Hashable)
>
> +
>  @unittest.skipUnless(c_functools, 'requires the C _functools module')
>  class TestCmpToKeyC(TestCmpToKey, unittest.TestCase):
>      if c_functools:
>          cmp_to_key = c_functools.cmp_to_key
>
> +
>  class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase):
>      cmp_to_key = staticmethod(py_functools.cmp_to_key)
>
> +
>  class TestTotalOrdering(unittest.TestCase):
>
>      def test_total_ordering_lt(self):
> @@ -642,6 +659,7 @@
>          with self.assertRaises(TypeError):
>              TestTO(8) <= ()
>
> +
>  class TestLRU(unittest.TestCase):
>
>      def test_lru(self):
> @@ -834,6 +852,353 @@
>                           DoubleEq(2))               # Verify the correct
> return value
>
>
> +class TestSingleDispatch(unittest.TestCase):
> +    def test_simple_overloads(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        def g_int(i):
> +            return "integer"
> +        g.register(int, g_int)
> +        self.assertEqual(g("str"), "base")
> +        self.assertEqual(g(1), "integer")
> +        self.assertEqual(g([1,2,3]), "base")
> +
> +    def test_mro(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        class C:
> +            pass
> +        class D(C):
> +            pass
> +        def g_C(c):
> +            return "C"
> +        g.register(C, g_C)
> +        self.assertEqual(g(C()), "C")
> +        self.assertEqual(g(D()), "C")
> +
> +    def test_classic_classes(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        class C:
> +            pass
> +        class D(C):
> +            pass
> +        def g_C(c):
> +            return "C"
> +        g.register(C, g_C)
> +        self.assertEqual(g(C()), "C")
> +        self.assertEqual(g(D()), "C")
> +
> +    def test_register_decorator(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        @g.register(int)
> +        def g_int(i):
> +            return "int %s" % (i,)
> +        self.assertEqual(g(""), "base")
> +        self.assertEqual(g(12), "int 12")
> +        self.assertIs(g.dispatch(int), g_int)
> +        self.assertIs(g.dispatch(object), g.dispatch(str))
> +        # Note: in the assert above this is not g.
> +        # @singledispatch returns the wrapper.
> +
> +    def test_wrapping_attributes(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            "Simple test"
> +            return "Test"
> +        self.assertEqual(g.__name__, "g")
> +        self.assertEqual(g.__doc__, "Simple test")
> +
> +    @unittest.skipUnless(decimal, 'requires _decimal')
> +    @support.cpython_only
> +    def test_c_classes(self):
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        @g.register(decimal.DecimalException)
> +        def _(obj):
> +            return obj.args
> +        subn = decimal.Subnormal("Exponent < Emin")
> +        rnd = decimal.Rounded("Number got rounded")
> +        self.assertEqual(g(subn), ("Exponent < Emin",))
> +        self.assertEqual(g(rnd), ("Number got rounded",))
> +        @g.register(decimal.Subnormal)
> +        def _(obj):
> +            return "Too small to care."
> +        self.assertEqual(g(subn), "Too small to care.")
> +        self.assertEqual(g(rnd), ("Number got rounded",))
> +
> +    def test_compose_mro(self):
> +        c = collections
> +        mro = functools._compose_mro
> +        bases = [c.Sequence, c.MutableMapping, c.Mapping, c.Set]
> +        for haystack in permutations(bases):
> +            m = mro(dict, haystack)
> +            self.assertEqual(m, [dict, c.MutableMapping, c.Mapping,
> object])
> +        bases = [c.Container, c.Mapping, c.MutableMapping, c.OrderedDict]
> +        for haystack in permutations(bases):
> +            m = mro(c.ChainMap, haystack)
> +            self.assertEqual(m, [c.ChainMap, c.MutableMapping, c.Mapping,
> +                                 c.Sized, c.Iterable, c.Container,
> object])
> +        # Note: The MRO order below depends on haystack ordering.
> +        m = mro(c.defaultdict, [c.Sized, c.Container, str])
> +        self.assertEqual(m, [c.defaultdict, dict, c.Container, c.Sized,
> object])
> +        m = mro(c.defaultdict, [c.Container, c.Sized, str])
> +        self.assertEqual(m, [c.defaultdict, dict, c.Sized, c.Container,
> object])
> +
> +    def test_register_abc(self):
> +        c = collections
> +        d = {"a": "b"}
> +        l = [1, 2, 3]
> +        s = {object(), None}
> +        f = frozenset(s)
> +        t = (1, 2, 3)
> +        @functools.singledispatch
> +        def g(obj):
> +            return "base"
> +        self.assertEqual(g(d), "base")
> +        self.assertEqual(g(l), "base")
> +        self.assertEqual(g(s), "base")
> +        self.assertEqual(g(f), "base")
> +        self.assertEqual(g(t), "base")
> +        g.register(c.Sized, lambda obj: "sized")
> +        self.assertEqual(g(d), "sized")
> +        self.assertEqual(g(l), "sized")
> +        self.assertEqual(g(s), "sized")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.MutableMapping, lambda obj: "mutablemapping")
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(g(l), "sized")
> +        self.assertEqual(g(s), "sized")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.ChainMap, lambda obj: "chainmap")
> +        self.assertEqual(g(d), "mutablemapping")  # irrelevant ABCs
> registered
> +        self.assertEqual(g(l), "sized")
> +        self.assertEqual(g(s), "sized")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.MutableSequence, lambda obj: "mutablesequence")
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "sized")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.MutableSet, lambda obj: "mutableset")
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.Mapping, lambda obj: "mapping")
> +        self.assertEqual(g(d), "mutablemapping")  # not specific enough
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sized")
> +        g.register(c.Sequence, lambda obj: "sequence")
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "sized")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(c.Set, lambda obj: "set")
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "set")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(dict, lambda obj: "dict")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "mutablesequence")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "set")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(list, lambda obj: "list")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(g(s), "mutableset")
> +        self.assertEqual(g(f), "set")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(set, lambda obj: "concrete-set")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(g(s), "concrete-set")
> +        self.assertEqual(g(f), "set")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(frozenset, lambda obj: "frozen-set")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(g(s), "concrete-set")
> +        self.assertEqual(g(f), "frozen-set")
> +        self.assertEqual(g(t), "sequence")
> +        g.register(tuple, lambda obj: "tuple")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(g(s), "concrete-set")
> +        self.assertEqual(g(f), "frozen-set")
> +        self.assertEqual(g(t), "tuple")
> +
> +    def test_mro_conflicts(self):
> +        c = collections
> +
> +        @functools.singledispatch
> +        def g(arg):
> +            return "base"
> +
> +        class O(c.Sized):
> +            def __len__(self):
> +                return 0
> +
> +        o = O()
> +        self.assertEqual(g(o), "base")
> +        g.register(c.Iterable, lambda arg: "iterable")
> +        g.register(c.Container, lambda arg: "container")
> +        g.register(c.Sized, lambda arg: "sized")
> +        g.register(c.Set, lambda arg: "set")
> +        self.assertEqual(g(o), "sized")
> +        c.Iterable.register(O)
> +        self.assertEqual(g(o), "sized")   # because it's explicitly in
> __mro__
> +        c.Container.register(O)
> +        self.assertEqual(g(o), "sized")   # see above: Sized is in __mro__
> +
> +        class P:
> +            pass
> +
> +        p = P()
> +        self.assertEqual(g(p), "base")
> +        c.Iterable.register(P)
> +        self.assertEqual(g(p), "iterable")
> +        c.Container.register(P)
> +        with self.assertRaises(RuntimeError) as re:
> +            g(p)
> +            self.assertEqual(
> +                str(re),
> +                ("Ambiguous dispatch: <class 'collections.abc.Container'>
> "
> +                    "or <class 'collections.abc.Iterable'>"),
> +            )
> +
> +        class Q(c.Sized):
> +            def __len__(self):
> +                return 0
> +
> +        q = Q()
> +        self.assertEqual(g(q), "sized")
> +        c.Iterable.register(Q)
> +        self.assertEqual(g(q), "sized")   # because it's explicitly in
> __mro__
> +        c.Set.register(Q)
> +        self.assertEqual(g(q), "set")     # because c.Set is a subclass of
> +                                          # c.Sized which is explicitly in
> +                                          # __mro__
> +
> +    def test_cache_invalidation(self):
> +        from collections import UserDict
> +        class TracingDict(UserDict):
> +            def __init__(self, *args, **kwargs):
> +                super(TracingDict, self).__init__(*args, **kwargs)
> +                self.set_ops = []
> +                self.get_ops = []
> +            def __getitem__(self, key):
> +                result = self.data[key]
> +                self.get_ops.append(key)
> +                return result
> +            def __setitem__(self, key, value):
> +                self.set_ops.append(key)
> +                self.data[key] = value
> +            def clear(self):
> +                self.data.clear()
> +        _orig_wkd = functools.WeakKeyDictionary
> +        td = TracingDict()
> +        functools.WeakKeyDictionary = lambda: td
> +        c = collections
> +        @functools.singledispatch
> +        def g(arg):
> +            return "base"
> +        d = {}
> +        l = []
> +        self.assertEqual(len(td), 0)
> +        self.assertEqual(g(d), "base")
> +        self.assertEqual(len(td), 1)
> +        self.assertEqual(td.get_ops, [])
> +        self.assertEqual(td.set_ops, [dict])
> +        self.assertEqual(td.data[dict], g.registry[object])
> +        self.assertEqual(g(l), "base")
> +        self.assertEqual(len(td), 2)
> +        self.assertEqual(td.get_ops, [])
> +        self.assertEqual(td.set_ops, [dict, list])
> +        self.assertEqual(td.data[dict], g.registry[object])
> +        self.assertEqual(td.data[list], g.registry[object])
> +        self.assertEqual(td.data[dict], td.data[list])
> +        self.assertEqual(g(l), "base")
> +        self.assertEqual(g(d), "base")
> +        self.assertEqual(td.get_ops, [list, dict])
> +        self.assertEqual(td.set_ops, [dict, list])
> +        g.register(list, lambda arg: "list")
> +        self.assertEqual(td.get_ops, [list, dict])
> +        self.assertEqual(len(td), 0)
> +        self.assertEqual(g(d), "base")
> +        self.assertEqual(len(td), 1)
> +        self.assertEqual(td.get_ops, [list, dict])
> +        self.assertEqual(td.set_ops, [dict, list, dict])
> +        self.assertEqual(td.data[dict],
> +                         functools._find_impl(dict, g.registry))
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(len(td), 2)
> +        self.assertEqual(td.get_ops, [list, dict])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list])
> +        self.assertEqual(td.data[list],
> +                         functools._find_impl(list, g.registry))
> +        class X:
> +            pass
> +        c.MutableMapping.register(X)   # Will not invalidate the cache,
> +                                       # not using ABCs yet.
> +        self.assertEqual(g(d), "base")
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(td.get_ops, [list, dict, dict, list])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list])
> +        g.register(c.Sized, lambda arg: "sized")
> +        self.assertEqual(len(td), 0)
> +        self.assertEqual(g(d), "sized")
> +        self.assertEqual(len(td), 1)
> +        self.assertEqual(td.get_ops, [list, dict, dict, list])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list, dict])
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(len(td), 2)
> +        self.assertEqual(td.get_ops, [list, dict, dict, list])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list])
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(g(d), "sized")
> +        self.assertEqual(td.get_ops, [list, dict, dict, list, list, dict])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list])
> +        g.dispatch(list)
> +        g.dispatch(dict)
> +        self.assertEqual(td.get_ops, [list, dict, dict, list, list, dict,
> +                                      list, dict])
> +        self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list])
> +        c.MutableSet.register(X)       # Will invalidate the cache.
> +        self.assertEqual(len(td), 2)   # Stale cache.
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(len(td), 1)
> +        g.register(c.MutableMapping, lambda arg: "mutablemapping")
> +        self.assertEqual(len(td), 0)
> +        self.assertEqual(g(d), "mutablemapping")
> +        self.assertEqual(len(td), 1)
> +        self.assertEqual(g(l), "list")
> +        self.assertEqual(len(td), 2)
> +        g.register(dict, lambda arg: "dict")
> +        self.assertEqual(g(d), "dict")
> +        self.assertEqual(g(l), "list")
> +        g._clear_cache()
> +        self.assertEqual(len(td), 0)
> +        functools.WeakKeyDictionary = _orig_wkd
> +
> +
>  def test_main(verbose=None):
>      test_classes = (
>          TestPartialC,
> @@ -846,6 +1211,7 @@
>          TestWraps,
>          TestReduce,
>          TestLRU,
> +        TestSingleDispatch,
>      )
>      support.run_unittest(*test_classes)
>
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -344,6 +344,9 @@
>    the default for linking if LDSHARED is not also overriden.  This
> restores
>    Distutils behavior introduced in 3.2.3 and inadvertently dropped in
> 3.3.0.
>
> +- Implement PEP 443 "Single-dispatch generic functions".
> +
> +
>  Tests
>  -----
>
> diff --git a/Modules/Setup.dist b/Modules/Setup.dist
> --- a/Modules/Setup.dist
> +++ b/Modules/Setup.dist
> @@ -116,6 +116,7 @@
>  _operator _operator.c          # operator.add() and similar goodies
>  _collections _collectionsmodule.c # Container types
>  itertools itertoolsmodule.c    # Functions creating iterators for
> efficient looping
> +atexit atexitmodule.c      # Register functions to be run at
> interpreter-shutdown
>
>  # access to ISO C locale support
>  _locale _localemodule.c  # -lintl
> @@ -170,7 +171,6 @@
>  #_weakref _weakref.c   # basic weak reference support
>  #_testcapi _testcapimodule.c    # Python C API test module
>  #_random _randommodule.c       # Random number generator
> -#atexit atexitmodule.c      # Register functions to be run at
> interpreter-shutdown
>  #_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H
> -DUSE_PYEXPAT_CAPI _elementtree.c        # elementtree accelerator
>  #_pickle _pickle.c     # pickle accelerator
>  #_datetime _datetimemodule.c   # datetime accelerator
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/a0c2b1ac/attachment-0001.html>

From lukasz at langa.pl  Wed Jun  5 17:52:56 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 5 Jun 2013 17:52:56 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
	implementation for PEP 443
In-Reply-To: <CAP1=2W5NNM0V-98FMaXKErdLwtvU-6_mJcKWJ_jm2vnb3BYBrA@mail.gmail.com>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAP1=2W5NNM0V-98FMaXKErdLwtvU-6_mJcKWJ_jm2vnb3BYBrA@mail.gmail.com>
Message-ID: <0F939F6D-63D1-4D93-9EA7-39C7C09E0F6A@langa.pl>

Dnia 5 cze 2013 o godz. 16:31 Brett Cannon <brett at python.org> napisa?(a):

> Any chance you could move your definitions for "generic function" and "single dispatch" to the glossary and just link to them here?

Sure thing.

-- 
?

From benjamin at python.org  Wed Jun  5 18:10:54 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 5 Jun 2013 09:10:54 -0700
Subject: [Python-Dev] PEP 442 accepted
Message-ID: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>

I (and Guido) are accepting PEP 442 (Safe object finalization) on the
condition that finalizers are only ever called once globally.

Congratulations to Antoine on writing yet another PEP that deeply
touches the core language in a way that everyone can agree is an
improvement.. I look forward to reviewing the code.

--
Regards,
Benjamin

From solipsis at pitrou.net  Wed Jun  5 18:23:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 05 Jun 2013 18:23:07 +0200
Subject: [Python-Dev] PEP 442 accepted
In-Reply-To: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
References: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
Message-ID: <51AF65EB.2070703@pitrou.net>

Le 05/06/2013 18:10, Benjamin Peterson a ?crit :
> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> condition that finalizers are only ever called once globally.
>
> Congratulations to Antoine on writing yet another PEP that deeply
> touches the core language in a way that everyone can agree is an
> improvement.. I look forward to reviewing the code.

Thank you!

Antoine (on holiday - is anyone I know in Hungary?).


>
> --
> Regards,
> Benjamin
>


From jpakkane at gmail.com  Wed Jun  5 21:21:53 2013
From: jpakkane at gmail.com (Jussi Pakkanen)
Date: Wed, 5 Jun 2013 22:21:53 +0300
Subject: [Python-Dev] Compiling Python with Python
Message-ID: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>

Hello all

I'd like to start this email by saying this is not a proposal to change
Python's build system. This is just the results of some experimentation you
might be interested it.

I have been working on a cross-platform build system called Meson, which is
implemented in Python 3. For symmetry I wanted to see if it could be used
to build Python itself. After about an evening worth of work, I got the
basic C parts (that is, the build targets in the main Makefile such as core
Python, pgen etc) built.

Main highlights:

- pyconfig.h generation in a fully cross-platform way without Autoconf (not
tested with Visual Studio but should work as Meson has unit tests for this,
tests for functions in header files and some others not done)

- builds in a separate build directory, can have arbitrarily many build
dirs with different configurations (for gcc/clang/static
analysis/debug/release/etc)

- configure time 5s, build time 8s on an i5 Macbook running Ubuntu
(Autotool-configure takes 37 s but it's not directly comparable because it
does a lot more)

- the file describing the build is 433 lines, most of which look like this:

if cc.has_header('io.h')
  pyconf.set('HAVE_IO_H', 1)
endif

- implementation of Meson is 100% Python 3, it does not have a dependency
on the shell and in fact already works on Windows

If you want to try it yourself, here are the steps (only 64 bit Linux
tested thus far):

- install python3-ply and Ninja (Ubuntu package ninja-build)
- get Meson git head: https://sourceforge.net/p/meson/code/
- get Python3 trunk
- download and extract the build files into trunk:
https://dl.dropboxusercontent.com/u/37517477/python-meson.tar.gz
- cd into Python trunk and do the following commands:

mkdir build
cd build
path/to/meson.py ..
ninja

And there you have it. You can't do much with it, though (except run pgen
to ensure that it actually did something ;) ).

If you have any questions that are not directly related to Python, feel
free to email me or the Meson mailing list.

Enjoy,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/97866294/attachment.html>

From rosuav at gmail.com  Wed Jun  5 21:37:24 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 6 Jun 2013 05:37:24 +1000
Subject: [Python-Dev] Compiling Python with Python
In-Reply-To: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
References: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
Message-ID: <CAPTjJmrpE+aPsyqn+Mk_W3zKsPVYZhzox5N1q691jM1SeseM+w@mail.gmail.com>

On Thu, Jun 6, 2013 at 5:21 AM, Jussi Pakkanen <jpakkane at gmail.com> wrote:
> - implementation of Meson is 100% Python 3, it does not have a dependency on
> the shell and in fact already works on Windows

Since you're talking about a bootstrap requirement here, the obvious
question is: What version of Python 3 does it require? Will it be a
lot of hassle to get hold of (say) Python 3.2, only to uninstall it
when you have your 3.4 built?

ChrisA

From jpakkane at gmail.com  Wed Jun  5 22:00:47 2013
From: jpakkane at gmail.com (Jussi Pakkanen)
Date: Wed, 5 Jun 2013 23:00:47 +0300
Subject: [Python-Dev] Compiling Python with Python
In-Reply-To: <CAPTjJmrpE+aPsyqn+Mk_W3zKsPVYZhzox5N1q691jM1SeseM+w@mail.gmail.com>
References: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
	<CAPTjJmrpE+aPsyqn+Mk_W3zKsPVYZhzox5N1q691jM1SeseM+w@mail.gmail.com>
Message-ID: <CAAjYPQmrARijtYvE_d9tFsAUpRtgXtaoN0=Qxp6zD9=cOW5baA@mail.gmail.com>

On Wed, Jun 5, 2013 at 10:37 PM, Chris Angelico <rosuav at gmail.com> wrote:


> Since you're talking about a bootstrap requirement here, the obvious
> question is: What version of Python 3 does it require? Will it be a
> lot of hassle to get hold of (say) Python 3.2, only to uninstall it
> when you have your 3.4 built?
>

The implementation does not use deep Python magic such as C extensions or
the like so it should work with future releases of Python. The current
version requires 3.3 or newer but only because it uses a couple of
pythonlib functions that were added at 3.3. Changing it to work with 3.2 or
earlier should not be a big task.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130605/2a672138/attachment.html>

From cpmicropro at gmail.com  Wed Jun  5 22:02:17 2013
From: cpmicropro at gmail.com (Hossein)
Date: Thu, 06 Jun 2013 00:32:17 +0430
Subject: [Python-Dev] HAVE_FSTAT?
In-Reply-To: <1368966094.28.0.0656360189268.issue12082@psf.upfronthosting.co.za>
References: <1368966094.28.0.0656360189268.issue12082@psf.upfronthosting.co.za>
Message-ID: <51AF9949.1090204@gmail.com>

Hi. My 2 cents about this: (well I'm only a noob)

I had this problem; I don't know about other people's environment, but 
my environment's problem was that it was actually not POSIX-compliant: 
it didn't have other file functions as well, but anyway the `fstat` 
error is the FIRST error you get when you compile in such environments, 
so people as unaware as me think the problem is with fstat only.

Anyway I think if you are going to remove anything, you should think in 
terms of POSIX-compliancy of the target system. Removing HAVE_FSTAT 
might be fine (as user can easily write his own version of the function 
and have it included into the python's sources), but if you instead 
provide the user with the ability to use his custom functions when POSIX 
one's aren't available, it would help make python compile on even more 
platforms. Sorry if this last one was off-topic.

Best regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130606/46117a38/attachment.html>

From a.cavallo at cavallinux.eu  Wed Jun  5 22:18:21 2013
From: a.cavallo at cavallinux.eu (Antonio Cavallo)
Date: Wed, 5 Jun 2013 21:18:21 +0100
Subject: [Python-Dev] Compiling Python with Python
In-Reply-To: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
References: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
Message-ID: <CF5F85AD-9B6A-42DE-8A11-BD2AF4147CA6@cavallinux.eu>

What's the advantage in writing a new build tool? I'm asking this because I'm doing the same using scons:

  https://bitbucket.org/cavallo71/fatpython

At the moment I'm very interested into this problem: the main advantages I see so far are (in scons) are node dependencies and the fact it is plain python syntax.

Thanks



On 5 Jun 2013, at 20:21, Jussi Pakkanen <jpakkane at gmail.com> wrote:

> Hello all
> 
> I'd like to start this email by saying this is not a proposal to change Python's build system. This is just the results of some experimentation you might be interested it.
> 
> I have been working on a cross-platform build system called Meson, which is implemented in Python 3. For symmetry I wanted to see if it could be used to build Python itself. After about an evening worth of work, I got the basic C parts (that is, the build targets in the main Makefile such as core Python, pgen etc) built.
> 
> Main highlights:
> 
> - pyconfig.h generation in a fully cross-platform way without Autoconf (not tested with Visual Studio but should work as Meson has unit tests for this, tests for functions in header files and some others not done)
> 
> - builds in a separate build directory, can have arbitrarily many build dirs with different configurations (for gcc/clang/static analysis/debug/release/etc)
> 
> - configure time 5s, build time 8s on an i5 Macbook running Ubuntu (Autotool-configure takes 37 s but it's not directly comparable because it does a lot more)
> 
> - the file describing the build is 433 lines, most of which look like this:
> 
> if cc.has_header('io.h')
>   pyconf.set('HAVE_IO_H', 1)
> endif
> 
> - implementation of Meson is 100% Python 3, it does not have a dependency on the shell and in fact already works on Windows
> 
> If you want to try it yourself, here are the steps (only 64 bit Linux tested thus far):
> 
> - install python3-ply and Ninja (Ubuntu package ninja-build)
> - get Meson git head: https://sourceforge.net/p/meson/code/
> - get Python3 trunk
> - download and extract the build files into trunk:
> https://dl.dropboxusercontent.com/u/37517477/python-meson.tar.gz
> - cd into Python trunk and do the following commands:
> 
> mkdir build
> cd build
> path/to/meson.py ..
> ninja
> 
> And there you have it. You can't do much with it, though (except run pgen to ensure that it actually did something ;) ).
> 
> If you have any questions that are not directly related to Python, feel free to email me or the Meson mailing list.
> 
> Enjoy,
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu


From rosuav at gmail.com  Wed Jun  5 22:19:55 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 6 Jun 2013 06:19:55 +1000
Subject: [Python-Dev] Compiling Python with Python
In-Reply-To: <CAAjYPQmrARijtYvE_d9tFsAUpRtgXtaoN0=Qxp6zD9=cOW5baA@mail.gmail.com>
References: <CAAjYPQ=bAs1=ZQVaE9wu=Nzbj=Mzkj-68Dh1q5Tg0Pe_DzB-xg@mail.gmail.com>
	<CAPTjJmrpE+aPsyqn+Mk_W3zKsPVYZhzox5N1q691jM1SeseM+w@mail.gmail.com>
	<CAAjYPQmrARijtYvE_d9tFsAUpRtgXtaoN0=Qxp6zD9=cOW5baA@mail.gmail.com>
Message-ID: <CAPTjJmqTCgWJPp+N=7WWKJ-ywL6+9JJ_grix09CaLwEtZZB+AA@mail.gmail.com>

On Thu, Jun 6, 2013 at 6:00 AM, Jussi Pakkanen <jpakkane at gmail.com> wrote:
>
> On Wed, Jun 5, 2013 at 10:37 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>>
>> Since you're talking about a bootstrap requirement here, the obvious
>> question is: What version of Python 3 does it require? Will it be a
>> lot of hassle to get hold of (say) Python 3.2, only to uninstall it
>> when you have your 3.4 built?
>
>
> The implementation does not use deep Python magic such as C extensions or
> the like so it should work with future releases of Python. The current
> version requires 3.3 or newer but only because it uses a couple of pythonlib
> functions that were added at 3.3. Changing it to work with 3.2 or earlier
> should not be a big task.

Newer versions shouldn't be a problem, older ones will. I'm mainly
thinking about systems that can't just casually apt-get a Python 3.3.
With Ubuntu, most people probably don't even need to worry about
building from source, as there'll be a decently-recent Python in the
repo; but what happens when you're on Debian Squeeze and the only
Python 3 you can get is 3.1.3? Even Wheezy (the current stable Debian)
comes with only 3.2.

I do like the symmetry of using Python to build Python. But I also
like using Python 3.3+ for everything, and not having to support the
older Pythons. Unfortunately those two capacities clash, my lords,
they clash!

ChrisA

From christian at python.org  Fri Jun  7 00:37:01 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 07 Jun 2013 00:37:01 +0200
Subject: [Python-Dev] ssl improvements and testing question
Message-ID: <kor2u7$h31$1@ger.gmane.org>

Hi,

I'm working on a couple of improvements for the ssl module:

http://bugs.python.org/issue17134
http://bugs.python.org/issue18138
http://bugs.python.org/issue18143
http://bugs.python.org/issue18147

#17134 is going to provide a way to use Window's crypt32.dll to load CA
certs from Window's CA cert storage. I have a working proof of concept
[1] that uses ctypes to interface crypt32.dll. I'll reimplement the code
in C.

#18138 implements the bits and pieces for #17134 in order to add DER and
PEM certs from memory (ASCII unicode or Py_Buffer). Until now the ssl
module can only load files from the file system.

#18143 and #18147 are diagnostic and debugging helpers that I would like
to add. The SSLContext() object is black box. You stuff in some PEM
files and don't know which CA certs have been loaded. The enhancements
implement a function to retrieve a list of CA certs (same format as
getpeercert()) and list of default CA locations for the platform.

I'm also thinking about OCSP support and X509v3 extension support for
_decode_certificate(). Both are a PITB ... Python has an easier and
better documented C API.

Question:
What's the minimum version of OpenSSL Python 3.4 is going to support? Do
we have an easy way to compile and link Python against a custom
installation of OpenSSL or do I have to fiddle around with CPPFLAGS and
CFLAGS?

Christian

[1] https://pypi.python.org/pypi/wincertstore


From thomas at python.org  Fri Jun  7 16:27:06 2013
From: thomas at python.org (Thomas Wouters)
Date: Fri, 7 Jun 2013 16:27:06 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
	implementation for PEP 443
In-Reply-To: <3bQQxs6YX7zRbZ@mail.python.org>
References: <3bQQxs6YX7zRbZ@mail.python.org>
Message-ID: <CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>

On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa <python-checkins at python.org>wrote:

> +from weakref import WeakKeyDictionary
>

FYI, this change exposes a bug in the atexit module involving
subinterpreters, causing the refleaks reported by Antoine's daily report:
interpreter startup now always imports weakref, which imports atexit and
registers a classmethod. Unfortunately the atexit module doesn't seem to
know subinterpreters from subtitles and so doesn't unregister this
classmethod when the subinterpreter is terminated.

This isn't a new bug, but it's exposed by always importing weakref and
atexit during interpreter startup. I'm wondering if that's really necessary
:)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130607/c6a9d9a2/attachment.html>

From brett at python.org  Fri Jun  7 17:16:05 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jun 2013 11:16:05 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Issue #17931: Resolve
 confusion on Windows between pids and process handles.
In-Reply-To: <3bQl996bPdz7LkX@mail.python.org>
References: <3bQl996bPdz7LkX@mail.python.org>
Message-ID: <CAP1=2W5G6gRjW_A7ggPV0_OAwRWUcaSz3QS6YMQoszKsZHn-RQ@mail.gmail.com>

I think this CL introduced a memory leak. The daily leak report went from 0
to not 0 between June 4 and June 5 and this is the only CL that touched C
code.


On Wed, Jun 5, 2013 at 6:31 PM, richard.oudkerk
<python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/0410bf251e10
> changeset:   84045:0410bf251e10
> user:        Richard Oudkerk <shibturn at gmail.com>
> date:        Wed Jun 05 23:29:30 2013 +0100
> summary:
>   Issue #17931: Resolve confusion on Windows between pids and process
> handles.
>
> files:
>   Include/longobject.h  |  13 +++++++++++++
>   Misc/NEWS             |   5 ++---
>   Modules/posixmodule.c |  25 +++++++++----------------
>   PC/msvcrtmodule.c     |   5 +++--
>   PC/pyconfig.h         |   4 ++--
>   5 files changed, 29 insertions(+), 23 deletions(-)
>
>
> diff --git a/Include/longobject.h b/Include/longobject.h
> --- a/Include/longobject.h
> +++ b/Include/longobject.h
> @@ -52,6 +52,19 @@
>  #error "sizeof(pid_t) is neither sizeof(int), sizeof(long) or sizeof(long
> long)"
>  #endif /* SIZEOF_PID_T */
>
> +#if SIZEOF_VOID_P == SIZEOF_INT
> +#  define _Py_PARSE_INTPTR "i"
> +#  define _Py_PARSE_UINTPTR "I"
> +#elif SIZEOF_VOID_P == SIZEOF_LONG
> +#  define _Py_PARSE_INTPTR "l"
> +#  define _Py_PARSE_UINTPTR "k"
> +#elif defined(SIZEOF_LONG_LONG) && SIZEOF_VOID_P == SIZEOF_LONG_LONG
> +#  define _Py_PARSE_INTPTR "L"
> +#  define _Py_PARSE_UINTPTR "K"
> +#else
> +#  error "void* different in size from int, long and long long"
> +#endif /* SIZEOF_VOID_P */
> +
>  /* Used by Python/mystrtoul.c. */
>  #ifndef Py_LIMITED_API
>  PyAPI_DATA(unsigned char) _PyLong_DigitValue[256];
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -10,9 +10,8 @@
>  Core and Builtins
>  -----------------
>
> -- Issue #17931: Fix PyLong_FromPid() on Windows 64-bit: processes are
> -  identified by their HANDLE which is a pointer (and not a long, which is
> -  smaller).
> +- Issue #17931: Resolve confusion on Windows between pids and process
> +  handles.
>
>  - Tweak the exception message when the magic number or size value in a
> bytecode
>    file is truncated.
> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
> --- a/Modules/posixmodule.c
> +++ b/Modules/posixmodule.c
> @@ -5014,11 +5014,7 @@
>      if (spawnval == -1)
>          return posix_error();
>      else
> -#if SIZEOF_LONG == SIZEOF_VOID_P
> -        return Py_BuildValue("l", (long) spawnval);
> -#else
> -        return Py_BuildValue("L", (PY_LONG_LONG) spawnval);
> -#endif
> +        return Py_BuildValue(_Py_PARSE_INTPTR, spawnval);
>  }
>
>
> @@ -5104,11 +5100,7 @@
>      if (spawnval == -1)
>          (void) posix_error();
>      else
> -#if SIZEOF_LONG == SIZEOF_VOID_P
> -        res = Py_BuildValue("l", (long) spawnval);
> -#else
> -        res = Py_BuildValue("L", (PY_LONG_LONG) spawnval);
> -#endif
> +        res = Py_BuildValue(_Py_PARSE_INTPTR, spawnval);
>
>      while (--envc >= 0)
>          PyMem_DEL(envlist[envc]);
> @@ -6178,16 +6170,17 @@
>  win32_kill(PyObject *self, PyObject *args)
>  {
>      PyObject *result;
> -    DWORD pid, sig, err;
> +    pid_t pid;
> +    DWORD sig, err;
>      HANDLE handle;
>
> -    if (!PyArg_ParseTuple(args, "kk:kill", &pid, &sig))
> +    if (!PyArg_ParseTuple(args, _Py_PARSE_PID "k:kill", &pid, &sig))
>          return NULL;
>
>      /* Console processes which share a common console can be sent CTRL+C
> or
>         CTRL+BREAK events, provided they handle said events. */
>      if (sig == CTRL_C_EVENT || sig == CTRL_BREAK_EVENT) {
> -        if (GenerateConsoleCtrlEvent(sig, pid) == 0) {
> +        if (GenerateConsoleCtrlEvent(sig, (DWORD)pid) == 0) {
>              err = GetLastError();
>              PyErr_SetFromWindowsErr(err);
>          }
> @@ -6197,7 +6190,7 @@
>
>      /* If the signal is outside of what GenerateConsoleCtrlEvent can use,
>         attempt to open and terminate the process. */
> -    handle = OpenProcess(PROCESS_ALL_ACCESS, FALSE, pid);
> +    handle = OpenProcess(PROCESS_ALL_ACCESS, FALSE, (DWORD)pid);
>      if (handle == NULL) {
>          err = GetLastError();
>          return PyErr_SetFromWindowsErr(err);
> @@ -6603,7 +6596,7 @@
>      Py_intptr_t pid;
>      int status, options;
>
> -    if (!PyArg_ParseTuple(args, _Py_PARSE_PID "i:waitpid", &pid,
> &options))
> +    if (!PyArg_ParseTuple(args, _Py_PARSE_INTPTR "i:waitpid", &pid,
> &options))
>          return NULL;
>      Py_BEGIN_ALLOW_THREADS
>      pid = _cwait(&status, pid, options);
> @@ -6612,7 +6605,7 @@
>          return posix_error();
>
>      /* shift the status left a byte so this is more like the POSIX
> waitpid */
> -    return Py_BuildValue("Ni", PyLong_FromPid(pid), status << 8);
> +    return Py_BuildValue(_Py_PARSE_INTPTR "i", pid, status << 8);
>  }
>  #endif /* HAVE_WAITPID || HAVE_CWAIT */
>
> diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c
> --- a/PC/msvcrtmodule.c
> +++ b/PC/msvcrtmodule.c
> @@ -113,11 +113,12 @@
>  static PyObject *
>  msvcrt_open_osfhandle(PyObject *self, PyObject *args)
>  {
> -    long handle;
> +    Py_intptr_t handle;
>      int flags;
>      int fd;
>
> -    if (!PyArg_ParseTuple(args, "li:open_osfhandle", &handle, &flags))
> +    if (!PyArg_ParseTuple(args, _Py_PARSE_INTPTR "i:open_osfhandle",
> +                          &handle, &flags))
>          return NULL;
>
>      fd = _open_osfhandle(handle, flags);
> diff --git a/PC/pyconfig.h b/PC/pyconfig.h
> --- a/PC/pyconfig.h
> +++ b/PC/pyconfig.h
> @@ -723,8 +723,8 @@
>  /* The size of `wchar_t', as computed by sizeof. */
>  #define SIZEOF_WCHAR_T 2
>
> -/* The size of `pid_t' (HANDLE). */
> -#define SIZEOF_PID_T SIZEOF_VOID_P
> +/* The size of `pid_t', as computed by sizeof. */
> +#define SIZEOF_PID_T SIZEOF_INT
>
>  /* Define if you have the dl library (-ldl).  */
>  /* #undef HAVE_LIBDL */
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130607/5b9cf444/attachment.html>

From thomas at python.org  Fri Jun  7 17:54:48 2013
From: thomas at python.org (Thomas Wouters)
Date: Fri, 7 Jun 2013 17:54:48 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #17931: Resolve
 confusion on Windows between pids and process handles.
In-Reply-To: <CAP1=2W5G6gRjW_A7ggPV0_OAwRWUcaSz3QS6YMQoszKsZHn-RQ@mail.gmail.com>
References: <3bQl996bPdz7LkX@mail.python.org>
	<CAP1=2W5G6gRjW_A7ggPV0_OAwRWUcaSz3QS6YMQoszKsZHn-RQ@mail.gmail.com>
Message-ID: <CAPdQG2ofZ+Uq8GOt_4f9ebmp2Ry2NQ7JiZx2DPinV0_UQ8pqyg@mail.gmail.com>

On Fri, Jun 7, 2013 at 5:16 PM, Brett Cannon <brett at python.org> wrote:

> I think this CL introduced a memory leak. The daily leak report went from
> 0 to not 0 between June 4 and June 5 and this is the only CL that touched C
> code.
>

It wasn't introduced by C code :) The refleak report is induced by the PEP
443 implementation, see my message to python-dev.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130607/bc666f27/attachment.html>

From ethan at stoneleaf.us  Fri Jun  7 17:37:08 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 07 Jun 2013 08:37:08 -0700
Subject: [Python-Dev] html documentation and table-of-contents
Message-ID: <51B1FE24.3060900@stoneleaf.us>

I just used the build system on the 3.4.0 docs, and some of the library modules (haven't checked the others) have the 
TOC showing up at the bottom of the page instead of the top.

Am I doing something wrong?

--
~Ethan~

From status at bugs.python.org  Fri Jun  7 18:07:36 2013
From: status at bugs.python.org (Python tracker)
Date: Fri,  7 Jun 2013 18:07:36 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130607160736.0C9565692C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-05-31 - 2013-06-07)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4024 (+27)
  closed 25905 (+21)
  total  29929 (+48)

Open issues with patches: 1793 


Issues opened (41)
==================

#8902: add datetime.time.now() for consistency
http://bugs.python.org/issue8902  reopened by belopolsky

#17206: Py_XDECREF() expands its argument multiple times
http://bugs.python.org/issue17206  reopened by haypo

#17222: py_compile.compile() replaces target files, breaking special f
http://bugs.python.org/issue17222  reopened by brett.cannon

#18110: Nested set comprehensions in class scope fail
http://bugs.python.org/issue18110  opened by Eric.Wieser

#18111: Add a default argument to min & max
http://bugs.python.org/issue18111  opened by Julian

#18112: PEP 442 implementation
http://bugs.python.org/issue18112  opened by pitrou

#18113: Memory leak in curses.panel
http://bugs.python.org/issue18113  opened by ishimoto

#18114: Update PyImport_ImportFrozenModuleObject() to use importlib
http://bugs.python.org/issue18114  opened by brett.cannon

#18115: Use importlib.util.module_to_load in all loaders in importlib
http://bugs.python.org/issue18115  opened by brett.cannon

#18116: getpass.unix_getpass() always fallback to sys.stdin
http://bugs.python.org/issue18116  opened by Nikratio

#18117: Missing symlink:Current after Mac OS X 3.3.2 package installat
http://bugs.python.org/issue18117  opened by gavan

#18119: urllib.FancyURLopener does not treat URL fragments correctly
http://bugs.python.org/issue18119  opened by takahashi.shuhei

#18121: antigravity leaks subprocess.Popen object
http://bugs.python.org/issue18121  opened by christian.heimes

#18122: RuntimeError: not holding the import lock
http://bugs.python.org/issue18122  opened by arigo

#18123: fnmatchicase for case insensitive file search
http://bugs.python.org/issue18123  opened by techtonik

#18125: Out-of-tree build cannot regenerate Makefile.pre
http://bugs.python.org/issue18125  opened by TBBle

#18126: Update links to NumPy resources in documentation
http://bugs.python.org/issue18126  opened by zaytsev

#18128: pygettext: non-standard timestamp format in POT-Creation-Date
http://bugs.python.org/issue18128  opened by jwilk

#18129: Fatal Python error: Cannot recover from stack overflow.
http://bugs.python.org/issue18129  opened by oscarbenjamin

#18131: Tkinter Variables require a proper master
http://bugs.python.org/issue18131  opened by terry.reedy

#18132: buttons in turtledemo disappear on small screens
http://bugs.python.org/issue18132  opened by JanKanis

#18135: _ssl module: possible integer overflow for very long strings (
http://bugs.python.org/issue18135  opened by haypo

#18136: Put local build paths before system build paths in configure.a
http://bugs.python.org/issue18136  opened by brett.cannon

#18137: format(float, str): integer overflow for huge precision
http://bugs.python.org/issue18137  opened by haypo

#18138: ssl.SSLContext.add_cert()
http://bugs.python.org/issue18138  opened by christian.heimes

#18139: email module should have a way to prepend and insert headers
http://bugs.python.org/issue18139  opened by kbandla

#18140: urlparse.urlsplit confused to fragment when password include #
http://bugs.python.org/issue18140  opened by anh.le

#18141: tkinter.Image.__del__ can throw an exception if module globals
http://bugs.python.org/issue18141  opened by JanKanis

#18142: Tests fail on Mageia Linux Cauldron x86-64 with some configure
http://bugs.python.org/issue18142  opened by shlomif

#18143: ssl.get_default_verify_paths()
http://bugs.python.org/issue18143  opened by christian.heimes

#18144: FD leak in urllib2
http://bugs.python.org/issue18144  opened by Claudio.Freire

#18147: SSL: diagnostic functions to list loaded CA certs
http://bugs.python.org/issue18147  opened by christian.heimes

#18148: Make of Python 3.2.2 fails on Solaris SPARC
http://bugs.python.org/issue18148  opened by eeiddne

#18149: filecmp.cmp() - cache invalidation fails when file modificatio
http://bugs.python.org/issue18149  opened by fbm

#18150: Duplicate test inside TestSingleDispatch
http://bugs.python.org/issue18150  opened by vajrasky

#18151: Idlelib: update to "with open ... except OSError"  (in 2.7, le
http://bugs.python.org/issue18151  opened by terry.reedy

#18152: Idle: add 2.7 backport script
http://bugs.python.org/issue18152  opened by terry.reedy

#18153: python imaplib - error 'unexpected repsonse'
http://bugs.python.org/issue18153  opened by tahnoon

#18154: make install failed on solaris
http://bugs.python.org/issue18154  opened by palm.kevin

#18155: csv.Sniffer.has_header doesn't escape characters used in regex
http://bugs.python.org/issue18155  opened by davechallis

#18156: Add an 'attr' attribute to AttributeError
http://bugs.python.org/issue18156  opened by brett.cannon



Most recent 15 issues with no replies (15)
==========================================

#18156: Add an 'attr' attribute to AttributeError
http://bugs.python.org/issue18156

#18149: filecmp.cmp() - cache invalidation fails when file modificatio
http://bugs.python.org/issue18149

#18148: Make of Python 3.2.2 fails on Solaris SPARC
http://bugs.python.org/issue18148

#18147: SSL: diagnostic functions to list loaded CA certs
http://bugs.python.org/issue18147

#18144: FD leak in urllib2
http://bugs.python.org/issue18144

#18138: ssl.SSLContext.add_cert()
http://bugs.python.org/issue18138

#18137: format(float, str): integer overflow for huge precision
http://bugs.python.org/issue18137

#18131: Tkinter Variables require a proper master
http://bugs.python.org/issue18131

#18129: Fatal Python error: Cannot recover from stack overflow.
http://bugs.python.org/issue18129

#18126: Update links to NumPy resources in documentation
http://bugs.python.org/issue18126

#18119: urllib.FancyURLopener does not treat URL fragments correctly
http://bugs.python.org/issue18119

#18114: Update PyImport_ImportFrozenModuleObject() to use importlib
http://bugs.python.org/issue18114

#18113: Memory leak in curses.panel
http://bugs.python.org/issue18113

#18112: PEP 442 implementation
http://bugs.python.org/issue18112

#18108: shutil.chown should support dir_fd and follow_symlinks keyword
http://bugs.python.org/issue18108



Most recent 15 issues waiting for review (15)
=============================================

#18155: csv.Sniffer.has_header doesn't escape characters used in regex
http://bugs.python.org/issue18155

#18154: make install failed on solaris
http://bugs.python.org/issue18154

#18151: Idlelib: update to "with open ... except OSError"  (in 2.7, le
http://bugs.python.org/issue18151

#18150: Duplicate test inside TestSingleDispatch
http://bugs.python.org/issue18150

#18147: SSL: diagnostic functions to list loaded CA certs
http://bugs.python.org/issue18147

#18143: ssl.get_default_verify_paths()
http://bugs.python.org/issue18143

#18141: tkinter.Image.__del__ can throw an exception if module globals
http://bugs.python.org/issue18141

#18138: ssl.SSLContext.add_cert()
http://bugs.python.org/issue18138

#18137: format(float, str): integer overflow for huge precision
http://bugs.python.org/issue18137

#18135: _ssl module: possible integer overflow for very long strings (
http://bugs.python.org/issue18135

#18132: buttons in turtledemo disappear on small screens
http://bugs.python.org/issue18132

#18126: Update links to NumPy resources in documentation
http://bugs.python.org/issue18126

#18116: getpass.unix_getpass() always fallback to sys.stdin
http://bugs.python.org/issue18116

#18112: PEP 442 implementation
http://bugs.python.org/issue18112

#18111: Add a default argument to min & max
http://bugs.python.org/issue18111



Top 10 most discussed issues (10)
=================================

#18111: Add a default argument to min & max
http://bugs.python.org/issue18111  28 msgs

#18116: getpass.unix_getpass() always fallback to sys.stdin
http://bugs.python.org/issue18116  17 msgs

#18021: Update broken link to Apple Publication Style Guide
http://bugs.python.org/issue18021   9 msgs

#16427: Faster hash implementation
http://bugs.python.org/issue16427   8 msgs

#17222: py_compile.compile() replaces target files, breaking special f
http://bugs.python.org/issue17222   7 msgs

#18078: threading.Condition to allow notify on a specific waiter
http://bugs.python.org/issue18078   7 msgs

#3329: API for setting the memory allocator used by Python
http://bugs.python.org/issue3329   6 msgs

#18054: Add more exception related assertions to unittest
http://bugs.python.org/issue18054   6 msgs

#18122: RuntimeError: not holding the import lock
http://bugs.python.org/issue18122   6 msgs

#18143: ssl.get_default_verify_paths()
http://bugs.python.org/issue18143   6 msgs



Issues closed (24)
==================

#11959: smtpd cannot be used without affecting global state
http://bugs.python.org/issue11959  closed by python-dev

#16450: test_missing_localfile masks problems in urlopen
http://bugs.python.org/issue16450  closed by orsenthil

#17314: Stop using imp.find_module() in multiprocessing
http://bugs.python.org/issue17314  closed by brett.cannon

#17342: datetime.strptime does not implement %z
http://bugs.python.org/issue17342  closed by belopolsky

#17486: datetime.timezone returns the wrong tzname()
http://bugs.python.org/issue17486  closed by belopolsky

#17931: PyLong_FromPid() is not correctly defined on Windows 64-bit
http://bugs.python.org/issue17931  closed by python-dev

#17932: Win64: possible integer overflow in iterobject.c
http://bugs.python.org/issue17932  closed by python-dev

#17967: urllib2.open failed to access a url when a perent directory of
http://bugs.python.org/issue17967  closed by orsenthil

#18065: set __path__ = [] for frozen packages
http://bugs.python.org/issue18065  closed by brett.cannon

#18066: Remove SGI-specific code from pty.py
http://bugs.python.org/issue18066  closed by akuchling

#18088: Create importlib.abc.Loader.init_module_attrs()
http://bugs.python.org/issue18088  closed by brett.cannon

#18089: Create importlib.abc.InspectLoader.load_module()
http://bugs.python.org/issue18089  closed by brett.cannon

#18094: Skip tests in test_uuid not silently
http://bugs.python.org/issue18094  closed by serhiy.storchaka

#18109: os.uname() crashes if hostname contains non-ascii characters
http://bugs.python.org/issue18109  closed by python-dev

#18118: curses utf8 output broken in Python2
http://bugs.python.org/issue18118  closed by haypo

#18120: multiprocessing: garbage collector fails to GC Pipe() end when
http://bugs.python.org/issue18120  closed by sbt

#18124: Broken build on target machine with incorrect hostname (non-as
http://bugs.python.org/issue18124  closed by dmi.baranov

#18127: Strange behaviour with default list argument
http://bugs.python.org/issue18127  closed by eric.smith

#18130: idlelib.configSectionNameDialog: fix and add tests and mocks
http://bugs.python.org/issue18130  closed by terry.reedy

#18133: Modulus not calculating properly?
http://bugs.python.org/issue18133  closed by pccreator25

#18134: zipfile extractall accepts wrong password
http://bugs.python.org/issue18134  closed by r.david.murray

#18145: Strange behavior when importing internal modules in the __init
http://bugs.python.org/issue18145  closed by brett.cannon

#18146: Document miss, Stats objects has no method called "print_resul
http://bugs.python.org/issue18146  closed by hanks

#1512124: OSX: debugger hangs IDLE
http://bugs.python.org/issue1512124  closed by ned.deily

From ethan at stoneleaf.us  Fri Jun  7 18:06:15 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 07 Jun 2013 09:06:15 -0700
Subject: [Python-Dev] doctest and pickle
Message-ID: <51B204F7.4030107@stoneleaf.us>

Is there a doctest mailing list?  I couldn't find it.


I'm try to use doctest to verify my docs (imagine that!) but I'm having trouble with the one that uses pickle (imagine 
that!).

Any advice on how to make it work?

Here's the excerpt:

===============================================================================
Pickling
--------

Enumerations can be pickled and unpickled::

     >>> from enum import Enum
     >>> class Fruit(Enum):
     ...     tomato = 1
     ...     banana = 2
     ...     cherry = 3
     ...
     >>> from pickle import dumps, loads
     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
     True

The usual restrictions for pickling apply: picklable enums must be defined in
the top level of a module, since unpickling requires them to be importable
from that module.
===============================================================================

Oh, and I just realized this is probably why the flufl.enum docs import from a preexisting module instead of creating a 
new class on the spot.  Still, it would be nice if this could work.

Any ideas?

--
~Ethan~

From olemis at gmail.com  Fri Jun  7 18:54:11 2013
From: olemis at gmail.com (Olemis Lang)
Date: Fri, 7 Jun 2013 11:54:11 -0500
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B204F7.4030107@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
Message-ID: <CAGMZAuMRrZpuoY0MOdOJXsBxYp1v9Xb1RngTXQaugpq3f6_h1g@mail.gmail.com>

On 6/7/13, Ethan Furman <ethan at stoneleaf.us> wrote:
> Is there a doctest mailing list?  I couldn't find it.
>

JFTR, Testing-in-Python (TiP) ML should be the right target for
general purpose questions about testing, considering docs even for
unittest and doctest
http://lists.idyll.org/listinfo/testing-in-python

[...]
> Any advice on how to make it work?
>
> Here's the excerpt:
>
> ===============================================================================
> Pickling
> --------
>
> Enumerations can be pickled and unpickled::
>
>      >>> from enum import Enum
>      >>> class Fruit(Enum):
>      ...     tomato = 1
>      ...     banana = 2
>      ...     cherry = 3
>      ...
>      >>> from pickle import dumps, loads
>      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>      True
>

... but it seems I'm still getting in tests an instance of Fruit after
invoking loads (do you ?)

[...]

-- 
Regards,

Olemis.

Apache? Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

From dreamingforward at gmail.com  Fri Jun  7 19:50:50 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Fri, 7 Jun 2013 10:50:50 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B204F7.4030107@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
Message-ID: <CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>

>     >>> from pickle import dumps, loads
>     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>     True

Why are you using is here instead of ==?  You're making a circular
loop using "is"
-- 
MarkJ
Tacoma, Washington

From dreamingforward at gmail.com  Fri Jun  7 19:54:57 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Fri, 7 Jun 2013 10:54:57 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
Message-ID: <CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>

On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen <dreamingforward at gmail.com> wrote:
>>     >>> from pickle import dumps, loads
>>     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>     True
>
> Why are you using is here instead of ==?  You're making a circular
> loop using "is"

I should add that when you're serializing with pickle and then
reloading, the objects should be seen as "essentially equivalent".
This means that they are either byte-by-byte equivalent (not sure
actually if Python actually guarantees this), or every element would
still compare equal and that is what matters.

-- 
MarkJ
Tacoma, Washington

From rdmurray at bitdance.com  Fri Jun  7 20:11:59 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 07 Jun 2013 14:11:59 -0400
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
	<CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
Message-ID: <20130607181205.3B93F250BEB@webabinitio.net>

On Fri, 07 Jun 2013 10:54:57 -0700, Mark Janssen <dreamingforward at gmail.com> wrote:
> On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen <dreamingforward at gmail.com> wrote:
> >>     >>> from pickle import dumps, loads
> >>     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
> >>     True
> >
> > Why are you using is here instead of ==?  You're making a circular
> > loop using "is"
> 
> I should add that when you're serializing with pickle and then
> reloading, the objects should be seen as "essentially equivalent".
> This means that they are either byte-by-byte equivalent (not sure
> actually if Python actually guarantees this), or every element would
> still compare equal and that is what matters.

Enums are supposed to be singletons, though, so the 'is' test
is exactly the point of this test.

--David

From ethan at stoneleaf.us  Fri Jun  7 19:53:14 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 07 Jun 2013 10:53:14 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAGMZAuMRrZpuoY0MOdOJXsBxYp1v9Xb1RngTXQaugpq3f6_h1g@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAGMZAuMRrZpuoY0MOdOJXsBxYp1v9Xb1RngTXQaugpq3f6_h1g@mail.gmail.com>
Message-ID: <51B21E0A.3090409@stoneleaf.us>

On 06/07/2013 09:54 AM, Olemis Lang wrote:
> On 6/7/13, Ethan Furman <ethan at stoneleaf.us> wrote:
>> Is there a doctest mailing list?  I couldn't find it.
>>
>
> JFTR, Testing-in-Python (TiP) ML should be the right target for
> general purpose questions about testing, considering docs even for
> unittest and doctest
> http://lists.idyll.org/listinfo/testing-in-python

Cool, thanks.


> [...]
>> Any advice on how to make it work?
>>
>> Here's the excerpt:
>>
>> ===============================================================================
>> Pickling
>> --------
>>
>> Enumerations can be pickled and unpickled::
>>
>>       >>> from enum import Enum
>>       >>> class Fruit(Enum):
>>       ...     tomato = 1
>>       ...     banana = 2
>>       ...     cherry = 3
>>       ...
>>       >>> from pickle import dumps, loads
>>       >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>       True
>>
>
> ... but it seems I'm still getting in tests an instance of Fruit after
> invoking loads (do you ?)

I'm not sure what you mean by "an instance of Fruit" -- you should be getting an instance of Fruit; specifically the 
tomato instance.

--
~Ethan~

From brett at python.org  Fri Jun  7 20:16:44 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jun 2013 14:16:44 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
 implementation for PEP 443
In-Reply-To: <CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
Message-ID: <CAP1=2W5=Pzcoyx2rdBVhu7=WH-r__RSZiFVz7=fHWSWVU0qZrA@mail.gmail.com>

On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas at python.org> wrote:

>
> On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa <python-checkins at python.org>wrote:
>
>> +from weakref import WeakKeyDictionary
>>
>
> FYI, this change exposes a bug in the atexit module involving
> subinterpreters, causing the refleaks reported by Antoine's daily report:
> interpreter startup now always imports weakref, which imports atexit and
> registers a classmethod. Unfortunately the atexit module doesn't seem to
> know subinterpreters from subtitles and so doesn't unregister this
> classmethod when the subinterpreter is terminated.
>
> This isn't a new bug, but it's exposed by always importing weakref and
> atexit during interpreter startup. I'm wondering if that's really necessary
> :)
>

Is there an issue tracking this?

-Brett


>
>
> --
> Thomas Wouters <thomas at python.org>
>
> Hi! I'm an email virus! Think twice before sending your email to help me
> spread!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130607/509eb8ad/attachment.html>

From ethan at stoneleaf.us  Fri Jun  7 19:55:59 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 07 Jun 2013 10:55:59 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
Message-ID: <51B21EAF.2000809@stoneleaf.us>

On 06/07/2013 10:50 AM, Mark Janssen wrote:
>>      >>> from pickle import dumps, loads
>>      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>      True
>
> Why are you using is here instead of ==?

I'm using `is` because I'm verifying that the instance returned by `pickle.loads` is the exact same object as the 
instance fed into `pickle.dumps`.  Enum members should be singletons.


> You're making a circular loop using "is"

Huh?

--
~Ethan~

From ethan at stoneleaf.us  Fri Jun  7 20:02:54 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 07 Jun 2013 11:02:54 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
	<CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
Message-ID: <51B2204E.2060007@stoneleaf.us>

On 06/07/2013 10:54 AM, Mark Janssen wrote:
> On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen <dreamingforward at gmail.com> wrote:
>>>      >>> from pickle import dumps, loads
>>>      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>>      True
>>
>> Why are you using is here instead of ==?  You're making a circular
>> loop using "is"
>
> I should add that when you're serializing with pickle and then
> reloading, the objects should be seen as "essentially equivalent".
> This means that they are either byte-by-byte equivalent (not sure
> actually if Python actually guarantees this), or every element would
> still compare equal and that is what matters.

In most cases, sure.  But one of the invariants of Enums is that there is only ever one of each member specifically so 
that `is` tests work.

--
~Ethan~

From pje at telecommunity.com  Fri Jun  7 20:30:39 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 7 Jun 2013 14:30:39 -0400
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
	<CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
Message-ID: <CALeMXf6kHwoooTdeSNfMCpD05ZwdYiomGAMgez0x==Wc_sV5ag@mail.gmail.com>

On Fri, Jun 7, 2013 at 1:54 PM, Mark Janssen <dreamingforward at gmail.com> wrote:
> On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen <dreamingforward at gmail.com> wrote:
>>>     >>> from pickle import dumps, loads
>>>     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>>     True
>>
>> Why are you using is here instead of ==?  You're making a circular
>> loop using "is"
>
> I should add that when you're serializing with pickle and then
> reloading, the objects should be seen as "essentially equivalent".
> This means that they are either byte-by-byte equivalent (not sure
> actually if Python actually guarantees this), or every element would
> still compare equal and that is what matters.

For global objects such as functions and classes -- and singletons
such as None, Ellipsis, True, and False -- pickling and unpickling is
actually supposed to retain the "is" relationship as well.

I don't know if enums *actually* preserve this invariant, but my
default expectation of the One Obvious Way would be that enums, being
uniquely-named objects that know their name and container, should be
considered global objects in the same fashion as classes and
functions, *and* that as singletons, they'd also be treated in the
same way as None, Ellipsis, etc.  That is, there are two independent
precedents for objects like that preserving "is" upon pickling and
unpickling.

(As another precedent, my own SymbolType library (available on PyPI)
preserves the "is"-ness of its named symbol objects upon pickling and
unpickling.)

From dreamingforward at gmail.com  Fri Jun  7 21:25:44 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Fri, 7 Jun 2013 12:25:44 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B21EAF.2000809@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
	<51B21EAF.2000809@stoneleaf.us>
Message-ID: <CAMjeLr_W9S4WTyW_TQKz2P3qHqBTL_3y0AUUiiz7vrH5idMjFA@mail.gmail.com>

>> Why are you using is here instead of ==?
>
>
> I'm using `is` because I'm verifying that the instance returned by
> `pickle.loads` is the exact same object as the instance fed into
> `pickle.dumps`.  Enum members should be singletons.

I see now.  That makes sense, but I don't think you'll be able to do
that.  "Supposed to be singletons" while at the same time you are
holding two instances on the interpreter line.   How are you going to
manage that?

-- 
MarkJ
Tacoma, Washington

From barry at python.org  Fri Jun  7 21:31:21 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 7 Jun 2013 15:31:21 -0400
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B204F7.4030107@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
Message-ID: <20130607153121.1a1418e0@anarchist>

On Jun 07, 2013, at 09:06 AM, Ethan Furman wrote:

>Oh, and I just realized this is probably why the flufl.enum docs import from
>a preexisting module instead of creating a new class on the spot.

Exactly. ;)

-Barry

From barry at python.org  Fri Jun  7 21:33:26 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 7 Jun 2013 15:33:26 -0400
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CALeMXf6kHwoooTdeSNfMCpD05ZwdYiomGAMgez0x==Wc_sV5ag@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<CAMjeLr_j0e0EDrAOF42cLGHo8PCnKPPYe6Re00B6TXQg_HzdeA@mail.gmail.com>
	<CAMjeLr9CFxjXdh-hfgJ9fY9A4_wvSZqoqBAB3q0fFtno019Q8A@mail.gmail.com>
	<CALeMXf6kHwoooTdeSNfMCpD05ZwdYiomGAMgez0x==Wc_sV5ag@mail.gmail.com>
Message-ID: <20130607153326.56d6424b@anarchist>

On Jun 07, 2013, at 02:30 PM, PJ Eby wrote:

>I don't know if enums *actually* preserve this invariant, but my
>default expectation of the One Obvious Way would be that enums, being
>uniquely-named objects that know their name and container, should be
>considered global objects in the same fashion as classes and
>functions, *and* that as singletons, they'd also be treated in the
>same way as None, Ellipsis, etc.  That is, there are two independent
>precedents for objects like that preserving "is" upon pickling and
>unpickling.

This is certainly how I thought of them in flufl.enum, and indeed the `is`
test in my own test suite proves that it works.

-Barry

From lukasz at langa.pl  Fri Jun  7 21:42:42 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 7 Jun 2013 21:42:42 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
	implementation for PEP 443
In-Reply-To: <CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
Message-ID: <DFCCEC78-B7C0-4E57-829A-74D712748F72@langa.pl>


On 7 cze 2013, at 16:27, Thomas Wouters <thomas at python.org> wrote:

> 
> On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa <python-checkins at python.org> wrote:
> +from weakref import WeakKeyDictionary
> 
> This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :)

We can easily move the import inside singledispatch but I would much rather try fixing the actual bug. What do you think?

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130607/4ce89a3a/attachment.html>

From pje at telecommunity.com  Fri Jun  7 22:50:53 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 7 Jun 2013 16:50:53 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
 implementation for PEP 443
In-Reply-To: <CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
Message-ID: <CALeMXf5kMFLQ-9-2fhTELeBfySM36OK8ANBrOGjOuoY9PLgA5Q@mail.gmail.com>

On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas at python.org> wrote:
> This isn't a new bug, but it's exposed by always importing weakref and
> atexit during interpreter startup. I'm wondering if that's really necessary
> :)

Importing it during startup isn't necessary per se; imports needed
only by the generic function implementation can and should be imported
late, rather than at the time functools is imported.

However, if pkgutil was/is migrated to using this implementation of
generics, then it's likely going to end up imported during startup
anyway, because at least the -m startup path involves pkgutil.

In short, the overall answer right now is, "maybe", and the answer
later is "rather likely".  ;-)

From lukasz at langa.pl  Fri Jun  7 23:16:35 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 7 Jun 2013 23:16:35 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
	implementation for PEP 443
In-Reply-To: <CALeMXf5kMFLQ-9-2fhTELeBfySM36OK8ANBrOGjOuoY9PLgA5Q@mail.gmail.com>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
	<CALeMXf5kMFLQ-9-2fhTELeBfySM36OK8ANBrOGjOuoY9PLgA5Q@mail.gmail.com>
Message-ID: <DA6546CF-F251-4D92-B607-C8021E1F7FF5@langa.pl>

On 7 cze 2013, at 22:50, PJ Eby <pje at telecommunity.com> wrote:

> On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas at python.org> wrote:
>> This isn't a new bug, but it's exposed by always importing weakref and
>> atexit during interpreter startup. I'm wondering if that's really necessary
>> :)
> 
> In short, the overall answer right now is, "maybe", and the answer
> later is "rather likely".  ;-)

I would rather say that it's "rather certain".

functools is necessary for setup.py to work while bootstrapping, whereas
pkgutil is used in runpy.py which is always imported in Modules/main.c.

So we're left with having to fix atexit to support subinterpreters. I wonder
how difficult that will be.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev


From pje at telecommunity.com  Sat Jun  8 06:38:03 2013
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 8 Jun 2013 00:38:03 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Add reference
 implementation for PEP 443
In-Reply-To: <DA6546CF-F251-4D92-B607-C8021E1F7FF5@langa.pl>
References: <3bQQxs6YX7zRbZ@mail.python.org>
	<CAPdQG2r5kH1kKcy7=--jmg5094Dvr9jk2q2=nephVGDA-7dDgA@mail.gmail.com>
	<CALeMXf5kMFLQ-9-2fhTELeBfySM36OK8ANBrOGjOuoY9PLgA5Q@mail.gmail.com>
	<DA6546CF-F251-4D92-B607-C8021E1F7FF5@langa.pl>
Message-ID: <CALeMXf7scvFd7=Fodw09F5YD0VkddBreg=Hv3qxJLZ8P=0Wh9w@mail.gmail.com>

On Fri, Jun 7, 2013 at 5:16 PM, ?ukasz Langa <lukasz at langa.pl> wrote:
> On 7 cze 2013, at 22:50, PJ Eby <pje at telecommunity.com> wrote:
>
>> On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas at python.org> wrote:
>>> This isn't a new bug, but it's exposed by always importing weakref and
>>> atexit during interpreter startup. I'm wondering if that's really necessary
>>> :)
>>
>> In short, the overall answer right now is, "maybe", and the answer
>> later is "rather likely".  ;-)
>
> I would rather say that it's "rather certain".
>
> functools is necessary for setup.py to work while bootstrapping, whereas
> pkgutil is used in runpy.py which is always imported in Modules/main.c.
>
> So we're left with having to fix atexit to support subinterpreters. I wonder
> how difficult that will be.

If the problem really has to do with interpreter startup, then there
actually is a workaround possible, at the cost of slightly hairier
code.

If dispatch() looked in the registry *first* and avoided the cache in
that case, and lazily created the cache (including the weakref
import), then interpreter startup would not trigger an import of
weakref in the default case.

(Of course, depending on whether site/sitecustomize results in the use
of importer subclasses and such, this might not help.  It might be
necessary to take the even more complex tack of avoiding the use of a
cache entirely until an ABC is registered, and walking mro's.)

Anyway, it remains to be seen whether these workarounds are easier or
more difficult than fixing the atexit problem.  ;-)

Hmm...  actually, there are a couple other ways around this.
singledispatch doesn't use finalize(), so it doesn't really need
atexit.  It doesn't even do much with WeakKeyDictionary, so it could
actually just "from _weakref import ref", and inline the relevant
operations.

Or, WeakKeyDictionary could be pulled out into a separate module,
where singledispatch could pull it from without importing finalize.

Or, weakref.finalize could be fixed so that the atexit import and
register() are deferred until actual use.

(Of all of these, that last one actually sounds like the least
invasive workaround, with fewest lines of code likely to be changed.)

From stephen at xemacs.org  Sat Jun  8 07:18:15 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 08 Jun 2013 14:18:15 +0900
Subject: [Python-Dev]  doctest and pickle
In-Reply-To: <51B204F7.4030107@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
Message-ID: <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:

 > Enumerations can be pickled and unpickled::
 > 
 >      >>> from enum import Enum
 >      >>> class Fruit(Enum):
 >      ...     tomato = 1
 >      ...     banana = 2
 >      ...     cherry = 3
 >      ...
 >      >>> from pickle import dumps, loads
 >      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
 >      True
 > [...]
 > Still, it would be nice if this could work.

Well, you could cheat and reverse the test. ;-)

I assume the problem is that loads proceeds to recreate the Fruit
enum, rather than checking if there already is one?  Maybe the
metaclass can check somehow?  At the very least, if this can't work in
this implementation, unpickling should be an error.


From steve at pearwood.info  Sat Jun  8 08:45:49 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 08 Jun 2013 16:45:49 +1000
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <51B2D31D.9080106@pearwood.info>

On 08/06/13 15:18, Stephen J. Turnbull wrote:
> Ethan Furman writes:
>
>   > Enumerations can be pickled and unpickled::
>   >
>   >      >>> from enum import Enum
>   >      >>> class Fruit(Enum):
>   >      ...     tomato = 1
>   >      ...     banana = 2
>   >      ...     cherry = 3
>   >      ...
>   >      >>> from pickle import dumps, loads
>   >      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>   >      True
>   > [...]
>   > Still, it would be nice if this could work.
>
> Well, you could cheat and reverse the test. ;-)
>
> I assume the problem is that loads proceeds to recreate the Fruit
> enum, rather than checking if there already is one?


I don't believe so. I understand that the problem is that pickle cannot find the Fruit enum in the __main__ module.

Untested, but adding this before the call to dumps might work:

import __main__
__main__.Fruit = Fruit


although that's the sort of thing that makes me think it's time to turn this into a unittest rather than a doctest.



-- 
Steven

From ethan at stoneleaf.us  Sat Jun  8 09:03:14 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 08 Jun 2013 00:03:14 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B2D31D.9080106@pearwood.info>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info>
Message-ID: <51B2D732.40405@stoneleaf.us>

On 06/07/2013 11:45 PM, Steven D'Aprano wrote:
> On 08/06/13 15:18, Stephen J. Turnbull wrote:
>> Ethan Furman writes:
>>
>>   > Enumerations can be pickled and unpickled::
>>   >
>>   >      >>> from enum import Enum
>>   >      >>> class Fruit(Enum):
>>   >      ...     tomato = 1
>>   >      ...     banana = 2
>>   >      ...     cherry = 3
>>   >      ...
>>   >      >>> from pickle import dumps, loads
>>   >      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>   >      True
>>   > [...]
>>   > Still, it would be nice if this could work.
>>
>> Well, you could cheat and reverse the test. ;-)
>>
>> I assume the problem is that loads proceeds to recreate the Fruit
>> enum, rather than checking if there already is one?
>
>
> I don't believe so. I understand that the problem is that pickle cannot find the Fruit enum in the __main__ module.
>
> Untested, but adding this before the call to dumps might work:
>
> import __main__
> __main__.Fruit = Fruit
>
>
> although that's the sort of thing that makes me think it's time to turn this into a unittest rather than a doctest.

Indeed, and it is already in several different ways.  But it would be nice to have a pickle example in the docs that 
worked with doctest.

I ended up doing what Barry did:

     >>> from test.test_enum import Fruit
     >>> from pickle import dumps, loads
     >>> Fruit.tomato is loads(dumps(Fruit.tomato))
     True

--
~Ethan~

From storchaka at gmail.com  Sat Jun  8 09:45:45 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 10:45:45 +0300
Subject: [Python-Dev] Obsoleted RFCs
Message-ID: <kounf5$6ir$1@ger.gmane.org>

Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, 
*.py, *.c and *.h files. I think it would be worthwhile to update the 
source code and documentation for more modern RFCs.

For example for updating RFC3548 to RFC4648 there is an issue #16995.
-------------- next part --------------
821: Simple Mail Transfer Protocol. (Obsoleted by RFC2821)
    Lib/smtpd.py
    Lib/smtplib.py
822: STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES. (Obsoleted by RFC2822)
    Doc/library/email-examples.rst
    Doc/library/email.rst
    Doc/library/imaplib.rst
    Lib/configparser.py
    Lib/email/_header_value_parser.py
    Lib/email/_parseaddr.py
    Lib/email/header.py
    Lib/imaplib.py
    Lib/test/test_email/data/msg_16.txt
    Lib/test/test_email/test_email.py
    Lib/test/test_http_cookiejar.py
850: Standard for interchange of USENET messages. (Obsoleted by RFC1036)
    Lib/email/_parseaddr.py
977: Network News Transfer Protocol. (Obsoleted by RFC3977)
    Lib/nntplib.py
    Lib/test/test_nntplib.py
1036: Standard for interchange of USENET messages. (Obsoleted by RFC5536, RFC5537)
    rfcuse.txt
1521: MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049)
    Lib/base64.py
    Lib/quopri.py
1522: MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049)
    Doc/library/binascii.rst
    Lib/quopri.py
1738: Uniform Resource Locators (URL). (Obsoleted by RFC4248, RFC4266)
    Lib/urllib/parse.py
1750: Randomness Recommendations for Security. (Obsoleted by RFC4086)
    Doc/library/ssl.rst
    Modules/_ssl.c
1766: Tags for the Identification of Languages. (Obsoleted by RFC3066, RFC3282)
    Lib/locale.py
1808: Relative Uniform Resource Locators. (Obsoleted by RFC3986)
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
1869: SMTP Service Extensions. (Obsoleted by RFC2821)
    Lib/smtpd.py
    Lib/smtplib.py
1894: An Extensible Message Format for Delivery Status Notifications. (Obsoleted by RFC3464)
    Lib/test/test_email/test_email.py
2048: Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures. (Obsoleted by RFC4288, RFC4289)
    rfcuse.txt
2060: Internet Message Access Protocol - Version 4rev1. (Obsoleted by RFC3501)
    Lib/imaplib.py
2068: Hypertext Transfer Protocol -- HTTP/1.1. (Obsoleted by RFC2616)
    Lib/http/cookies.py
    Lib/urllib/request.py
2069: An Extension to HTTP : Digest Access Authentication. (Obsoleted by RFC2617)
    Lib/urllib/request.py
2070: Internationalization of the Hypertext Markup Language. (Obsoleted by RFC2854)
    Lib/html/entities.py
2109: HTTP State Management Mechanism. (Obsoleted by RFC2965)
    Doc/library/http.cookiejar.rst
    Lib/http/cookiejar.py
    Lib/http/cookies.py
    Lib/test/test_http_cookiejar.py
2133: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC2553)
    Modules/getaddrinfo.c
    Modules/getnameinfo.c
2292: Advanced Sockets API for IPv6. (Obsoleted by RFC3542)
    Modules/socketmodule.c
2373: IP Version 6 Addressing Architecture. (Obsoleted by RFC3513)
    Lib/ipaddress.py
2396: Uniform Resource Identifiers (URI): Generic Syntax. (Obsoleted by RFC3986)
    Lib/http/cookiejar.py
    Lib/test/test_urllib.py
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
2434: Guidelines for Writing an IANA Considerations Section in RFCs. (Obsoleted by RFC5226)
    rfc3454.txt
2440: OpenPGP Message Format. (Obsoleted by RFC4880)
    Lib/test/test_email/data/msg_45.txt
2487: SMTP Service Extension for Secure SMTP over TLS. (Obsoleted by RFC3207)
    Lib/smtplib.py
2518: HTTP Extensions for Distributed Authoring -- WEBDAV. (Obsoleted by RFC4918)
    Doc/library/http.client.rst
2553: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC3493)
    Modules/addrinfo.h
    Modules/socketmodule.c
    rfcuse.txt
2554: SMTP Service Extension for Authentication. (Obsoleted by RFC4954)
    Lib/smtplib.py
2718: Guidelines for new URL Schemes. (Obsoleted by RFC4395)
    Lib/http/cookiejar.py
2732: Format for Literal IPv6 Addresses in URL's. (Obsoleted by RFC3986)
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
2821: Simple Mail Transfer Protocol. (Obsoleted by RFC5321)
    Lib/smtplib.py
    rfcuse.txt
2822: Internet Message Format. (Obsoleted by RFC5322)
    Doc/tutorial/stdlib.rst
    Lib/email/_header_value_parser.py
    Lib/email/_parseaddr.py
    Lib/email/feedparser.py
    Lib/email/generator.py
    Lib/email/header.py
    Lib/email/message.py
    Lib/email/mime/message.py
    Lib/email/parser.py
    Lib/email/utils.py
    Lib/http/client.py
    Lib/smtplib.py
    Lib/test/test_email/data/msg_35.txt
    Lib/test/test_email/test_email.py
    rfcuse.txt
3066: Tags for the Identification of Languages. (Obsoleted by RFC4646, RFC4647)
    rfcuse.txt
3171: IANA Guidelines for IPv4 Multicast Address Assignments. (Obsoleted by RFC5771)
    Lib/ipaddress.py
3280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. (Obsoleted by RFC5280)
    Doc/library/ssl.rst
3330: Special-Use IPv4 Addresses. (Obsoleted by RFC5735)
    Lib/ipaddress.py
3513: Internet Protocol Version 6 (IPv6) Addressing Architecture. (Obsoleted by RFC4291)
    Lib/ipaddress.py
    rfcuse.txt
3548: The Base16, Base32, and Base64 Data Encodings. (Obsoleted by RFC4648)
    Doc/library/base64.rst
    Doc/whatsnew/2.4.rst
    Lib/base64.py
4366: Transport Layer Security (TLS) Extensions. (Obsoleted by RFC5246)
    Doc/library/ssl.rst

From storchaka at gmail.com  Sat Jun  8 10:07:11 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 11:07:11 +0300
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B2D732.40405@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
Message-ID: <kouonc$goh$1@ger.gmane.org>

08.06.13 10:03, Ethan Furman ???????(??):
> Indeed, and it is already in several different ways.  But it would be
> nice to have a pickle example in the docs that worked with doctest.
>
> I ended up doing what Barry did:
>
>      >>> from test.test_enum import Fruit
>      >>> from pickle import dumps, loads
>      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>      True

I think that the documentation is there for people. If you need tests, 
add them separately, but the documentation should be clear and 
understandable. In this case it is better to exclude a code example from 
doctests or add auxiliary code (i.e. as Steven suggested) to pass the 
doctest.



From benjamin at python.org  Sat Jun  8 10:23:25 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 8 Jun 2013 01:23:25 -0700
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kounf5$6ir$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org>
Message-ID: <CAPZV6o9d4OYX_BEUGEX165K1PuusO1NAPfT+0=aPsgBTLeUG+g@mail.gmail.com>

2013/6/8 Serhiy Storchaka <storchaka at gmail.com>:
> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt,
> *.py, *.c and *.h files. I think it would be worthwhile to update the source
> code and documentation for more modern RFCs.

Just because you change the reference, doesn't mean the code is
automatically compliant with the updated RFC. :)

>
> For example for updating RFC3548 to RFC4648 there is an issue #16995.




--
Regards,
Benjamin

From mal at egenix.com  Sat Jun  8 10:42:21 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 08 Jun 2013 10:42:21 +0200
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kounf5$6ir$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org>
Message-ID: <51B2EE6D.2020101@egenix.com>

On 08.06.2013 09:45, Serhiy Storchaka wrote:
> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I
> think it would be worthwhile to update the source code and documentation for more modern RFCs.

Thanks for creating such a list.

BTW: What is rfcuse.txt that's mentioned several times in the list ?

> For example for updating RFC3548 to RFC4648 there is an issue #16995.

Given that more recent RFCs tend to introduce new functionality and
sometimes backwards incompatible changes, I think each RFC update would
need to be handled in a separate ticket.

Some updates could probably be done in one go, e.g. RFC 821 -> 2821 ->
5321

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 08 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2013-07-01: EuroPython 2013, Florence, Italy ...           23 days to go
2013-07-16: Python Meeting Duesseldorf ...                 38 days to go

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From storchaka at gmail.com  Sat Jun  8 10:59:03 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 11:59:03 +0300
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <CAPZV6o9d4OYX_BEUGEX165K1PuusO1NAPfT+0=aPsgBTLeUG+g@mail.gmail.com>
References: <kounf5$6ir$1@ger.gmane.org>
	<CAPZV6o9d4OYX_BEUGEX165K1PuusO1NAPfT+0=aPsgBTLeUG+g@mail.gmail.com>
Message-ID: <kourol$aop$1@ger.gmane.org>

08.06.13 11:23, Benjamin Peterson ???????(??):
> 2013/6/8 Serhiy Storchaka <storchaka at gmail.com>:
>> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt,
>> *.py, *.c and *.h files. I think it would be worthwhile to update the source
>> code and documentation for more modern RFCs.
>
> Just because you change the reference, doesn't mean the code is
> automatically compliant with the updated RFC. :)

Of course. Maintainers should review his modules and conclude what 
should be made for supporting more modern RFCs.

I'm surprised that even new ipaddress module uses obsoleted RFC.


From ethan at stoneleaf.us  Sat Jun  8 10:47:35 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 08 Jun 2013 01:47:35 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <kouonc$goh$1@ger.gmane.org>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org>
Message-ID: <51B2EFA7.6060703@stoneleaf.us>

On 06/08/2013 01:07 AM, Serhiy Storchaka wrote:
> 08.06.13 10:03, Ethan Furman ???????(??):
>> Indeed, and it is already in several different ways.  But it would be
>> nice to have a pickle example in the docs that worked with doctest.
>>
>> I ended up doing what Barry did:
>>
>>      >>> from test.test_enum import Fruit
>>      >>> from pickle import dumps, loads
>>      >>> Fruit.tomato is loads(dumps(Fruit.tomato))
>>      True
>
> I think that the documentation is there for people.

I agree.

>If you need tests, add them separately,

I have.

> but the documentation should be clear and understandable.

And the example code should be testable.

> In this case it is better to exclude a code example from doctests or
> add auxiliary code (i.e. as Steven suggested) to pass the doctest.

Are you saying there is something wrong about what I have in place now?  I would think that one line showing something 
you might actually do (importing an Enum from another module) is better than two lines showing esoteric workarounds 
(importing __main__ and setting an attribute on it).

Apologies if I sound gruff -- it's way past my bedtime.  In fact, I'll think I'll go sleep now.  :)

--
~Ethan~

From storchaka at gmail.com  Sat Jun  8 11:51:23 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 12:51:23 +0300
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <51B2EE6D.2020101@egenix.com>
References: <kounf5$6ir$1@ger.gmane.org> <51B2EE6D.2020101@egenix.com>
Message-ID: <kouuqt$5bv$1@ger.gmane.org>

08.06.13 11:42, M.-A. Lemburg ???????(??):
> On 08.06.2013 09:45, Serhiy Storchaka wrote:
>> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I
>> think it would be worthwhile to update the source code and documentation for more modern RFCs.
>
> Thanks for creating such a list.
>
> BTW: What is rfcuse.txt that's mentioned several times in the list ?

Oh, sorry. It is here by mistake. Just ignore it.

>> For example for updating RFC3548 to RFC4648 there is an issue #16995.
>
> Given that more recent RFCs tend to introduce new functionality and
> sometimes backwards incompatible changes, I think each RFC update would
> need to be handled in a separate ticket.
>
> Some updates could probably be done in one go, e.g. RFC 821 -> 2821 ->
> 5321

Of course. This list is only a start point.



From storchaka at gmail.com  Sat Jun  8 12:00:27 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 13:00:27 +0300
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kounf5$6ir$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org>
Message-ID: <kouvbv$9tf$1@ger.gmane.org>

By mistake some local files were added to the list. Here's an updated 
list. It now also contains low-case references.

Attached also a script used to generate this list.
-------------- next part --------------
3: Documentation conventions. (Obsoleted by RFC0010)
    Lib/test/math_testcases.txt
10: Documentation conventions. (Obsoleted by RFC0016)
    Lib/test/math_testcases.txt
11: Implementation of the Host - Host Software Procedures in GORDO. (Obsoleted by RFC0033)
    Lib/test/math_testcases.txt
821: Simple Mail Transfer Protocol. (Obsoleted by RFC2821)
    Lib/smtpd.py
    Lib/smtplib.py
822: STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES. (Obsoleted by RFC2822)
    Doc/distutils/apiref.rst
    Doc/library/email-examples.rst
    Doc/library/email.iterators.rst
    Doc/library/email.message.rst
    Doc/library/email.mime.rst
    Doc/library/email.parser.rst
    Doc/library/email.rst
    Doc/library/imaplib.rst
    Doc/whatsnew/2.2.rst
    Doc/whatsnew/2.4.rst
    Lib/configparser.py
    Lib/distutils/dist.py
    Lib/distutils/tests/test_util.py
    Lib/distutils/util.py
    Lib/email/_header_value_parser.py
    Lib/email/_parseaddr.py
    Lib/email/feedparser.py
    Lib/email/generator.py
    Lib/email/header.py
    Lib/email/message.py
    Lib/email/mime/message.py
    Lib/email/utils.py
    Lib/imaplib.py
    Lib/mimetypes.py
    Lib/test/test_email/data/msg_05.txt
    Lib/test/test_email/data/msg_06.txt
    Lib/test/test_email/data/msg_11.txt
    Lib/test/test_email/data/msg_16.txt
    Lib/test/test_email/data/msg_25.txt
    Lib/test/test_email/data/msg_28.txt
    Lib/test/test_email/data/msg_42.txt
    Lib/test/test_email/data/msg_43.txt
    Lib/test/test_email/data/msg_46.txt
    Lib/test/test_email/test_defect_handling.py
    Lib/test/test_email/test_email.py
    Lib/test/test_email/torture_test.py
    Lib/test/test_http_cookiejar.py
    Tools/scripts/mailerdaemon.py
850: Standard for interchange of USENET messages. (Obsoleted by RFC1036)
    Lib/email/_parseaddr.py
    Lib/http/cookiejar.py
    Lib/test/test_http_cookiejar.py
931: Authentication server. (Obsoleted by RFC1413)
    Lib/http/server.py
977: Network News Transfer Protocol. (Obsoleted by RFC3977)
    Lib/nntplib.py
    Lib/test/test_nntplib.py
1521: MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049)
    Lib/base64.py
    Lib/quopri.py
    Modules/binascii.c
1522: MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049)
    Doc/library/binascii.rst
    Lib/quopri.py
1738: Uniform Resource Locators (URL). (Obsoleted by RFC4248, RFC4266)
    Lib/urllib/parse.py
1750: Randomness Recommendations for Security. (Obsoleted by RFC4086)
    Doc/library/ssl.rst
    Modules/_ssl.c
1766: Tags for the Identification of Languages. (Obsoleted by RFC3066, RFC3282)
    Lib/locale.py
1808: Relative Uniform Resource Locators. (Obsoleted by RFC3986)
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
1869: SMTP Service Extensions. (Obsoleted by RFC2821)
    Lib/smtpd.py
    Lib/smtplib.py
1894: An Extensible Message Format for Delivery Status Notifications. (Obsoleted by RFC3464)
    Lib/test/test_email/test_email.py
2060: Internet Message Access Protocol - Version 4rev1. (Obsoleted by RFC3501)
    Lib/imaplib.py
2068: Hypertext Transfer Protocol -- HTTP/1.1. (Obsoleted by RFC2616)
    Lib/http/cookies.py
    Lib/urllib/request.py
2069: An Extension to HTTP : Digest Access Authentication. (Obsoleted by RFC2617)
    Lib/urllib/request.py
2070: Internationalization of the Hypertext Markup Language. (Obsoleted by RFC2854)
    Lib/html/entities.py
2109: HTTP State Management Mechanism. (Obsoleted by RFC2965)
    Doc/library/http.cookiejar.rst
    Lib/http/cookiejar.py
    Lib/http/cookies.py
    Lib/test/test_http_cookiejar.py
2133: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC2553)
    Modules/getaddrinfo.c
    Modules/getnameinfo.c
2279: UTF-8, a transformation format of ISO 10646. (Obsoleted by RFC3629)
    Lib/test/test_unicode.py
2292: Advanced Sockets API for IPv6. (Obsoleted by RFC3542)
    Modules/socketmodule.c
2368: The mailto URL scheme. (Obsoleted by RFC6068)
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
2373: IP Version 6 Addressing Architecture. (Obsoleted by RFC3513)
    Lib/ipaddress.py
2396: Uniform Resource Identifiers (URI): Generic Syntax. (Obsoleted by RFC3986)
    Lib/http/cookiejar.py
    Lib/test/test_urllib.py
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
2440: OpenPGP Message Format. (Obsoleted by RFC4880)
    Lib/test/test_email/data/msg_45.txt
2487: SMTP Service Extension for Secure SMTP over TLS. (Obsoleted by RFC3207)
    Lib/smtplib.py
2518: HTTP Extensions for Distributed Authoring -- WEBDAV. (Obsoleted by RFC4918)
    Doc/library/http.client.rst
2553: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC3493)
    Modules/addrinfo.h
    Modules/socketmodule.c
2554: SMTP Service Extension for Authentication. (Obsoleted by RFC4954)
    Lib/smtplib.py
2718: Guidelines for new URL Schemes. (Obsoleted by RFC4395)
    Lib/http/cookiejar.py
2732: Format for Literal IPv6 Addresses in URL's. (Obsoleted by RFC3986)
    Lib/test/test_urlparse.py
    Lib/urllib/parse.py
2821: Simple Mail Transfer Protocol. (Obsoleted by RFC5321)
    Lib/smtplib.py
2822: Internet Message Format. (Obsoleted by RFC5322)
    Doc/tutorial/stdlib.rst
    Doc/whatsnew/2.2.rst
    Lib/email/_header_value_parser.py
    Lib/email/_parseaddr.py
    Lib/email/feedparser.py
    Lib/email/generator.py
    Lib/email/header.py
    Lib/email/message.py
    Lib/email/mime/message.py
    Lib/email/parser.py
    Lib/email/utils.py
    Lib/http/client.py
    Lib/smtplib.py
    Lib/test/test_email/data/msg_35.txt
    Lib/test/test_email/test_email.py
2965: HTTP State Management Mechanism. (Obsoleted by RFC6265)
    Doc/library/http.cookiejar.rst
    Doc/library/urllib.request.rst
    Lib/http/cookiejar.py
    Lib/test/test_http_cookiejar.py
    Lib/urllib/request.py
3171: IANA Guidelines for IPv4 Multicast Address Assignments. (Obsoleted by RFC5771)
    Lib/ipaddress.py
3280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. (Obsoleted by RFC5280)
    Doc/library/ssl.rst
3330: Special-Use IPv4 Addresses. (Obsoleted by RFC5735)
    Lib/ipaddress.py
3490: Internationalizing Domain Names in Applications (IDNA). (Obsoleted by RFC5890, RFC5891)
    Doc/library/codecs.rst
3513: Internet Protocol Version 6 (IPv6) Addressing Architecture. (Obsoleted by RFC4291)
    Lib/ipaddress.py
3548: The Base16, Base32, and Base64 Data Encodings. (Obsoleted by RFC4648)
    Doc/library/base64.rst
    Doc/whatsnew/2.4.rst
    Lib/base64.py
4366: Transport Layer Security (TLS) Extensions. (Obsoleted by RFC5246, RFC6066)
    Doc/library/ssl.rst
5735: Special Use IPv4 Addresses. (Obsoleted by RFC6890)
    Lib/ipaddress.py
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rfc_obsoleted.py
Type: text/x-python
Size: 985 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130608/1c4ac21e/attachment.py>

From storchaka at gmail.com  Sat Jun  8 12:09:21 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 08 Jun 2013 13:09:21 +0300
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B2EFA7.6060703@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
Message-ID: <kouvsb$ebm$1@ger.gmane.org>

08.06.13 11:47, Ethan Furman ???????(??):
>> In this case it is better to exclude a code example from doctests or
>> add auxiliary code (i.e. as Steven suggested) to pass the doctest.
>
> Are you saying there is something wrong about what I have in place now?
> I would think that one line showing something you might actually do
> (importing an Enum from another module) is better than two lines showing
> esoteric workarounds (importing __main__ and setting an attribute on it).

test.test_enum is not here. The reader should look into the external 
test module (which may not be supplied along with the module and 
documentation) to understand the example. Or rely on assumptions.

Is it possible to add "invisible" code which doesn't displayed in the 
resulting documentation, but taken into account by doctest?



From stefan at drees.name  Sat Jun  8 12:36:45 2013
From: stefan at drees.name (Stefan Drees)
Date: Sat, 08 Jun 2013 12:36:45 +0200
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kourol$aop$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org>
	<CAPZV6o9d4OYX_BEUGEX165K1PuusO1NAPfT+0=aPsgBTLeUG+g@mail.gmail.com>
	<kourol$aop$1@ger.gmane.org>
Message-ID: <51B3093D.6060001@drees.name>

On 2013-06-08 10:59, Serhiy Storchaka wrote:
> 08.06.13 11:23, Benjamin Peterson ... :
>> 2013/6/8 Serhiy Storchaka ... :
>>> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt,
>>> *.py, *.c and *.h files. I think it would be worthwhile to update the
>>> source
>>> code and documentation for more modern RFCs.
>>
>> Just because you change the reference, doesn't mean the code is
>> automatically compliant with the updated RFC. :)
>
> Of course. Maintainers should review his modules and conclude what
> should be made for supporting more modern RFCs.
>
> I'm surprised that even new ipaddress module uses obsoleted RFC.

I for one am not :-) Let's pick that one ... as it seems to depend on 
the perspective chosen (web docs vs. python code comments) (cf. Details)

Summary:

Besides the magic, that lies in the process that transforms the 
documentation inside the python source into the web presented form, it 
should be easy and semantically correct, to simply replace the two 
occurences of "RFC 3513 2.5.6" with "RFC 4291 2.5.7".


Details:

The current web doc (3.4.0a0) at

http://docs.python.org/dev/library/ipaddress#ipaddress.IPv6Address

only shows this (no RFC3513 in sight):

"""
The following constitutes a valid IPv6 address:

1. A string consisting of ... See RFC 4291 for details. ...
"""

the current source at Lib/ipaddress.py you checked has:

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

class IPv6Network(_BaseV6, _BaseNetwork):
     # ... snipped methods, reformatted, shortened ...

     @property
     def is_site_local(self):
         """Test if the address is reserved for site-local.

         Note that the site-local address space has been
         deprecated by RFC 3879.
         Use is_private to test if this address is in the
         space of unique local addresses as defined by RFC 4193.

         Returns:
             A boolean, True if the ad... is res... per RFC 3513 2.5.6.

         """
         sitelocal_network = IPv6Network('fec0::/10')
         return self in sitelocal_network

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

and later:

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

class IPv6Network(_BaseV6, _BaseNetwork):
     # ... again snipped methods, reformatted, shortened ...

     @property
     def is_site_local(self):
         """Test if the address is reserved for site-local.

         Note that the site-local address space has been
         deprecated by RFC 3879.
         Use is_private to test if this address is in the
         space of unique local addresses as defined by RFC 4193.

         Returns:
             A boolean, True if the ad... is res... per RFC 3513 2.5.6.

         """
         return (self.network_address.is_site_local and
                 self.broadcast_address.is_site_local)

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

The RFC 3513 (April 2003) has been obsoleted by RFC 4291 (February 2006) 
and the latter has been updated by RFC 5952 (August 2010) and RFC 6052 
(October 2010).

The given motivation for referencing inside the python source file is in 
both cases the detailed specification of what a reserved address 
actaully is.

Looking at RFC 3513 has at section 2.5.6 (the referenced one):

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

2.5.6 Local-Use IPv6 Unicast Addresses

    There are two types of local-use unicast addresses defined.  These
    are Link-Local and Site-Local.  The Link-Local is for use on a single
    link and the Site-Local is for use in a single site.  Link-Local
    addresses have the following format:

    |   10     |
    |  bits    |         54 bits         |          64 bits           |
    +----------+-------------------------+----------------------------+
    |1111111010|           0             |       interface ID         |
    +----------+-------------------------+----------------------------+

    Link-Local addresses are designed to be used for addressing on a
    single link for purposes such as automatic address configuration,
    neighbor discovery, or when no routers are present.

    Routers must not forward any packets with link-local source or
    destination addresses to other links.

    Site-Local addresses have the following format:

    |   10     |
    |  bits    |         54 bits         |         64 bits            |
    +----------+-------------------------+----------------------------+
    |1111111011|        subnet ID        |       interface ID         |
    +----------+-------------------------+----------------------------+

    Site-local addresses are designed to be used for addressing inside of
    a site without the need for a global prefix.  Although a subnet ID
    may be up to 54-bits long, it is expected that globally-connected
    sites will use the same subnet IDs for site-local and global
    prefixes.

    Routers must not forward any packets with site-local source or
    destination addresses outside of the site.

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

The obsoleting RFC 4291 has separated the description of Link-Local and 
Site-Local IPv6 Unicast adresses, thus if the section 2.5.6 reference 
for the is_site_local method was correct from the start - which seems to 
be a quite reasonable assumption - then the new section level reference 
should be "RFC 4291 2.5.7":

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

2.5.7.  Site-Local IPv6 Unicast Addresses

    Site-Local addresses were originally designed to be used for
    addressing inside of a site without the need for a global prefix.
    Site-local addresses are now deprecated as defined in [SLDEP].

    Site-Local addresses have the following format:

    |   10     |
    |  bits    |         54 bits         |         64 bits            |
    +----------+-------------------------+----------------------------+
    |1111111011|        subnet ID        |       interface ID         |
    +----------+-------------------------+----------------------------+

    The special behavior of this prefix defined in [RFC3513] must no
    longer be supported in new implementations (i.e., new implementations
    must treat this prefix as Global Unicast).

    Existing implementations and deployments may continue to use this
    prefix.

# --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ----

But there seems to be a problem when actually using Site-Local 
addresses, as the new RFC (deviating from the obsoleted one) states 
(c.f. above):

"""
Site-local addresses are now deprecated as defined in [SLDEP].
"""

Otherwise as a format reference all is well.

The updateing RFCs seem to not interfere with the detail pointed to (by 
trusting the relation between title and content :-):

[RFC5952] "A Recommendation for IPv6 Address Text Representation"
[RFC6052] "IPv6 Addressing of IPv4/IPv6 Translators"




All the best,
Stefan.

From rdmurray at bitdance.com  Sat Jun  8 16:47:18 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 08 Jun 2013 10:47:18 -0400
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kouuqt$5bv$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org> <51B2EE6D.2020101@egenix.com>
	<kouuqt$5bv$1@ger.gmane.org>
Message-ID: <20130608144719.1B092250BF2@webabinitio.net>

On Sat, 08 Jun 2013 12:51:23 +0300, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 08.06.13 11:42, M.-A. Lemburg ??????????????(????):
> > On 08.06.2013 09:45, Serhiy Storchaka wrote:
> >> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I
> >> think it would be worthwhile to update the source code and documentation for more modern RFCs.
> >
> > Given that more recent RFCs tend to introduce new functionality and
> > sometimes backwards incompatible changes, I think each RFC update would
> > need to be handled in a separate ticket.
> >
> > Some updates could probably be done in one go, e.g. RFC 821 -> 2821 ->
> > 5321
> 
> Of course. This list is only a start point.

We've already done some (most?) of the work for the smtplib update.
A thorough review is needed to make sure there aren't any leftovers
and to update any comments and docs that haven't been fixed yet.

For the email package, the support for RFC 5322 is mostly done, in
the new policies.  The legacy policies will continue to only support
the older RFCs formally, though in many cases the code is also
conformant with the newer RFCs because the relevant details have
not changed.

--David

From ethan at stoneleaf.us  Sat Jun  8 17:41:00 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 08 Jun 2013 08:41:00 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <kouvsb$ebm$1@ger.gmane.org>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org>
Message-ID: <51B3508C.5000905@stoneleaf.us>

On 06/08/2013 03:09 AM, Serhiy Storchaka wrote:
> 08.06.13 11:47, Ethan Furman ???????(??):
>>> In this case it is better to exclude a code example from doctests or
>>> add auxiliary code (i.e. as Steven suggested) to pass the doctest.
>>
>> Are you saying there is something wrong about what I have in place now?
>> I would think that one line showing something you might actually do
>> (importing an Enum from another module) is better than two lines showing
>> esoteric workarounds (importing __main__ and setting an attribute on it).
>
> test.test_enum is not here. The reader should look into the external test module (which may not be supplied along with
> the module and documentation) to understand the example. Or rely on assumptions.

Fair point.  But I suppose that if the end-user is running a doc test, it is not too much to require that the other 
tests be installed as well.  Plus, we definitely have the other tests.  :)


> Is it possible to add "invisible" code which doesn't displayed in the resulting documentation, but taken into account by
> doctest?

I have no idea.  This is my first time using doctest.

--
~Ethan~

From stephen at xemacs.org  Sat Jun  8 18:18:47 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 09 Jun 2013 01:18:47 +0900
Subject: [Python-Dev] Obsoleted RFCs
In-Reply-To: <kourol$aop$1@ger.gmane.org>
References: <kounf5$6ir$1@ger.gmane.org>
	<CAPZV6o9d4OYX_BEUGEX165K1PuusO1NAPfT+0=aPsgBTLeUG+g@mail.gmail.com>
	<kourol$aop$1@ger.gmane.org>
Message-ID: <87wqq48tjs.fsf@uwakimon.sk.tsukuba.ac.jp>

Serhiy Storchaka writes:
 > 08.06.13 11:23, Benjamin Peterson ???????(??):
 > > 2013/6/8 Serhiy Storchaka <storchaka at gmail.com>:
 > >> Here is attached a list of obsoleted RFCs referred in the *.rst,
 > >> *.txt, *.py, *.c and *.h files. I think it would be worthwhile
 > >> to update the source code and documentation for more modern
 > >> RFCs.
 > >
 > > Just because you change the reference, doesn't mean the code is
 > > automatically compliant with the updated RFC. :)
 > 
 > Of course. Maintainers should review his modules and conclude what 
 > should be made for supporting more modern RFCs.

I suspect in many cases the answer is going to be "nothing".  Grepping
out the references and checking for obsoleted RFCs is useful
information, of course.  Good GSoC fodder, for one thing.  But I'd be
cautious about even creating an issue without consideration of each
case individually.

This can be a *lot* of work, for very little gain.  In the case of
mail, consider that STD 11 still points to RFC 822![1]  Also, even the
most modern RFC 5322 REQUIREs parsers to accept the "obsolete" syntax
of section 4, which I believe is that of RFC 822.  In any case, it's
pretty close.  So you wouldn't want to change the parser anyway.
Whether it would be worth auditing the generative functions for 5322
conformance, and creating tests, is a more difficult question, but it
still sounds like much work for little gain.

The analysis is surely different for other RFCs, but for this
particular series I see little harm in letting each component of the
email module continue to explicitly target whichever of the RFCs
happened to be current when its author started coding.


Footnotes: 
[1]  http://tools.ietf.org/html/std11


From olemis at gmail.com  Sat Jun  8 19:17:41 2013
From: olemis at gmail.com (Olemis Lang)
Date: Sat, 8 Jun 2013 12:17:41 -0500
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B3508C.5000905@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org> <51B3508C.5000905@stoneleaf.us>
Message-ID: <CAGMZAuPtWHRyY3-jSLZc9tDRCbXHX1v3QMebpksxOFU98qeLCw@mail.gmail.com>

On 6/8/13, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/08/2013 03:09 AM, Serhiy Storchaka wrote:
>> 08.06.13 11:47, Ethan Furman ???????(??):
[...]
>
> Fair point.  But I suppose that if the end-user is running a doc test, it is
> not too much to require that the other
> tests be installed as well.  Plus, we definitely have the other tests.  :)
>
>
>> Is it possible to add "invisible" code which doesn't displayed in the
>> resulting documentation, but taken into account by
>> doctest?
>
> I have no idea.  This is my first time using doctest.
>

<ot>
No ... if using doctest . To get such things done you'll need
something like dutest [1]_ [2]_ , but that's not an option for testing
docs in stdlib .
</ot>

.. [1] dutest @ PyPI
        (https://pypi.python.org/pypi/dutest?)

.. [2] Purpose of Doctests [Was: Best practices for Enum]
        (http://goo.gl/F7Afy)

-- 
Regards,

Olemis.

Apache? Bloodhound contributor
http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

From lrekucki at gmail.com  Sat Jun  8 19:54:18 2013
From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=)
Date: Sat, 8 Jun 2013 19:54:18 +0200
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <51B3508C.5000905@stoneleaf.us>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org> <51B3508C.5000905@stoneleaf.us>
Message-ID: <CAEZs-E+xnwe04d2v6KFQy4Pgr9n9hY=5BfXZ4EE7Q1=FagWukw@mail.gmail.com>

On 8 June 2013 17:41, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/08/2013 03:09 AM, Serhiy Storchaka wrote:
>>
>> Is it possible to add "invisible" code which doesn't displayed in the
>> resulting documentation, but taken into account by
>> doctest?
>
>
> I have no idea.  This is my first time using doctest.
>

AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At
least if you're running them via Sphinx.

[1]: http://sphinx-doc.org/ext/doctest.html

--
?ukasz Rekucki

From rdmurray at bitdance.com  Sat Jun  8 20:17:14 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 08 Jun 2013 14:17:14 -0400
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CAEZs-E+xnwe04d2v6KFQy4Pgr9n9hY=5BfXZ4EE7Q1=FagWukw@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org> <51B3508C.5000905@stoneleaf.us>
	<CAEZs-E+xnwe04d2v6KFQy4Pgr9n9hY=5BfXZ4EE7Q1=FagWukw@mail.gmail.com>
Message-ID: <20130608181714.B3112250BF3@webabinitio.net>

On Sat, 08 Jun 2013 19:54:18 +0200, =?UTF-8?Q?=C5=81ukasz_Rekucki?= <lrekucki at gmail.com> wrote:
> On 8 June 2013 17:41, Ethan Furman <ethan at stoneleaf.us> wrote:
> > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote:
> >>
> >> Is it possible to add "invisible" code which doesn't displayed in the
> >> resulting documentation, but taken into account by
> >> doctest?
> >
> >
> > I have no idea.  This is my first time using doctest.
> >
> 
> AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At
> least if you're running them via Sphinx.

Running the doctests via Sphinx is still, I believe a Future Objective :).
I think we are getting close to being able to do that, though.  (Well,
you already can for Python2; but not, I think, for Python3).

If the test module from which you are importing is short, you could use
the Sphinx literalinclude directive to include the contents of the file
in the docs just before you show the 'is' example.  You could also explain
(briefly) *why* it is necessary to use an imported class for this example,
which IMO is useful information.

--David

From lukasz at langa.pl  Sat Jun  8 23:49:22 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sat, 8 Jun 2013 23:49:22 +0200
Subject: [Python-Dev] Unicode minus sign in numeric conversions
Message-ID: <E6E425D3-160C-4ED8-B40D-9FE23FACDB64@langa.pl>

Expected behaviour:
>>> float('\N{MINUS SIGN}12.34')
-12.34


Current behaviour:
Traceback (most recent call last):
...
ValueError: could not convert string to float: '?12.34'


Please note: '\N{MINUS SIGN}' == '\u2212'

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev


From guido at python.org  Sun Jun  9 00:30:52 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 8 Jun 2013 15:30:52 -0700
Subject: [Python-Dev] Unicode minus sign in numeric conversions
In-Reply-To: <E6E425D3-160C-4ED8-B40D-9FE23FACDB64@langa.pl>
References: <E6E425D3-160C-4ED8-B40D-9FE23FACDB64@langa.pl>
Message-ID: <CAP7+vJK-FnNa7QX7Lqd1fKKpCTTa1s1MF4b1A+4C5dLmTCR4cw@mail.gmail.com>

[Diverting to python-ideas, since this isn't as clear-cut as you think.]

Why exactly is that expected behavior? What's the use case? (Surely
you don't have a keyboard that generates \u2212 when you hit the minus
key? :-)

Is there a Unicode standard for parsing numbers? IIRC there are a
variety of other things marked as "digits" in the Unicode standard --
do we do anything with those? If we do anything we should be
consistent. For now, I think we *are* consistent -- we only support
the ASCII representation of numbers. (And that's the only
representation we generate as output as well -- think about symmetry
too.)

This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode

--Guido

On Sat, Jun 8, 2013 at 2:49 PM, ?ukasz Langa <lukasz at langa.pl> wrote:
> Expected behaviour:
>>>> float('\N{MINUS SIGN}12.34')
> -12.34
>
>
> Current behaviour:
> Traceback (most recent call last):
> ...
> ValueError: could not convert string to float: '?12.34'
>
>
> Please note: '\N{MINUS SIGN}' == '\u2212'
>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Sun Jun  9 02:39:59 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 09 Jun 2013 01:39:59 +0100
Subject: [Python-Dev] Unicode minus sign in numeric conversions
In-Reply-To: <CAP7+vJK-FnNa7QX7Lqd1fKKpCTTa1s1MF4b1A+4C5dLmTCR4cw@mail.gmail.com>
References: <E6E425D3-160C-4ED8-B40D-9FE23FACDB64@langa.pl>
	<CAP7+vJK-FnNa7QX7Lqd1fKKpCTTa1s1MF4b1A+4C5dLmTCR4cw@mail.gmail.com>
Message-ID: <51B3CEDF.4050106@mrabarnett.plus.com>

On 08/06/2013 23:30, Guido van Rossum wrote:
> [Diverting to python-ideas, since this isn't as clear-cut as you think.]
>
> Why exactly is that expected behavior? What's the use case? (Surely
> you don't have a keyboard that generates \u2212 when you hit the minus
> key? :-)
>
> Is there a Unicode standard for parsing numbers? IIRC there are a
> variety of other things marked as "digits" in the Unicode standard --
> do we do anything with those? If we do anything we should be
> consistent. For now, I think we *are* consistent -- we only support
> the ASCII representation of numbers. (And that's the only
> representation we generate as output as well -- think about symmetry
> too.)
>
We already recognise at least some of the digits:

 >>> float("\N{ARABIC-INDIC DIGIT ONE}")
1.0

(I haven't check all of them!)

> This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode
>
> --Guido
>
> On Sat, Jun 8, 2013 at 2:49 PM, ?ukasz Langa <lukasz at langa.pl> wrote:
>> Expected behaviour:
>>>>> float('\N{MINUS SIGN}12.34')
>> -12.34
>>
>>
>> Current behaviour:
>> Traceback (most recent call last):
>> ...
>> ValueError: could not convert string to float: '?12.34'
>>
>>
>> Please note: '\N{MINUS SIGN}' == '\u2212'
>>


From rdmurray at bitdance.com  Sun Jun  9 03:46:38 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 08 Jun 2013 21:46:38 -0400
Subject: [Python-Dev] Unicode minus sign in numeric conversions
In-Reply-To: <51B3CEDF.4050106@mrabarnett.plus.com>
References: <E6E425D3-160C-4ED8-B40D-9FE23FACDB64@langa.pl>
	<CAP7+vJK-FnNa7QX7Lqd1fKKpCTTa1s1MF4b1A+4C5dLmTCR4cw@mail.gmail.com>
	<51B3CEDF.4050106@mrabarnett.plus.com>
Message-ID: <20130609014639.46E14250BF3@webabinitio.net>

On Sun, 09 Jun 2013 01:39:59 +0100, MRAB <python at mrabarnett.plus.com> wrote:
> On 08/06/2013 23:30, Guido van Rossum wrote:
> > [Diverting to python-ideas, since this isn't as clear-cut as you think.]
> >
> > Why exactly is that expected behavior? What's the use case? (Surely
> > you don't have a keyboard that generates \u2212 when you hit the minus
> > key? :-)
> >
> > Is there a Unicode standard for parsing numbers? IIRC there are a
> > variety of other things marked as "digits" in the Unicode standard --
> > do we do anything with those? If we do anything we should be
> > consistent. For now, I think we *are* consistent -- we only support
> > the ASCII representation of numbers. (And that's the only
> > representation we generate as output as well -- think about symmetry
> > too.)
> >
> We already recognise at least some of the digits:
> 
>  >>> float("\N{ARABIC-INDIC DIGIT ONE}")
> 1.0
> 
> (I haven't check all of them!)
> 
> > This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode

http://bugs.python.org/issue6632 contains a bunch of good information 
relevant to this discussion.  It looks like the argument there was
that there is no standard for the signs, therefore we should not
support them.

As Guido said, the issue is non-trivial.

--David

From ncoghlan at gmail.com  Sun Jun  9 06:21:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 9 Jun 2013 14:21:02 +1000
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <20130608181714.B3112250BF3@webabinitio.net>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org> <51B3508C.5000905@stoneleaf.us>
	<CAEZs-E+xnwe04d2v6KFQy4Pgr9n9hY=5BfXZ4EE7Q1=FagWukw@mail.gmail.com>
	<20130608181714.B3112250BF3@webabinitio.net>
Message-ID: <CADiSq7eiFpgusdsgxbXOs1fC==hYUjcv8-UNXDnbazt=5Uzv7Q@mail.gmail.com>

On 9 June 2013 04:17, R. David Murray <rdmurray at bitdance.com> wrote:
> On Sat, 08 Jun 2013 19:54:18 +0200, =?UTF-8?Q?=C5=81ukasz_Rekucki?= <lrekucki at gmail.com> wrote:
>> On 8 June 2013 17:41, Ethan Furman <ethan at stoneleaf.us> wrote:
>> > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote:
>> >>
>> >> Is it possible to add "invisible" code which doesn't displayed in the
>> >> resulting documentation, but taken into account by
>> >> doctest?
>> >
>> >
>> > I have no idea.  This is my first time using doctest.
>> >
>>
>> AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At
>> least if you're running them via Sphinx.
>
> Running the doctests via Sphinx is still, I believe a Future Objective :).
> I think we are getting close to being able to do that, though.  (Well,
> you already can for Python2; but not, I think, for Python3).
>
> If the test module from which you are importing is short, you could use
> the Sphinx literalinclude directive to include the contents of the file
> in the docs just before you show the 'is' example.  You could also explain
> (briefly) *why* it is necessary to use an imported class for this example,
> which IMO is useful information.

Right - for the singleton behaviour of enums to be honoured by pickle,
the enum definition *must* be reachable through the import system
based on its __name__ and __module__ attributes (PEP 3154 will
eventually change that limitation to __qualname__ and __module__). The
temporary namespaces created to run doctests generally aren't
reachable that way, so the unpickling either fails or creates a new
instance (depending on the details of the situation).

Using the test suite in the enum docstrings initially is fine. In the
future, once we migrate a module like socket to using enum.IntEnum
instead of bare integers, it would be appropriate to change the enum
docs to reference that rather than the test suite.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sun Jun  9 06:47:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 08 Jun 2013 21:47:29 -0700
Subject: [Python-Dev] doctest and pickle
In-Reply-To: <CADiSq7eiFpgusdsgxbXOs1fC==hYUjcv8-UNXDnbazt=5Uzv7Q@mail.gmail.com>
References: <51B204F7.4030107@stoneleaf.us>
	<87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp>
	<51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us>
	<kouonc$goh$1@ger.gmane.org> <51B2EFA7.6060703@stoneleaf.us>
	<kouvsb$ebm$1@ger.gmane.org> <51B3508C.5000905@stoneleaf.us>
	<CAEZs-E+xnwe04d2v6KFQy4Pgr9n9hY=5BfXZ4EE7Q1=FagWukw@mail.gmail.com>
	<20130608181714.B3112250BF3@webabinitio.net>
	<CADiSq7eiFpgusdsgxbXOs1fC==hYUjcv8-UNXDnbazt=5Uzv7Q@mail.gmail.com>
Message-ID: <51B408E1.4090508@stoneleaf.us>

On 06/08/2013 09:21 PM, Nick Coghlan wrote:
>
> Using the test suite in the enum docstrings initially is fine. In the
> future, once we migrate a module like socket to using enum.IntEnum
> instead of bare integers, it would be appropriate to change the enum
> docs to reference that rather than the test suite.

Good point.  I'll keep that in mind.

--
~Ethan~

From alexander.belopolsky at gmail.com  Mon Jun 10 05:13:43 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Sun, 9 Jun 2013 23:13:43 -0400
Subject: [Python-Dev] Ordering keyword dicts
In-Reply-To: <CAP7+vJJ+Np-rijxvg6qgOeRCQ5LpYN+U1bcywRYvLQDQa2VJQg@mail.gmail.com>
References: <CAP7+vJJ+Np-rijxvg6qgOeRCQ5LpYN+U1bcywRYvLQDQa2VJQg@mail.gmail.com>
Message-ID: <CAP7h-xbSz1KfiarD5HDYt+5NzjpacKhLr28hVdO+cdsdNCkPPQ@mail.gmail.com>

On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum <guido at python.org> wrote:

> I'm slow at warming up to the idea. My main concern is speed -- since
> most code doesn't need it and function calls are already slow (and
> obviously very common :-) it would be a shame if this slowed down
> function calls that don't need it noticeably.
>

Here is an idea that will not affect functions that don't need to know the
order of keywords: a special __kworder__ local variable.  The use of this
variable inside the function will signal compiler to generate additional
bytecode to copy keyword names from the stack to a tuple and save it in
__kworder__.    With that feature, an OrderedDict constructor, for example
can be written as

def odict(**kwargs):
      return OrderedDict([(key, kwargs[key]) for key in __kworder__])
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130609/f09b6403/attachment.html>

From ron3200 at gmail.com  Mon Jun 10 17:08:46 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 10 Jun 2013 10:08:46 -0500
Subject: [Python-Dev] Ordering keyword dicts
In-Reply-To: <CAP7h-xbSz1KfiarD5HDYt+5NzjpacKhLr28hVdO+cdsdNCkPPQ@mail.gmail.com>
References: <CAP7+vJJ+Np-rijxvg6qgOeRCQ5LpYN+U1bcywRYvLQDQa2VJQg@mail.gmail.com>
	<CAP7h-xbSz1KfiarD5HDYt+5NzjpacKhLr28hVdO+cdsdNCkPPQ@mail.gmail.com>
Message-ID: <51B5EBFE.7080605@gmail.com>



On 06/09/2013 10:13 PM, Alexander Belopolsky wrote:
>
> On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum <guido at python.org
> <mailto:guido at python.org>> wrote:
>
>     I'm slow at warming up to the idea. My main concern is speed -- since
>     most code doesn't need it and function calls are already slow (and
>     obviously very common :-) it would be a shame if this slowed down
>     function calls that don't need it noticeably.


And we could remove this bit from the docs...

"""
The OrderedDict constructor and update() method both accept keyword
arguments, but their order is lost because Python?s function call semantics
pass-in keyword arguments using a regular unordered dictionary.
"""


> Here is an idea that will not affect functions that don't need to know the
> order of keywords: a special __kworder__ local variable.  The use of this
> variable inside the function will signal compiler to generate additional
> bytecode to copy keyword names from the stack to a tuple and save it in
> __kworder__.

I like the idea of an ordered dict acting the same as a regular dict with 
an optional way to access order.  It also will catch uses of unordered 
dicts in situations where an ordered dict is expected by having the ordered 
key words accessed in a way that isn't available in unordered dicts.


I don't care for a magic local variable inside of a function.


> With that feature, an OrderedDict constructor, for example
> can be written as
 >
> def odict(**kwargs):
>        return OrderedDict([(key, kwargs[key]) for key in __kworder__])


There is two situations where this would matter...  The packing of 
arguments when **kwargs is used in the function definition, and the 
unpacking of arguments when **kwargs is used in the function call.

By having the unpackng side also work with ordered dicts, maybe we could 
then replace the many *very common* occurrences of (*args, **kwargs) with 
just (**all_args) or (***all_args).  :)

It's the unordered nature of dictionaries that requires *args to be used 
along with **kwargs when forwarding function arguments to other functions.


Another variation of this is to have a way to forward a functions 
"signature name space" to another function directly and bypass the 
signature parsing those cases.

Cheers,
    Ron















From ron3200 at gmail.com  Mon Jun 10 17:08:46 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 10 Jun 2013 10:08:46 -0500
Subject: [Python-Dev] Ordering keyword dicts
In-Reply-To: <CAP7h-xbSz1KfiarD5HDYt+5NzjpacKhLr28hVdO+cdsdNCkPPQ@mail.gmail.com>
References: <CAP7+vJJ+Np-rijxvg6qgOeRCQ5LpYN+U1bcywRYvLQDQa2VJQg@mail.gmail.com>
	<CAP7h-xbSz1KfiarD5HDYt+5NzjpacKhLr28hVdO+cdsdNCkPPQ@mail.gmail.com>
Message-ID: <51B5EBFE.7080605@gmail.com>



On 06/09/2013 10:13 PM, Alexander Belopolsky wrote:
>
> On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum <guido at python.org
> <mailto:guido at python.org>> wrote:
>
>     I'm slow at warming up to the idea. My main concern is speed -- since
>     most code doesn't need it and function calls are already slow (and
>     obviously very common :-) it would be a shame if this slowed down
>     function calls that don't need it noticeably.


And we could remove this bit from the docs...

"""
The OrderedDict constructor and update() method both accept keyword
arguments, but their order is lost because Python?s function call semantics
pass-in keyword arguments using a regular unordered dictionary.
"""


> Here is an idea that will not affect functions that don't need to know the
> order of keywords: a special __kworder__ local variable.  The use of this
> variable inside the function will signal compiler to generate additional
> bytecode to copy keyword names from the stack to a tuple and save it in
> __kworder__.

I like the idea of an ordered dict acting the same as a regular dict with 
an optional way to access order.  It also will catch uses of unordered 
dicts in situations where an ordered dict is expected by having the ordered 
key words accessed in a way that isn't available in unordered dicts.


I don't care for a magic local variable inside of a function.


> With that feature, an OrderedDict constructor, for example
> can be written as
 >
> def odict(**kwargs):
>        return OrderedDict([(key, kwargs[key]) for key in __kworder__])


There is two situations where this would matter...  The packing of 
arguments when **kwargs is used in the function definition, and the 
unpacking of arguments when **kwargs is used in the function call.

By having the unpackng side also work with ordered dicts, maybe we could 
then replace the many *very common* occurrences of (*args, **kwargs) with 
just (**all_args) or (***all_args).  :)

It's the unordered nature of dictionaries that requires *args to be used 
along with **kwargs when forwarding function arguments to other functions.


Another variation of this is to have a way to forward a functions 
"signature name space" to another function directly and bypass the 
signature parsing those cases.

Cheers,
    Ron
















From ethan at stoneleaf.us  Mon Jun 10 21:02:30 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 10 Jun 2013 12:02:30 -0700
Subject: [Python-Dev] Python 3 metaclasses, doctest, and unittest
Message-ID: <51B622C6.7060209@stoneleaf.us>

I just wanted to take a minute and say THANK YOU to everyone involved in getting Python 3 to where it is today.

It is so much easier to use, especially metaclasses (nearly pulled my hair out trying to get Enum working on Py2!).

Also a big thank you to the doctest folks as it was invaluable in getting my Py2 examples correct, and, of course, the 
unittest folks as I would have been lost without those.

Thank you thank you thank you.

--
~Ethan~

P.S.  I'll get a backported package on PyPI after the code commit for 3.4.

From jas at corpus-callosum.com  Tue Jun 11 07:57:57 2013
From: jas at corpus-callosum.com (Jeff Sickel)
Date: Tue, 11 Jun 2013 00:57:57 -0500
Subject: [Python-Dev] new Plan 9 (and family) port of v2.7.5
Message-ID: <FF044C62-D66C-42BE-B72C-35386D1796CA@corpus-callosum.com>

I have a set of patches I've developed based on the 2.7 branch that provides a working port of Python 2.7.5 to Plan 9 386 and amd64 builds (an arm version is mostly working, but will need a few more updates before being hauled off on the wagon).  Most of the changes, including mkfiles, are local to a Plan9 subdirectory.  There are a few changes to Lib/*.py files specifically site.py and sysconfig.py to make living in a semi-POSIX world tolerable with many current Python modules.

Because Plan 9 does not have shared libraries, we must link in all the C modules we want at build time.  This means that I've included specific targets to pull in Mercurial sources during the compilation and linking phases. That brings us up to a current version of Mercurial 2.6.2.  There is no ssh support for cloning, but we do have a viable version of https: using TLS/SSLv3 that has worked well with bitbucket.  That said, I can prepare a patch from my mq work area and submit it upstream in one of various ways for review.

Is there a preferred method that does not require an initial `hg clone ssh://hg at hg.python.org/cpython`?

acme# pwd
/usr/jas/src/cmd/cpython
acme# hg sum
parent: 83814:75a6bbf41cc8 plan9 qbase qtip
 update setup base install locations
branch: 2.7
commit: 8 unknown (clean)
update: 37 new changesets, 2 branch heads (merge)
mq:     1 applied
ac


-jas


From brett at python.org  Tue Jun 11 19:25:09 2013
From: brett at python.org (Brett Cannon)
Date: Tue, 11 Jun 2013 13:25:09 -0400
Subject: [Python-Dev] new Plan 9 (and family) port of v2.7.5
In-Reply-To: <FF044C62-D66C-42BE-B72C-35386D1796CA@corpus-callosum.com>
References: <FF044C62-D66C-42BE-B72C-35386D1796CA@corpus-callosum.com>
Message-ID: <CAP1=2W5eqvqMrJqvfUXpnZzpwKAjqGGtTT1whec6V9inQoboQQ@mail.gmail.com>

On Tue, Jun 11, 2013 at 1:57 AM, Jeff Sickel <jas at corpus-callosum.com>wrote:

> I have a set of patches I've developed based on the 2.7 branch that
> provides a working port of Python 2.7.5 to Plan 9 386 and amd64 builds (an
> arm version is mostly working, but will need a few more updates before
> being hauled off on the wagon).  Most of the changes, including mkfiles,
> are local to a Plan9 subdirectory.  There are a few changes to Lib/*.py
> files specifically site.py and sysconfig.py to make living in a semi-POSIX
> world tolerable with many current Python modules.
>
> Because Plan 9 does not have shared libraries, we must link in all the C
> modules we want at build time.  This means that I've included specific
> targets to pull in Mercurial sources during the compilation and linking
> phases. That brings us up to a current version of Mercurial 2.6.2.  There
> is no ssh support for cloning, but we do have a viable version of https:
> using TLS/SSLv3 that has worked well with bitbucket.  That said, I can
> prepare a patch from my mq work area and submit it upstream in one of
> various ways for review.
>

We have moved away from trying to support non-mainstream OSs directly in
the code base and prefer people to maintain a patched branch elsewhere.
That being said, if any of the changes are reasonable in general feel free
to submit patches at bugs.python.org.


>
> Is there a preferred method that does not require an initial `hg clone
> ssh://hg at hg.python.org/cpython`?
>
>
Not quite sure what you mean, but I assume you are asking about submitting
patches, in which case you can use an HTTPS clone. SSH access is only for
core developers to push changes to the main repo.

-Brett



> acme# pwd
> /usr/jas/src/cmd/cpython
> acme# hg sum
> parent: 83814:75a6bbf41cc8 plan9 qbase qtip
>  update setup base install locations
> branch: 2.7
> commit: 8 unknown (clean)
> update: 37 new changesets, 2 branch heads (merge)
> mq:     1 applied
> ac
>
>
> -jas
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130611/56467d81/attachment.html>

From skip at pobox.com  Tue Jun 11 21:10:41 2013
From: skip at pobox.com (Skip Montanaro)
Date: Tue, 11 Jun 2013 14:10:41 -0500
Subject: [Python-Dev] Doubly linked lists in Python core?
Message-ID: <CANc-5Uzti4_T_-cqPCdG+YVxGEj6qxp4Jked0daPL7VM_2oDYw@mail.gmail.com>

I encountered this disconcerting message yesterday on a Linux system
running Python 2.7.2:

*** glibc detected *** /opt/local/bin/python: corrupted double-linked
list: 0x0000000003b01c90 ***

Of course, no core file or other information about where the problem
occurred was left behind, just the raw pointer value.  Running under
valgrind didn't report anything obvious, at least nothing different
than a run with slightly different command line parameters which
didn't elicit the message.

I suspect the problem actually lies in one of our C++ libraries here
at work, but before I attempt any debugging gymnastics (ow! my back
hurts) I thought I would check to see if Python makes much use of
double linked lists in a non-debug build.  I don't recall a lot from
days of yore, perhaps some in the dict implementation?

Thanks,

Skip

From benjamin at python.org  Tue Jun 11 21:14:55 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 11 Jun 2013 12:14:55 -0700
Subject: [Python-Dev] Doubly linked lists in Python core?
In-Reply-To: <CANc-5Uzti4_T_-cqPCdG+YVxGEj6qxp4Jked0daPL7VM_2oDYw@mail.gmail.com>
References: <CANc-5Uzti4_T_-cqPCdG+YVxGEj6qxp4Jked0daPL7VM_2oDYw@mail.gmail.com>
Message-ID: <CAPZV6o9p=BifOWJ6hK+rygj69i7=mMwAR0YCnZy3=ac_OwWBBw@mail.gmail.com>

2013/6/11 Skip Montanaro <skip at pobox.com>:
> I encountered this disconcerting message yesterday on a Linux system
> running Python 2.7.2:
>
> *** glibc detected *** /opt/local/bin/python: corrupted double-linked
> list: 0x0000000003b01c90 ***

I suspect that's a corrupt linked list interal to glibc.


--
Regards,
Benjamin

From dmalcolm at redhat.com  Tue Jun 11 21:38:47 2013
From: dmalcolm at redhat.com (David Malcolm)
Date: Tue, 11 Jun 2013 15:38:47 -0400
Subject: [Python-Dev] Doubly linked lists in Python core?
In-Reply-To: <CAPZV6o9p=BifOWJ6hK+rygj69i7=mMwAR0YCnZy3=ac_OwWBBw@mail.gmail.com>
References: <CANc-5Uzti4_T_-cqPCdG+YVxGEj6qxp4Jked0daPL7VM_2oDYw@mail.gmail.com>
	<CAPZV6o9p=BifOWJ6hK+rygj69i7=mMwAR0YCnZy3=ac_OwWBBw@mail.gmail.com>
Message-ID: <1370979527.28429.39.camel@surprise>

On Tue, 2013-06-11 at 12:14 -0700, Benjamin Peterson wrote:
> 2013/6/11 Skip Montanaro <skip at pobox.com>:
> > I encountered this disconcerting message yesterday on a Linux system
> > running Python 2.7.2:
> >
> > *** glibc detected *** /opt/local/bin/python: corrupted double-linked
> > list: 0x0000000003b01c90 ***
> 
> I suspect that's a corrupt linked list interal to glibc.

Yes: almost certainly the one inside glibc's implementation of malloc.

Somewhere in the process you have a double-free, or a buffer overrun
that's splatting the links that live in memory between the allocated
bufffers.

You may want to try running the process under valgrind.

Hope this is helpful
Dave


From skip at pobox.com  Tue Jun 11 21:49:06 2013
From: skip at pobox.com (Skip Montanaro)
Date: Tue, 11 Jun 2013 14:49:06 -0500
Subject: [Python-Dev] Doubly linked lists in Python core?
In-Reply-To: <1370979527.28429.39.camel@surprise>
References: <CANc-5Uzti4_T_-cqPCdG+YVxGEj6qxp4Jked0daPL7VM_2oDYw@mail.gmail.com>
	<CAPZV6o9p=BifOWJ6hK+rygj69i7=mMwAR0YCnZy3=ac_OwWBBw@mail.gmail.com>
	<1370979527.28429.39.camel@surprise>
Message-ID: <CANc-5UyEs2OmCPpRNdXy2N6F0y5JoAaRJ-rjF0R9Y6C=5efRhg@mail.gmail.com>

> You may want to try running the process under valgrind.

Thanks.  I'm trying that but have been so far unable to get valgrind
to report any problems within Python or our libraries before that
message, just a couple things at startup which seem to occur before
Python gets going:

==21374== Invalid read of size 1
==21374==    at 0x4C2DEA9: putenv (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==21374==    by 0x400E71: append_env_vars (binary-wrapper-main.c:175)
==21374==    by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58)
==21374==    by 0x40168D: main (binary-wrapper-main.c:382)
==21374==  Address 0x51d6a84 is 0 bytes after a block of size 52 alloc'd
==21374==    at 0x4C2ACCE: realloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==21374==    by 0x400E61: append_env_vars (binary-wrapper-main.c:173)
==21374==    by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58)
==21374==    by 0x40168D: main (binary-wrapper-main.c:382)
==21374==
==21374== Syscall param execve(envp[i]) points to unaddressable byte(s)
==21374==    at 0x4EE7067: execve (in /lib64/libc-2.15.so)
==21374==    by 0x4016AA: main (binary-wrapper-main.c:385)
==21374==  Address 0x51d6a84 is 0 bytes after a block of size 52 alloc'd
==21374==    at 0x4C2ACCE: realloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==21374==    by 0x400E61: append_env_vars (binary-wrapper-main.c:173)
==21374==    by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58)
==21374==    by 0x40168D: main (binary-wrapper-main.c:382)

Skip

From victor.stinner at gmail.com  Thu Jun 13 01:06:30 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 13 Jun 2013 01:06:30 +0200
Subject: [Python-Dev] Enhancement of Python memory allocators
Message-ID: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>

Hi,

I would like to improve memory allocators of Python. My two use cases
are replacing memory allocators with custom allocators in embedded
system and hooking allocators to track usage of memory.

I wrote a patch for this, I'm going to commit it if nobody complains:
http://bugs.python.org/issue3329

Using this patch, detecting memory corruptions (buffer underflow and
overflow) can be done without recompilation. We may add an environment
variable to enable Python debug functions at runtime, example:
PYDEBUGMALLOC=1. There is just a restriction: the environment variable
would not be ignored with -E command line option, because command line
options are parsed after the first memory allocation. What do you
think?

*****

The patch adds the following functions:

void PyMem_GetAllocators(
    void **ctx_p,
    void* (**malloc_p) (void *ctx, size_t size),
    void* (**realloc_p) (void *ctx, void *ptr, size_t size),
    void (**free_p) (void *ctx, void *ptr));

void PyMem_SetAllocators(
    void *ctx,
    void* (*malloc) (void *ctx, size_t size),
    void* (*realloc) (void *ctx, void *ptr, size_t size),
    void (*free) (void *ctx, void *ptr));

It adds 4 similar functions (get/set) for PyObject_Malloc() and
allocators of pymalloc arenas.

*****

For the "track usage of memory" use case, see the following project
which hooks memory allocators using PyMem_SetAllocators() and
PyObject_SetAllocators() to get allocated bytes per filename and line
number.
https://pypi.python.org/pypi/pytracemalloc

*****

Another issue proposes to use VirtualAlloc() and VirtualFree() for
pymalloc arenas, see:
http://bugs.python.org/issue13483

I don't know if it would be interesting, but it would now possible to
choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc,
...) at runtime, with an environment variable for example.

Victor

From ncoghlan at gmail.com  Thu Jun 13 01:21:42 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Jun 2013 09:21:42 +1000
Subject: [Python-Dev] Enhancement of Python memory allocators
In-Reply-To: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>
References: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>
Message-ID: <CADiSq7fRc=1C9P6cVCpmeD9SFAgVF9ndn1xqFNAeOEDnXp9gbA@mail.gmail.com>

On 13 Jun 2013 09:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> Hi,
>
> I would like to improve memory allocators of Python. My two use cases
> are replacing memory allocators with custom allocators in embedded
> system and hooking allocators to track usage of memory.
>
> I wrote a patch for this, I'm going to commit it if nobody complains:
> http://bugs.python.org/issue3329
>
> Using this patch, detecting memory corruptions (buffer underflow and
> overflow) can be done without recompilation. We may add an environment
> variable to enable Python debug functions at runtime, example:
> PYDEBUGMALLOC=1. There is just a restriction: the environment variable
> would not be ignored with -E command line option, because command line
> options are parsed after the first memory allocation. What do you
> think?

The rest of it sounds fine, but please don't add the runtime switching
support to our existing main function. Interpreter startup is a mess
already. If you were interested in helping directly with PEP 432, though,
that would be good - I haven't been able to spend much time on it lately.

Cheers,
Nick.

>
> *****
>
> The patch adds the following functions:
>
> void PyMem_GetAllocators(
>     void **ctx_p,
>     void* (**malloc_p) (void *ctx, size_t size),
>     void* (**realloc_p) (void *ctx, void *ptr, size_t size),
>     void (**free_p) (void *ctx, void *ptr));
>
> void PyMem_SetAllocators(
>     void *ctx,
>     void* (*malloc) (void *ctx, size_t size),
>     void* (*realloc) (void *ctx, void *ptr, size_t size),
>     void (*free) (void *ctx, void *ptr));
>
> It adds 4 similar functions (get/set) for PyObject_Malloc() and
> allocators of pymalloc arenas.
>
> *****
>
> For the "track usage of memory" use case, see the following project
> which hooks memory allocators using PyMem_SetAllocators() and
> PyObject_SetAllocators() to get allocated bytes per filename and line
> number.
> https://pypi.python.org/pypi/pytracemalloc
>
> *****
>
> Another issue proposes to use VirtualAlloc() and VirtualFree() for
> pymalloc arenas, see:
> http://bugs.python.org/issue13483
>
> I don't know if it would be interesting, but it would now possible to
> choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc,
> ...) at runtime, with an environment variable for example.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130613/76c7f867/attachment.html>

From victor.stinner at gmail.com  Thu Jun 13 02:09:18 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 13 Jun 2013 02:09:18 +0200
Subject: [Python-Dev] Enhancement of Python memory allocators
In-Reply-To: <CADiSq7fRc=1C9P6cVCpmeD9SFAgVF9ndn1xqFNAeOEDnXp9gbA@mail.gmail.com>
References: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>
	<CADiSq7fRc=1C9P6cVCpmeD9SFAgVF9ndn1xqFNAeOEDnXp9gbA@mail.gmail.com>
Message-ID: <CAMpsgwa+gUEDGgvPwFuQefS8-kTQ3ijABKG39PzpyfNEww6FSg@mail.gmail.com>

2013/6/13 Nick Coghlan <ncoghlan at gmail.com>:
> On 13 Jun 2013 09:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>> Using this patch, detecting memory corruptions (buffer underflow and
>> overflow) can be done without recompilation. We may add an environment
>> variable to enable Python debug functions at runtime, example:
>> PYDEBUGMALLOC=1. There is just a restriction: the environment variable
>> would not be ignored with -E command line option, because command line
>> options are parsed after the first memory allocation. What do you
>> think?
>
> The rest of it sounds fine, but please don't add the runtime switching
> support to our existing main function. Interpreter startup is a mess
> already. If you were interested in helping directly with PEP 432, though,
> that would be good - I haven't been able to spend much time on it lately.

I proposed an environment variable to solve the following issue: when
memory allocators are replaced with custom allocators, debug hooks
cannot be used. Debug hooks must be set before the first memory
allocation.

Another option is to add a new function (ex: PyMem_SetDebugHook()) to
install explicitly debug hooks, so it can be called after
PyMem_SetAllocators() and before the first memory allocation.

Victor

From brett at python.org  Thu Jun 13 02:13:49 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 12 Jun 2013 20:13:49 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Move test___all__ over
 to unittest.main() and use ModuleNotFoundError
In-Reply-To: <3bW54Q4xGQzRrw@mail.python.org>
References: <3bW54Q4xGQzRrw@mail.python.org>
Message-ID: <CAP1=2W5wGOgRcyk0UQ9whZWKbZS00Vw1sHcPYqC41iM-ETT6vQ@mail.gmail.com>

Obviously the commit message is a little misleading since changes I was
about to stage accidentally went in on this change. Sorry about that. Same
basic concept of the changes, just to more modules.


On Wed, Jun 12, 2013 at 8:12 PM, brett.cannon <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/c4d7228421df
> changeset:   84106:c4d7228421df
> user:        Brett Cannon <brett at python.org>
> date:        Wed Jun 12 20:12:30 2013 -0400
> summary:
>   Move test___all__ over to unittest.main() and use ModuleNotFoundError
>
> files:
>   Lib/test/regrtest.py     |  16 ++++++++--------
>   Lib/test/support.py      |  14 +++++++-------
>   Lib/test/test___all__.py |   7 ++-----
>   Lib/xmlrpc/server.py     |   2 +-
>   Lib/zipfile.py           |   6 +++---
>   5 files changed, 21 insertions(+), 24 deletions(-)
>
>
> diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
> --- a/Lib/test/regrtest.py
> +++ b/Lib/test/regrtest.py
> @@ -146,11 +146,11 @@
>
>  try:
>      import threading
> -except ImportError:
> +except ModuleNotFoundError:
>      threading = None
>  try:
>      import multiprocessing.process
> -except ImportError:
> +except ModuleNotFoundError:
>      multiprocessing = None
>
>
> @@ -180,7 +180,7 @@
>  if sys.platform == 'darwin':
>      try:
>          import resource
> -    except ImportError:
> +    except ModuleNotFoundError:
>          pass
>      else:
>          soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
> @@ -571,7 +571,7 @@
>      if findleaks:
>          try:
>              import gc
> -        except ImportError:
> +        except ModuleNotFoundError:
>              print('No GC available, disabling findleaks.')
>              findleaks = False
>          else:
> @@ -692,7 +692,7 @@
>      if use_mp:
>          try:
>              from threading import Thread
> -        except ImportError:
> +        except ModuleNotFoundError:
>              print("Multiprocess option requires thread support")
>              sys.exit(2)
>          from queue import Queue
> @@ -1396,7 +1396,7 @@
>      pic = sys.path_importer_cache.copy()
>      try:
>          import zipimport
> -    except ImportError:
> +    except ModuleNotFoundError:
>          zdc = None # Run unmodified on platforms without zipimport support
>      else:
>          zdc = zipimport._zip_directory_cache.copy()
> @@ -1473,7 +1473,7 @@
>      sys.path_importer_cache.update(pic)
>      try:
>          import zipimport
> -    except ImportError:
> +    except ModuleNotFoundError:
>          pass # Run unmodified on platforms without zipimport support
>      else:
>          zipimport._zip_directory_cache.clear()
> @@ -1510,7 +1510,7 @@
>      doctest.master = None
>      try:
>          import ctypes
> -    except ImportError:
> +    except ModuleNotFoundError:
>          # Don't worry about resetting the cache if ctypes is not supported
>          pass
>      else:
> diff --git a/Lib/test/support.py b/Lib/test/support.py
> --- a/Lib/test/support.py
> +++ b/Lib/test/support.py
> @@ -29,27 +29,27 @@
>
>  try:
>      import _thread, threading
> -except ImportError:
> +except ModuleNotFoundError:
>      _thread = None
>      threading = None
>  try:
>      import multiprocessing.process
> -except ImportError:
> +except ModuleNotFoundError:
>      multiprocessing = None
>
>  try:
>      import zlib
> -except ImportError:
> +except ModuleNotFoundError:
>      zlib = None
>
>  try:
>      import bz2
> -except ImportError:
> +except ModuleNotFoundError:
>      bz2 = None
>
>  try:
>      import lzma
> -except ImportError:
> +except ModuleNotFoundError:
>      lzma = None
>
>  __all__ = [
> @@ -116,7 +116,7 @@
>      with _ignore_deprecated_imports(deprecated):
>          try:
>              return importlib.import_module(name)
> -        except ImportError as msg:
> +        except ModuleNotFoundError as msg:
>              if sys.platform.startswith(tuple(required_on)):
>                  raise
>              raise unittest.SkipTest(str(msg))
> @@ -188,7 +188,7 @@
>                  if not _save_and_block_module(blocked_name, orig_modules):
>                      names_to_remove.append(blocked_name)
>              fresh_module = importlib.import_module(name)
> -        except ImportError:
> +        except ModuleNotFoundError:
>              fresh_module = None
>          finally:
>              for orig_name, module in orig_modules.items():
> diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py
> --- a/Lib/test/test___all__.py
> +++ b/Lib/test/test___all__.py
> @@ -75,7 +75,7 @@
>          try:
>              import rlcompleter
>              import locale
> -        except ImportError:
> +        except ModuleNotFoundError:
>              pass
>          else:
>              locale.setlocale(locale.LC_CTYPE, 'C')
> @@ -113,8 +113,5 @@
>              print('Following modules failed to be imported:',
> failed_imports)
>
>
> -def test_main():
> -    support.run_unittest(AllTest)
> -
>  if __name__ == "__main__":
> -    test_main()
> +    unittest.main()
> diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py
> --- a/Lib/xmlrpc/server.py
> +++ b/Lib/xmlrpc/server.py
> @@ -116,7 +116,7 @@
>  import traceback
>  try:
>      import fcntl
> -except ImportError:
> +except ModuleNotFoundError:
>      fcntl = None
>
>  def resolve_dotted_attribute(obj, attr, allow_dotted_names=True):
> diff --git a/Lib/zipfile.py b/Lib/zipfile.py
> --- a/Lib/zipfile.py
> +++ b/Lib/zipfile.py
> @@ -18,18 +18,18 @@
>  try:
>      import zlib # We may need its compression method
>      crc32 = zlib.crc32
> -except ImportError:
> +except ModuleNotFoundError:
>      zlib = None
>      crc32 = binascii.crc32
>
>  try:
>      import bz2 # We may need its compression method
> -except ImportError:
> +except ModuleNotFoundError:
>      bz2 = None
>
>  try:
>      import lzma # We may need its compression method
> -except ImportError:
> +except ModuleNotFoundError:
>      lzma = None
>
>  __all__ = ["BadZipFile", "BadZipfile", "error",
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130612/28ccaba8/attachment.html>

From ncoghlan at gmail.com  Thu Jun 13 04:40:35 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Jun 2013 12:40:35 +1000
Subject: [Python-Dev] Enhancement of Python memory allocators
In-Reply-To: <CAMpsgwa+gUEDGgvPwFuQefS8-kTQ3ijABKG39PzpyfNEww6FSg@mail.gmail.com>
References: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>
	<CADiSq7fRc=1C9P6cVCpmeD9SFAgVF9ndn1xqFNAeOEDnXp9gbA@mail.gmail.com>
	<CAMpsgwa+gUEDGgvPwFuQefS8-kTQ3ijABKG39PzpyfNEww6FSg@mail.gmail.com>
Message-ID: <CADiSq7ddA2qOEuxN7LEtrgcgFSMfVXtJwNjAy0TvedJJ3JTsYA@mail.gmail.com>

On 13 Jun 2013 10:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2013/6/13 Nick Coghlan <ncoghlan at gmail.com>:
> > On 13 Jun 2013 09:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:
> >> Using this patch, detecting memory corruptions (buffer underflow and
> >> overflow) can be done without recompilation. We may add an environment
> >> variable to enable Python debug functions at runtime, example:
> >> PYDEBUGMALLOC=1. There is just a restriction: the environment variable
> >> would not be ignored with -E command line option, because command line
> >> options are parsed after the first memory allocation. What do you
> >> think?
> >
> > The rest of it sounds fine, but please don't add the runtime switching
> > support to our existing main function. Interpreter startup is a mess
> > already. If you were interested in helping directly with PEP 432,
though,
> > that would be good - I haven't been able to spend much time on it
lately.
>
> I proposed an environment variable to solve the following issue: when
> memory allocators are replaced with custom allocators, debug hooks
> cannot be used. Debug hooks must be set before the first memory
> allocation.
>
> Another option is to add a new function (ex: PyMem_SetDebugHook()) to
> install explicitly debug hooks, so it can be called after
> PyMem_SetAllocators() and before the first memory allocation.

Yes, that sounds better. One of the biggest problems with the current
startup sequence is the way it relies on environment variables for
configuration, which makes life hard for other applications that want to
embed the CPython runtime.

Cheers,
Nick.

>
> Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130613/4cd7580f/attachment.html>

From victor.stinner at gmail.com  Thu Jun 13 13:03:46 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 13 Jun 2013 13:03:46 +0200
Subject: [Python-Dev] Enhancement of Python memory allocators
In-Reply-To: <CADiSq7ddA2qOEuxN7LEtrgcgFSMfVXtJwNjAy0TvedJJ3JTsYA@mail.gmail.com>
References: <CAMpsgwaYw6VVu83izkF+x9Mnn=PTa6GExsZ79hdp_yzVR3HB5g@mail.gmail.com>
	<CADiSq7fRc=1C9P6cVCpmeD9SFAgVF9ndn1xqFNAeOEDnXp9gbA@mail.gmail.com>
	<CAMpsgwa+gUEDGgvPwFuQefS8-kTQ3ijABKG39PzpyfNEww6FSg@mail.gmail.com>
	<CADiSq7ddA2qOEuxN7LEtrgcgFSMfVXtJwNjAy0TvedJJ3JTsYA@mail.gmail.com>
Message-ID: <CAMpsgwYOV6F03ETfgQHo9CQAmrexrJ-p17KQ-MJFoa-Dp0im3Q@mail.gmail.com>

2013/6/13 Nick Coghlan <ncoghlan at gmail.com>:
> Yes, that sounds better. One of the biggest problems with the current
> startup sequence is the way it relies on environment variables for
> configuration, which makes life hard for other applications that want to
> embed the CPython runtime.

I wrote a new patch (attached to issue #3329) adding a new
PyMem_SetupDebugHooks() function. So if an application replaces Python
memory allocator functions, it can still can PyMem_SetupDebugHooks()
to benefit of the Python debug hooks detecting memory bugs. The
function does nothing if hooks are already installed or if Python is
not compiled in debug mode. With this function, the API is now
complete for all use cases.

The PEP 432 helps to configure embedded Python, but the new "Set"
functions (ex: PyMem_SetAllocators) are still needed for my
pytracemalloc module which installs hooks at runtime, when Python is
fully initialized (the hooks can be installed anytime). pytracemalloc
is just an example, you may use PyMem_SetAllocators for other debug or
performance purpose.

With my patch, allocator functions like PyMem_Malloc() are no more
macro, and are always the same function. This helps the stable ABI: C
extension modules do not need to be recompiled to benefit of the debug
hooks ;-)

Victor

From solipsis at pitrou.net  Thu Jun 13 20:56:26 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 13 Jun 2013 20:56:26 +0200
Subject: [Python-Dev] ssl improvements and testing question
References: <kor2u7$h31$1@ger.gmane.org>
Message-ID: <20130613205626.5f379704@fsol>

On Fri, 07 Jun 2013 00:37:01 +0200
Christian Heimes <christian at python.org> wrote:
> 
> I'm also thinking about OCSP support and X509v3 extension support for
> _decode_certificate(). Both are a PITB ... Python has an easier and
> better documented C API.

Yes, OpenSSL's documentation is often a joke, unfortunately.

> Question:
> What's the minimum version of OpenSSL Python 3.4 is going to support?

Judging by the kind of machines we run on, I would say 0.9.7something.
Basically I don't think we should remove any existing #ifdef for 3.4.

> Do
> we have an easy way to compile and link Python against a custom
> installation of OpenSSL or do I have to fiddle around with CPPFLAGS and
> CFLAGS?

You have to fiddle around, sadly.
(and you will also have to fiddle around with LD_LIBRARY_PATH)
If you find a way to improve that, your contribution is much
welcome :-)

Regards

Antoine.



From solipsis at pitrou.net  Thu Jun 13 20:59:13 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 13 Jun 2013 20:59:13 +0200
Subject: [Python-Dev] cpython: fixd refleak
References: <3bTSdr5GRqzRNm@mail.python.org>
Message-ID: <20130613205913.39f389b1@fsol>

On Mon, 10 Jun 2013 10:47:28 +0200 (CEST)
christian.heimes <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/6860263c05b3
> changeset:   84077:6860263c05b3
> user:        Christian Heimes <christian at cheimes.de>
> date:        Mon Jun 10 10:47:22 2013 +0200
> summary:
>   fixd refleak
> 
[...]
> -    return Py_BuildValue("(OOOO)", ofile_env, ofile, odir_env, odir);
> +    if ((tup = PyTuple_New(4)) == NULL) {
> +        goto error;
> +    }
> +    PyTuple_SET_ITEM(tup, 0, ofile_env);
> +    PyTuple_SET_ITEM(tup, 1, ofile);
> +    PyTuple_SET_ITEM(tup, 2, odir_env);
> +    PyTuple_SET_ITEM(tup, 3, odir);
> +    return tup;

How about

    return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir);

?



From christian at python.org  Thu Jun 13 23:39:49 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 13 Jun 2013 23:39:49 +0200
Subject: [Python-Dev] cpython: fixd refleak
In-Reply-To: <20130613205913.39f389b1@fsol>
References: <3bTSdr5GRqzRNm@mail.python.org> <20130613205913.39f389b1@fsol>
Message-ID: <51BA3C25.50506@python.org>

Am 13.06.2013 20:59, schrieb Antoine Pitrou:
> How about
> 
>     return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir);
> 
> ?

Oh right, I forgot about 'N'. The PyArg_Parse*() methods don't have it.

Do you want me to modify the code to use Py_BuildValue()?

Christian

From solipsis at pitrou.net  Thu Jun 13 23:42:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 13 Jun 2013 23:42:21 +0200
Subject: [Python-Dev] cpython: fixd refleak
In-Reply-To: <51BA3C25.50506@python.org>
References: <3bTSdr5GRqzRNm@mail.python.org> <20130613205913.39f389b1@fsol>
	<51BA3C25.50506@python.org>
Message-ID: <20130613234221.171de591@fsol>

On Thu, 13 Jun 2013 23:39:49 +0200
Christian Heimes <christian at python.org> wrote:
> Am 13.06.2013 20:59, schrieb Antoine Pitrou:
> > How about
> > 
> >     return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir);
> > 
> > ?
> 
> Oh right, I forgot about 'N'. The PyArg_Parse*() methods don't have it.
> 
> Do you want me to modify the code to use Py_BuildValue()?

Always better to use the simple form, IMO. Go for it!

Regards

Antoine.

From victor.stinner at gmail.com  Fri Jun 14 00:04:19 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 14 Jun 2013 00:04:19 +0200
Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in
	Python 3.4
Message-ID: <CAMpsgwbcnR7=JgfYTcxVZJrm_qZednDr-YnRO7FoWL4jJ8645w@mail.gmail.com>

Hi,

I would like to remove the "GIL must be held" restriction from
PyMem_Malloc(). In my opinion, the restriction was motived by a bug in
Python, bug fixed by the issue #3329. Let me explain why.


The PyMem_Malloc() function is a thin wrapper to malloc(). It returns
NULL if the size is lager than PY_SSIZE_T_MAX and have a well defined
behaviour for PyMem_Malloc(0) (don't return NULL). So it is surprising
to read in Include/pymem.h:
"The GIL must be held when using these APIs."

The reason is more surprising: in debug mode, PyMem_Malloc() is no
more a thin wrapper to malloc(), but it calls internally
PyObject_Malloc(), the "Python allocator" (called pymalloc). (Many
other checks are done in debug mode, but it's unrelated to my point.)
The problem is that PyObject_Malloc() is not thread-safe, the GIL must
be held.

Short history:

  fb45791150d1 (Mar 23 2002) "gives Python a debug-mode pymalloc"
  f294fdd18b5b (Mar 28 2002) removes the "check API family"
  e16dbf875303 (Apr 22 2002) redirects indirectly PyMem_Malloc() to
PyObject_Malloc() in debug mode
  b6aff7a59803 (Sep 28 2009) reintroduces API checks

So the GIL issue is almost as old as the debug mode for Python memory
allocators.

My patch attached to http://bugs.python.org/issue3329 changes the
design of the debug memory allocators: they are now wrapper (hooks) on
the underlying memory allocator (PyMem: malloc, PyObject: pymalloc),
instead of always redirecting to pymalloc (ex: PyObject_Malloc).

Using my patch, PyMem_Malloc() now always calls malloc(), even in
debug mode. Removing the "GIL must be held" restriction is now safe.
Do you agree?

May it cause backward compatibility issue? PyMem_Malloc() and
PyMem_MALLOC() call malloc(), except if the Python source code was
manually modified. Does this use case concern many developers?


Removing the GIL restriction would help to replace direct calls to
malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an
application would be able to replace memory allocators, and these
allocators would be used "everywhere".
=> see http://bugs.python.org/issue18203

Victor

From solipsis at pitrou.net  Fri Jun 14 10:46:03 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 14 Jun 2013 10:46:03 +0200
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
References: <3bWqYP3sqgzQdN@mail.python.org>
Message-ID: <20130614104603.35f83a9c@fsol>

On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
raymond.hettinger <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/5accb0ac8bfb
> changeset:   84116:5accb0ac8bfb
> branch:      2.7
> parent:      84095:ca8e86711403
> user:        Raymond Hettinger <python at rcn.com>
> date:        Fri Jun 14 01:06:33 2013 -0400
> summary:
>   Fix comment blocks.  Adjust blocksize to a power-of-two for better divmod computations.

Is there any rationale for changing the heuristic from "fits in a whole
number of cachelines" to "allows fast division by the blocksize"?

I personally would prefer if such changes were justified a bit more
than by a one-liner changeset message without any corresponding open
issue.

Regards

Antoine.



From kristjan at ccpgames.com  Fri Jun 14 11:29:17 2013
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Fri, 14 Jun 2013 09:29:17 +0000
Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held
 in	Python 3.4
In-Reply-To: <CAMpsgwbcnR7=JgfYTcxVZJrm_qZednDr-YnRO7FoWL4jJ8645w@mail.gmail.com>
References: <CAMpsgwbcnR7=JgfYTcxVZJrm_qZednDr-YnRO7FoWL4jJ8645w@mail.gmail.com>
Message-ID: <EFE3877620384242A686D52278B7CCD35394B5BE@RKV-IT-EXCH104.ccp.ad.local>


> -----Original Message-----
> I would like to remove the "GIL must be held" restriction from
> PyMem_Malloc(). In my opinion, the restriction was motived by a bug in
> Python, bug fixed by the issue #3329. Let me explain why.
> 
...

> 
> Removing the GIL restriction would help to replace direct calls to
> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
> would be able to replace memory allocators, and these allocators would be
> used "everywhere".
> => see http://bugs.python.org/issue18203
> 

To keep this interesting, I have a somewhat different opinion to Victor :)
  have put comments in the original defect, but would like to repeat them here.
IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details.  Requiring it to be GIL free severely limits what it can do.  For example, it would be forbidden to delegate to PyObject_MALLOC when debugging. 

The approach CCP has taken (we have replaced all raw malloc calls with api calls) is this:
a) Add a "raw" api, PyMem_MALLOC_RAW.  This is guaranteed to be thread safe and call directly to the external memory api of python, as set by Py_SetAllocator()
b) Replace calls to malloc() in the source code with PyMem_MALLOC/PyMem_MALLOC_RAW as appropriate (in our case, using  an include file with #defines to mimimize changes)

There are only two or three places in the source code that require non-GIL protected malloc.  IMHO, requiring PyMem_MALLOC to be threadsafe just to cater to those three places is an overkill, and does more harm than good by limiting our options.

Cheers!
Kristj?n


From storchaka at gmail.com  Fri Jun 14 15:47:50 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 14 Jun 2013 16:47:50 +0300
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <20130614104603.35f83a9c@fsol>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
Message-ID: <kpf6u5$826$1@ger.gmane.org>

14.06.13 11:46, Antoine Pitrou ???????(??):
> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
> raymond.hettinger <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/5accb0ac8bfb
>> changeset:   84116:5accb0ac8bfb
>>    Fix comment blocks.  Adjust blocksize to a power-of-two for better divmod computations.
>
> Is there any rationale for changing the heuristic from "fits in a whole
> number of cachelines" to "allows fast division by the blocksize"?
>
> I personally would prefer if such changes were justified a bit more
> than by a one-liner changeset message without any corresponding open
> issue.

I share the doubts of Antoine and I was going to write the same comment. 
I thought there were good reasons for previous code. What has changed?



From storchaka at gmail.com  Fri Jun 14 15:59:52 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 14 Jun 2013 16:59:52 +0300
Subject: [Python-Dev] cpython: Move test_pep352 over to unittest.main()
In-Reply-To: <3bWkVM0wM9zP5m@mail.python.org>
References: <3bWkVM0wM9zP5m@mail.python.org>
Message-ID: <kpf7ki$evs$1@ger.gmane.org>

14.06.13 04:18, brett.cannon ???????(??):
> http://hg.python.org/cpython/rev/af27c661d4fb
> changeset:   84115:af27c661d4fb
> user:        Brett Cannon <brett at python.org>
> date:        Thu Jun 13 21:18:43 2013 -0400
> summary:
>    Move test_pep352 over to unittest.main()

You forgot about:

from test.support import run_unittest



From pynix.wang at gmail.com  Fri Jun 14 15:58:59 2013
From: pynix.wang at gmail.com (Pynix Wang)
Date: Fri, 14 Jun 2013 21:58:59 +0800
Subject: [Python-Dev] python symbolizition
Message-ID: <CAOSo4hPMXsMPbBUhgWg2kE=YehKTQ8Zg+eB6Nckv-ehVkPUWOA@mail.gmail.com>

1.lambda expression

c#
a.(x, y) => x == y
b.() => SomeMethod()

ruby:
 -> {|msg| puts msg}

python can use c# like and remove "lambda" keyword.

2.global variable

ruby
$glo_var

python can use $ or @ or another and remove "global".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130614/3d4a456f/attachment.html>

From markus at unterwaditzer.net  Fri Jun 14 16:50:53 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Fri, 14 Jun 2013 16:50:53 +0200
Subject: [Python-Dev] python symbolizition
In-Reply-To: <CAOSo4hPMXsMPbBUhgWg2kE=YehKTQ8Zg+eB6Nckv-ehVkPUWOA@mail.gmail.com>
References: <CAOSo4hPMXsMPbBUhgWg2kE=YehKTQ8Zg+eB6Nckv-ehVkPUWOA@mail.gmail.com>
Message-ID: <f05c70d5-c7b3-4e4a-886c-6c2debc92791@email.android.com>

But why?

-- Markus (from phone)

Pynix Wang <pynix.wang at gmail.com> wrote:
>1.lambda expression
>
>c#
>a.(x, y) => x == y
>b.() => SomeMethod()
>
>ruby:
> -> {|msg| puts msg}
>
>python can use c# like and remove "lambda" keyword.
>
>2.global variable
>
>ruby
>$glo_var
>
>python can use $ or @ or another and remove "global".
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net


From rdmurray at bitdance.com  Fri Jun 14 17:03:44 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 14 Jun 2013 11:03:44 -0400
Subject: [Python-Dev] python symbolizition
In-Reply-To: <f05c70d5-c7b3-4e4a-886c-6c2debc92791@email.android.com>
References: <CAOSo4hPMXsMPbBUhgWg2kE=YehKTQ8Zg+eB6Nckv-ehVkPUWOA@mail.gmail.com>
	<f05c70d5-c7b3-4e4a-886c-6c2debc92791@email.android.com>
Message-ID: <20130614150345.51EE2250BD7@webabinitio.net>

This discussion is better suited for python-ideas than it
is for python-dev.

On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer <markus at unterwaditzer.net> wrote:
> But why?
> 
> -- Markus (from phone)
> 
> Pynix Wang <pynix.wang at gmail.com> wrote:
> >1.lambda expression
> >
> >c#
> >a.(x, y) => x == y
> >b.() => SomeMethod()
> >
> >ruby:
> > -> {|msg| puts msg}
> >
> >python can use c# like and remove "lambda" keyword.
> >
> >2.global variable
> >
> >ruby
> >$glo_var
> >
> >python can use $ or @ or another and remove "global".
> >
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >Python-Dev mailing list
> >Python-Dev at python.org
> >http://mail.python.org/mailman/listinfo/python-dev
> >Unsubscribe:
> >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

From walter at livinglogic.de  Fri Jun 14 16:55:24 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Fri, 14 Jun 2013 16:55:24 +0200
Subject: [Python-Dev] eval and triple quoted strings
Message-ID: <51BB2EDC.80604@livinglogic.de>

Hello all!

This surprised me:

    >>> eval("'''\r\n'''")
    '\n'

Where did the \r go? ast.literal_eval() has the same problem:

    >>> ast.literal_eval("'''\r\n'''")
    '\n'

Is this a bug/worth fixing?

Servus,
    Walter

From guido at python.org  Fri Jun 14 17:36:48 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Jun 2013 08:36:48 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BB2EDC.80604@livinglogic.de>
References: <51BB2EDC.80604@livinglogic.de>
Message-ID: <CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>

Not a bug. The same is done for file input -- CRLF is changed to LF before
tokenizing.
On Jun 14, 2013 8:27 AM, "Walter D?rwald" <walter at livinglogic.de> wrote:

> Hello all!
>
> This surprised me:
>
>    >>> eval("'''\r\n'''")
>    '\n'
>
> Where did the \r go? ast.literal_eval() has the same problem:
>
>    >>> ast.literal_eval("'''\r\n'''")
>    '\n'
>
> Is this a bug/worth fixing?
>
> Servus,
>    Walter
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130614/e8a158fb/attachment.html>

From status at bugs.python.org  Fri Jun 14 18:07:28 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 14 Jun 2013 18:07:28 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130614160728.2A40956A39@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-06-07 - 2013-06-14)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4042 (+18)
  closed 25946 (+41)
  total  29988 (+59)

Open issues with patches: 1795 


Issues opened (41)
==================

#17947: Code, test, and doc review for PEP-0435 Enum
http://bugs.python.org/issue17947  reopened by ncoghlan

#18159: ConfigParser getters not available on SectionProxy
http://bugs.python.org/issue18159  opened by JBernardo

#18161: call fchdir if subprocess.Popen(cwd=integer|fileobject)
http://bugs.python.org/issue18161  opened by mmarkk

#18162: Add index attribute to IndexError
http://bugs.python.org/issue18162  opened by brett.cannon

#18163: Add a 'key' attribute to KeyError
http://bugs.python.org/issue18163  opened by brett.cannon

#18164: Embedding Python doc incorrectly refers to LINKFORSHARED
http://bugs.python.org/issue18164  opened by ned.deily

#18165: Add 'unexpected_type' to TypeError
http://bugs.python.org/issue18165  opened by brett.cannon

#18166: 'value' attribute for ValueError
http://bugs.python.org/issue18166  opened by brett.cannon

#18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n
http://bugs.python.org/issue18167  opened by catlee

#18168: plistlib output self-sorted dictionary
http://bugs.python.org/issue18168  opened by halfjuice

#18170: define built-in exceptions in Python code
http://bugs.python.org/issue18170  opened by brett.cannon

#18173: Add MixedTypeKey to reprlib
http://bugs.python.org/issue18173  opened by ncoghlan

#18174: Make regrtest with --huntrleaks check for fd leaks
http://bugs.python.org/issue18174  opened by sbt

#18176: Builtins documentation refers to old version of UCD.
http://bugs.python.org/issue18176  opened by belopolsky

#18177: Incorect quote marks in code section-headers in PDF version of
http://bugs.python.org/issue18177  opened by icedream91

#18178: Redefinition of malloc(3) family of functions at build time
http://bugs.python.org/issue18178  opened by cvs-src

#18179: SMTP.local_hostname is undocumented
http://bugs.python.org/issue18179  opened by jonash

#18181: super vs. someclass.__getattribute__
http://bugs.python.org/issue18181  opened by ronaldoussoren

#18182: xml.dom.createElement() does not take implicit namespaces into
http://bugs.python.org/issue18182  opened by Alexander.Tobias.Heinrich

#18184: Add range check  for %c in PyUnicode_FromFormat
http://bugs.python.org/issue18184  opened by serhiy.storchaka

#18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods
http://bugs.python.org/issue18185  opened by syeberman

#18188: ERROR: test_no_optimize_flag on Mageia Linux Cauldron x86-64 w
http://bugs.python.org/issue18188  opened by shlomif

#18189: IDLE Improvements: Unit test for Delegator.py
http://bugs.python.org/issue18189  opened by JayKrish

#18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly
http://bugs.python.org/issue18191  opened by jd

#18192: Move imp.get_magic() to importlib
http://bugs.python.org/issue18192  opened by brett.cannon

#18193: Move imp.reload() to importlib
http://bugs.python.org/issue18193  opened by brett.cannon

#18194: Move imp.source_from_cache/cache_from_source to importlib
http://bugs.python.org/issue18194  opened by brett.cannon

#18195: error when deep copying module is confusing
http://bugs.python.org/issue18195  opened by mrjbq7

#18197: insufficient error checking causes crash on windows
http://bugs.python.org/issue18197  opened by maxdeliso

#18199: Windows: support path longer than 260 bytes using "\\?\" prefi
http://bugs.python.org/issue18199  opened by Voo

#18201: distutils write into symlinks instead of replacing them
http://bugs.python.org/issue18201  opened by mgorny

#18202: Minor fixes for test_coding
http://bugs.python.org/issue18202  opened by serhiy.storchaka

#18203: Replace calls to malloc() with PyMem_Malloc()
http://bugs.python.org/issue18203  opened by haypo

#18206: license url in site.py should always use X.Y.Z form of version
http://bugs.python.org/issue18206  opened by py.user

#18207: OpenSSL may ignore seconds in notAfter
http://bugs.python.org/issue18207  opened by christian.heimes

#18209: Bytearray type not supported as a mutable object in the fcntl.
http://bugs.python.org/issue18209  opened by vxgmichel

#18211: -Werror=statement-after-declaration problem
http://bugs.python.org/issue18211  opened by ronaldoussoren

#18212: No way to check whether Future is finished?
http://bugs.python.org/issue18212  opened by cool-RR

#18213: py-bt errors on backtrace with PyRun_SimpleString and friends
http://bugs.python.org/issue18213  opened by pmuldoon

#18214: Stop purging modules which are garbage collected before shutdo
http://bugs.python.org/issue18214  opened by sbt

#18215: Script to test multiple versions of OpenSSL
http://bugs.python.org/issue18215  opened by christian.heimes



Most recent 15 issues with no replies (15)
==========================================

#18215: Script to test multiple versions of OpenSSL
http://bugs.python.org/issue18215

#18214: Stop purging modules which are garbage collected before shutdo
http://bugs.python.org/issue18214

#18212: No way to check whether Future is finished?
http://bugs.python.org/issue18212

#18211: -Werror=statement-after-declaration problem
http://bugs.python.org/issue18211

#18209: Bytearray type not supported as a mutable object in the fcntl.
http://bugs.python.org/issue18209

#18207: OpenSSL may ignore seconds in notAfter
http://bugs.python.org/issue18207

#18202: Minor fixes for test_coding
http://bugs.python.org/issue18202

#18189: IDLE Improvements: Unit test for Delegator.py
http://bugs.python.org/issue18189

#18184: Add range check  for %c in PyUnicode_FromFormat
http://bugs.python.org/issue18184

#18173: Add MixedTypeKey to reprlib
http://bugs.python.org/issue18173

#18166: 'value' attribute for ValueError
http://bugs.python.org/issue18166

#18165: Add 'unexpected_type' to TypeError
http://bugs.python.org/issue18165

#18161: call fchdir if subprocess.Popen(cwd=integer|fileobject)
http://bugs.python.org/issue18161

#18144: FD leak in urllib2
http://bugs.python.org/issue18144

#18137: format(float, str): integer overflow for huge precision
http://bugs.python.org/issue18137



Most recent 15 issues waiting for review (15)
=============================================

#18214: Stop purging modules which are garbage collected before shutdo
http://bugs.python.org/issue18214

#18202: Minor fixes for test_coding
http://bugs.python.org/issue18202

#18197: insufficient error checking causes crash on windows
http://bugs.python.org/issue18197

#18193: Move imp.reload() to importlib
http://bugs.python.org/issue18193

#18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly
http://bugs.python.org/issue18191

#18189: IDLE Improvements: Unit test for Delegator.py
http://bugs.python.org/issue18189

#18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods
http://bugs.python.org/issue18185

#18184: Add range check  for %c in PyUnicode_FromFormat
http://bugs.python.org/issue18184

#18181: super vs. someclass.__getattribute__
http://bugs.python.org/issue18181

#18179: SMTP.local_hostname is undocumented
http://bugs.python.org/issue18179

#18176: Builtins documentation refers to old version of UCD.
http://bugs.python.org/issue18176

#18174: Make regrtest with --huntrleaks check for fd leaks
http://bugs.python.org/issue18174

#18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n
http://bugs.python.org/issue18167

#18164: Embedding Python doc incorrectly refers to LINKFORSHARED
http://bugs.python.org/issue18164

#18155: csv.Sniffer.has_header doesn't escape characters used in regex
http://bugs.python.org/issue18155



Top 10 most discussed issues (10)
=================================

#17947: Code, test, and doc review for PEP-0435 Enum
http://bugs.python.org/issue17947  16 msgs

#18181: super vs. someclass.__getattribute__
http://bugs.python.org/issue18181  11 msgs

#3329: API for setting the memory allocator used by Python
http://bugs.python.org/issue3329  10 msgs

#17860: subprocess docs lack info how to use output result
http://bugs.python.org/issue17860   8 msgs

#18149: filecmp.cmp() incorrect results when previously compared file 
http://bugs.python.org/issue18149   7 msgs

#18163: Add a 'key' attribute to KeyError
http://bugs.python.org/issue18163   7 msgs

#10581: Review and document string format accepted in numeric data typ
http://bugs.python.org/issue10581   6 msgs

#14455: plistlib unable to read json and binary plist files
http://bugs.python.org/issue14455   6 msgs

#18188: ERROR: test_no_optimize_flag on Mageia Linux Cauldron x86-64 w
http://bugs.python.org/issue18188   6 msgs

#18203: Replace calls to malloc() with PyMem_Malloc()
http://bugs.python.org/issue18203   6 msgs



Issues closed (38)
==================

#5492: Error on leaving IDLE with quit() or exit() under Linux
http://bugs.python.org/issue5492  closed by roger.serwy

#6632: Include more chars in the decimal codec
http://bugs.python.org/issue6632  closed by belopolsky

#10382: Command line error marker misplaced on unicode entry
http://bugs.python.org/issue10382  closed by belopolsky

#14797: Deprecate imp.find_module()/load_module()
http://bugs.python.org/issue14797  closed by brett.cannon

#14813: Can't build under VS2008 anymore
http://bugs.python.org/issue14813  closed by christian.heimes

#15239: Abandoned Tools/unicode/mkstringprep.py
http://bugs.python.org/issue15239  closed by serhiy.storchaka

#15767: add ModuleNotFoundError
http://bugs.python.org/issue15767  closed by brett.cannon

#16102: uuid._netbios_getnode() is outdated
http://bugs.python.org/issue16102  closed by serhiy.storchaka

#17010: Windows launcher ignores active virtual environment
http://bugs.python.org/issue17010  closed by vinay.sajip

#17354: TypeError when running setup.py upload --show-response
http://bugs.python.org/issue17354  closed by berker.peksag

#17511: Idle find function closes after each find operation
http://bugs.python.org/issue17511  closed by roger.serwy

#17691: Fix test discovery for test_univnewlines.py
http://bugs.python.org/issue17691  closed by ezio.melotti

#17838: Can't assign a different value for sys.stdin in IDLE
http://bugs.python.org/issue17838  closed by roger.serwy

#17903: Python launcher for windows should search path for #!/usr/bin/
http://bugs.python.org/issue17903  closed by vinay.sajip

#18038: Unhelpful error message on invalid encoding specification
http://bugs.python.org/issue18038  closed by serhiy.storchaka

#18048: Merging test_pep263.py and test_coding.py
http://bugs.python.org/issue18048  closed by serhiy.storchaka

#18055: Stop using imp in IDLE
http://bugs.python.org/issue18055  closed by brett.cannon

#18126: Update links to NumPy resources in documentation
http://bugs.python.org/issue18126  closed by ezio.melotti

#18150: Duplicate test inside TestSingleDispatch
http://bugs.python.org/issue18150  closed by lukasz.langa

#18157: remove usage of imp.load_module() from pydoc
http://bugs.python.org/issue18157  closed by brett.cannon

#18160: Packaging more coherent Python
http://bugs.python.org/issue18160  closed by ezio.melotti

#18169: struct.pack() behaves strangely for 'L' on 64bit Linux
http://bugs.python.org/issue18169  closed by skrah

#18171: os.path.expanduser does not use the system encoding
http://bugs.python.org/issue18171  closed by r.david.murray

#18172: New easter egg: insecure string pickle
http://bugs.python.org/issue18172  closed by brian.curtin

#18175: os.listdir(fd) leaks fd on error
http://bugs.python.org/issue18175  closed by larry

#18180: Refleak in test_imp on Windows
http://bugs.python.org/issue18180  closed by sbt

#18183: Calling .lower() on certain unicode string raises SystemError
http://bugs.python.org/issue18183  closed by benjamin.peterson

#18186: 2.x subprocess contains set notation
http://bugs.python.org/issue18186  closed by berker.peksag

#18187: Fix broken link in Doc/library/venv.rst
http://bugs.python.org/issue18187  closed by ned.deily

#18190: RuntimeError raised with re.search + re.DOTALL on empty string
http://bugs.python.org/issue18190  closed by serhiy.storchaka

#18196: IDLE: forward apply patch for handling SystemExit
http://bugs.python.org/issue18196  closed by roger.serwy

#18198: unittest discover should provide a way to define discovery at 
http://bugs.python.org/issue18198  closed by michael.foord

#18200: Update stdlib to use ModuleNotFoundError
http://bugs.python.org/issue18200  closed by brett.cannon

#18204: distutils error showing upload error message
http://bugs.python.org/issue18204  closed by berker.peksag

#18205: PyOS_ReadlineFunctionPointer violates PyMem_Malloc() API: the 
http://bugs.python.org/issue18205  closed by haypo

#18208: Wrong bytecode generated for 'in' operation
http://bugs.python.org/issue18208  closed by amaury.forgeotdarc

#18210: Problem creating extension in python3.3 with mingw32
http://bugs.python.org/issue18210  closed by amaury.forgeotdarc

#18158: Delete test_importhooks
http://bugs.python.org/issue18158  closed by brett.cannon

From ijmorlan at uwaterloo.ca  Fri Jun 14 17:43:49 2013
From: ijmorlan at uwaterloo.ca (Isaac Morland)
Date: Fri, 14 Jun 2013 11:43:49 -0400 (EDT)
Subject: [Python-Dev] python symbolizition
In-Reply-To: <20130614150345.51EE2250BD7@webabinitio.net>
References: <CAOSo4hPMXsMPbBUhgWg2kE=YehKTQ8Zg+eB6Nckv-ehVkPUWOA@mail.gmail.com>
	<f05c70d5-c7b3-4e4a-886c-6c2debc92791@email.android.com>
	<20130614150345.51EE2250BD7@webabinitio.net>
Message-ID: <alpine.DEB.2.02.1306141143010.15328@ubuntu1204-102.cs.uwaterloo.ca>

On Fri, 14 Jun 2013, R. David Murray wrote:

> This discussion is better suited for python-ideas than it
> is for python-dev.

Better yet, perl-ideas.

> On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer <markus at unterwaditzer.net> wrote:
>> But why?
>>
>> -- Markus (from phone)
>>
>> Pynix Wang <pynix.wang at gmail.com> wrote:
>>> 1.lambda expression
>>>
>>> c#
>>> a.(x, y) => x == y
>>> b.() => SomeMethod()
>>>
>>> ruby:
>>> -> {|msg| puts msg}
>>>
>>> python can use c# like and remove "lambda" keyword.
>>>
>>> 2.global variable
>>>
>>> ruby
>>> $glo_var
>>>
>>> python can use $ or @ or another and remove "global".

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From solipsis at pitrou.net  Fri Jun 14 19:34:41 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 14 Jun 2013 19:34:41 +0200
Subject: [Python-Dev] PEP 442 accepted
In-Reply-To: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
References: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
Message-ID: <20130614193441.04b219c8@fsol>

On Wed, 5 Jun 2013 09:10:54 -0700
Benjamin Peterson <benjamin at python.org> wrote:
> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> condition that finalizers are only ever called once globally.

Ok, so there's an issue with that condition: it can't be upholded on
non-GC objects. Creating a non-GC object is quite obscure and rare,
though, since it requires basically a class with no __dict__ and an
empty __slots__:

class C:
    __slots__ = ()
    survivors = []

    def __del__(self):
        self.survivors.append(self)


In this case, a C instance's __del__ will be called every time
destruction is attempted, not only once. Is that a realistic problem?

Regards

Antoine.

From solipsis at pitrou.net  Fri Jun 14 20:09:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 14 Jun 2013 20:09:08 +0200
Subject: [Python-Dev] PEP 442 accepted
References: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
	<20130614193441.04b219c8@fsol>
Message-ID: <20130614200908.6f7fce29@fsol>

On Fri, 14 Jun 2013 19:34:41 +0200
Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 5 Jun 2013 09:10:54 -0700
> Benjamin Peterson <benjamin at python.org> wrote:
> > I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> > condition that finalizers are only ever called once globally.
> 
> Ok, so there's an issue with that condition: it can't be upholded on
> non-GC objects. Creating a non-GC object is quite obscure and rare,
> though, since it requires basically a class with no __dict__ and an
> empty __slots__: [...]

One more problem, albeit minor: there's a test_descr failure in some
part marked as "CPython implementation detail":

======================================================================
FAIL: test_subtype_resurrection (test.test_descr.ClassPropertiesAndMethods)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/cpython/finalize/Lib/test/test_descr.py", line 3749, in test_subtype_resurrection
    self.assertEqual(len(C.container), 1)
AssertionError: 0 != 1


However, the third-party test suites (Tornado, Tulip, SQLAlchemy,
Django, zope.interface) show no regressions.

Regards

Antoine.



From ron3200 at gmail.com  Fri Jun 14 20:11:11 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Fri, 14 Jun 2013 13:11:11 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
Message-ID: <51BB5CBF.5070308@gmail.com>



On 06/14/2013 10:36 AM, Guido van Rossum wrote:
> Not a bug. The same is done for file input -- CRLF is changed to LF before
> tokenizing.


Should this be the same?


python3 -c 'print(bytes("""\r\n""", "utf8"))'
b'\r\n'


 >>> eval('print(bytes("""\r\n""", "utf8"))')
b'\n'



Ron



> On Jun 14, 2013 8:27 AM, "Walter D?rwald" <walter at livinglogic.de
> <mailto:walter at livinglogic.de>> wrote:
>
>     Hello all!
>
>     This surprised me:
>
>         >>> eval("'''\r\n'''")
>         '\n'
>
>     Where did the \r go? ast.literal_eval() has the same problem:
>
>         >>> ast.literal_eval("'''\r\n'''")
>         '\n'
>
>     Is this a bug/worth fixing?
>
>     Servus,
>         Walter
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     http://mail.python.org/__mailman/listinfo/python-dev
>     <http://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     http://mail.python.org/__mailman/options/python-dev/__guido%40python.org <http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>


From ron3200 at gmail.com  Fri Jun 14 20:11:11 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Fri, 14 Jun 2013 13:11:11 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
Message-ID: <51BB5CBF.5070308@gmail.com>



On 06/14/2013 10:36 AM, Guido van Rossum wrote:
> Not a bug. The same is done for file input -- CRLF is changed to LF before
> tokenizing.


Should this be the same?


python3 -c 'print(bytes("""\r\n""", "utf8"))'
b'\r\n'


 >>> eval('print(bytes("""\r\n""", "utf8"))')
b'\n'



Ron



> On Jun 14, 2013 8:27 AM, "Walter D?rwald" <walter at livinglogic.de
> <mailto:walter at livinglogic.de>> wrote:
>
>     Hello all!
>
>     This surprised me:
>
>         >>> eval("'''\r\n'''")
>         '\n'
>
>     Where did the \r go? ast.literal_eval() has the same problem:
>
>         >>> ast.literal_eval("'''\r\n'''")
>         '\n'
>
>     Is this a bug/worth fixing?
>
>     Servus,
>         Walter
>     _________________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     http://mail.python.org/__mailman/listinfo/python-dev
>     <http://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     http://mail.python.org/__mailman/options/python-dev/__guido%40python.org <http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>

From maschu09 at gmail.com  Fri Jun 14 21:12:00 2013
From: maschu09 at gmail.com (Martin Schultz)
Date: Fri, 14 Jun 2013 21:12:00 +0200
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
Message-ID: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>

As much as I love python, the following drives me crazy, and I would wish
that some future version would come up with a more consistent approach for
this. And please don't reply with "Too bad if you don't know what type your
data are..." - if I want to implement some generic functionality, I want to
avoid constrains on the user where not absolutely necessary, and I believe
this approach is truely pythonic.

OK - here comes the problem set. These are in fact several related issues.
Clearly, a solution exists for each of them, but you will have to admit
that they are all very different in style and therefore unnecessarily
complicate matters. If you want to write code which works under many
circumstances, it will become quite convoluted.

1. Testing for None:

 From what I read elsewhere, the preferred solution is `if x is None` (or
`if not x is None` in the opposite case). This is fine and it works for
scalars, lists, sets, numpy ndarrays,...

2. Testing for empty lists or empty ndarrays:

 In principle, `len(x) == 0` will do the trick. **BUT** there are several
caveats here:
   - `len(scalar)` raises a TypeError, so you will have to use try and
except or find some other way of testing for a scalar value
   - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
a TypeError ("unsized object")
   - `len([[]])` returns a length of 1, which is somehow understandable,
but - I would argue - perhaps not what one might expect initially

 Alternatively, numpy arrays have a size attribute, and
`numpy.array([]).size`, `numpy.array(8.).size`, and
`numpy.array([8.]).size` all return what you would expect. And even
`numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
a numpy array, this might work. But have you ever tried to assign a list of
mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
will fail, even though the list inside is perfectly fine as a list.

3. Testing for scalar:

 Let's suppose we knew the number of non-empty elements, and this is 1.
Then there are occasions when you want to know if you have in fact `6` or
`[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is
also relevant for numpy arrays. For the latter, a combination of size and
ndim can help. For other objects, I would be tempted to use something like
`isiterable()`, however, this function doesn't exist, and there are
numerous discussions how one could or should find out if an object is
iterable - none of them truly intuitive. (and is it true that **every**
iterable object is a descendant of collections.Iterable?)

4. Finding the number of elements in an object:

 From the discussion above, it is already clear that `len(x)` is not very
robust for doing this. Just to mention another complication: `len("abcd")`
returns 4, even though this is only one string. Of course this is correct,
but it's a nuisance if you need to to find the number of elements of a list
of strings and if it can happen that you have a scalar string instead of a
1-element list. And, believe me, such situations do occur!

5. Forcing a scalar to become a 1-element list:

 Unfortunately, `list(77)` throws an error, because 77 is not iterable.
`numpy.array(77)` works, but - as we saw above - there will be no len
defined for it. Simply writing `[x]` is dangerous, because if x is a list
already, it will create `[[77]]`, which you generally don't want. Also,
`numpy.array([x])` would create a 2D array if x is already a 1D array or a
list. Often, it would be quite useful to know for sure that a function
result is provided as a list, regardless of how many elements it contains
(because then you can write `res[0]` without risking the danger to throw an
exception). Does anyone have a good suggestion for this one?

6. Detecting None values in a list:

 This is just for completeness. I have seen solutions using `all` which
solve this problem (see [question #1270920][1]). I haven't digged into them
extensively, but I fear that these will also suffer from the
above-mentioned issues if you don't know for sure if you are starting from
a list, a numpy array, or a scalar.

Enough complaining. Here comes my prayer to the python gods: **Please**

 - add a good `isiterable` function
 - add a `size` attribute to all objects (I wouldn't mind if this is None
in case you don't really know how to define the size of something, but it
would be good to have it, so that `anything.size` would never throw an error
 - add an `isscalar` function which would at least try to test if something
is a scalar (meaning a single entity). Note that this might give different
results compared to `isiterable`, because one would consider a scalar
string as a scalar even though it is iterable. And if `isscalar` would
throw exceptions in cases where it doesn't know what to do: fine - this can
be easily captured.
 - enable the `len()` function for scalar variables such as integers or
floats. I would tend to think that 1 is a natural answer to what the length
of a number is.



  [1]:
http://stackoverflow.com/questions/1270920/most-concise-way-to-check-whether-a-list-is-empty-or-contains-only-none
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130614/b56c6d72/attachment.html>

From brett at python.org  Fri Jun 14 22:03:41 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 14 Jun 2013 16:03:41 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
Message-ID: <CAP1=2W7BteURib-rZxDNxSNGmqDog3mm-ZtRROLxLZ-Fyhmd=w@mail.gmail.com>

On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz <maschu09 at gmail.com> wrote:

> As much as I love python, the following drives me crazy, and I would wish
> that some future version would come up with a more consistent approach for
> this. And please don't reply with "Too bad if you don't know what type your
> data are..." - if I want to implement some generic functionality, I want to
> avoid constrains on the user where not absolutely necessary, and I believe
> this approach is truely pythonic.
>
> OK - here comes the problem set. These are in fact several related issues.
> Clearly, a solution exists for each of them, but you will have to admit
> that they are all very different in style and therefore unnecessarily
> complicate matters. If you want to write code which works under many
> circumstances, it will become quite convoluted.
>
> 1. Testing for None:
>
>  From what I read elsewhere, the preferred solution is `if x is None` (or
> `if not x is None` in the opposite case). This is fine and it works for
> scalars, lists, sets, numpy ndarrays,...
>

Should actually be ``if x is not None``.

[SNIP]

>
>  - add a good `isiterable` function
>

Done: isinstance(x, collections.abc.Iterable)


>  - add a `size` attribute to all objects (I wouldn't mind if this is None
> in case you don't really know how to define the size of something, but it
> would be good to have it, so that `anything.size` would never throw an error
>

This is what len() is for. I don't know why numpy doesn't define the
__len__ method on their array types for that. We can't force every object
to have it as it doesn't make sense in all cases.


>  - add an `isscalar` function which would at least try to test if
> something is a scalar (meaning a single entity). Note that this might give
> different results compared to `isiterable`, because one would consider a
> scalar string as a scalar even though it is iterable. And if `isscalar`
> would throw exceptions in cases where it doesn't know what to do: fine -
> this can be easily captured.
>

The numbers module has a bunch of ABCs you can use to test for integrals,
etc. just as I suggested for iterables.


>   - enable the `len()` function for scalar variables such as integers or
> floats. I would tend to think that 1 is a natural answer to what the length
> of a number is.
>

The len() function is specifically for containers so it would not make
sense to ask for the length of something you can't put something into or
iterate over.

This is actually a perfect case of using the new single-dispatch generic
function work that has landed in Python 3.4 (
http://docs.python.org/3.4/library/functools.html#functools.singledispatch).
With that you could write your own custom_len() function that dispatches on
the type to return exactly what you are after. E.g.::

  @functools.singledispatch
  def custom_length(ob): return len(ob)

  @custom_length.register(int)
  _(ob): return 1

And on you go for all the types you want to special-case.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130614/5a2bbbef/attachment.html>

From robert.kern at gmail.com  Fri Jun 14 22:20:50 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 14 Jun 2013 21:20:50 +0100
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <CAP1=2W7BteURib-rZxDNxSNGmqDog3mm-ZtRROLxLZ-Fyhmd=w@mail.gmail.com>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<CAP1=2W7BteURib-rZxDNxSNGmqDog3mm-ZtRROLxLZ-Fyhmd=w@mail.gmail.com>
Message-ID: <kpftut$7sr$1@ger.gmane.org>

On 2013-06-14 21:03, Brett Cannon wrote:
>
> On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz <maschu09 at gmail.com
> <mailto:maschu09 at gmail.com>> wrote:

>       - add a `size` attribute to all objects (I wouldn't mind if this is None
>     in case you don't really know how to define the size of something, but it
>     would be good to have it, so that `anything.size` would never throw an error
>
> This is what len() is for. I don't know why numpy doesn't define the __len__
> method on their array types for that.

It does. It gives the size of the first axis, i.e. the one accessed by simple 
indexing with an integer: some_array[i]. The `size` attribute givens the total 
number of items in the possibly-multidimensional array. However, one of the 
other axes can be 0-length, so the array will have no elements but the length 
will be nonzero.

[~]
|4> np.empty([3,4,0])
array([], shape=(3, 4, 0), dtype=float64)

[~]
|5> np.empty([3,4,0])[1]
array([], shape=(4, 0), dtype=float64)

[~]
|6> len(np.empty([3,4,0]))
3

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From rdmurray at bitdance.com  Fri Jun 14 22:55:33 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 14 Jun 2013 16:55:33 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
	and list/ndarray? A prayer to the gods of Python
In-Reply-To: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
Message-ID: <20130614205534.A595A250BE9@webabinitio.net>

On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz <maschu09 at gmail.com> wrote:
> 2. Testing for empty lists or empty ndarrays:
> 
>  In principle, `len(x) == 0` will do the trick. **BUT** there are several
> caveats here:
>    - `len(scalar)` raises a TypeError, so you will have to use try and
> except or find some other way of testing for a scalar value
>    - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
> a TypeError ("unsized object")
>    - `len([[]])` returns a length of 1, which is somehow understandable,
> but - I would argue - perhaps not what one might expect initially
> 
>  Alternatively, numpy arrays have a size attribute, and
> `numpy.array([]).size`, `numpy.array(8.).size`, and
> `numpy.array([8.]).size` all return what you would expect. And even
> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
> a numpy array, this might work. But have you ever tried to assign a list of
> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
> will fail, even though the list inside is perfectly fine as a list.

In general you test whether nor not something is empty in Python by
testing its truth value.  Empty things are False.  Numpy seems to
follow this using size, from the limited examples you have given

   >>> bool(numpy.array([[]])
   False
   >>> bool(numpy.array([[1]])
   True

I have no idea what the definition of numpy.array.size is that it would
return 0 for [[]], so its return value obviously defies my intuition
as much as len([[]]) seems to have initially defied yours :)

> 3. Testing for scalar:
> 
>  Let's suppose we knew the number of non-empty elements, and this is 1.
> Then there are occasions when you want to know if you have in fact `6` or
> `[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is
> also relevant for numpy arrays. For the latter, a combination of size and
> ndim can help. For other objects, I would be tempted to use something like
> `isiterable()`, however, this function doesn't exist, and there are
> numerous discussions how one could or should find out if an object is
> iterable - none of them truly intuitive. (and is it true that **every**
> iterable object is a descendant of collections.Iterable?)

No, but...I'm not 100% sure about this as I tend to stay away from ABCs
myself, but my understanding is that collections.Iterable checks if an
object is iterable when you use it in an isinstance check.  There are
probably ways to fool it, but I think you could argue that any such data
types are broken.

> 4. Finding the number of elements in an object:
> 
>  From the discussion above, it is already clear that `len(x)` is not very
> robust for doing this. Just to mention another complication: `len("abcd")`
> returns 4, even though this is only one string. Of course this is correct,
> but it's a nuisance if you need to to find the number of elements of a list
> of strings and if it can happen that you have a scalar string instead of a
> 1-element list. And, believe me, such situations do occur!

len is robust when you consider that it only applies to sequences (see
below).  (I don't know what it means to "code a scaler as a numpy
array", but if it is still a scaler, it makes sense that it raises
a TypeError on len...it should.)

> 5. Forcing a scalar to become a 1-element list:
> 
>  Unfortunately, `list(77)` throws an error, because 77 is not iterable.
> `numpy.array(77)` works, but - as we saw above - there will be no len
> defined for it. Simply writing `[x]` is dangerous, because if x is a list
> already, it will create `[[77]]`, which you generally don't want. Also,
> `numpy.array([x])` would create a 2D array if x is already a 1D array or a
> list. Often, it would be quite useful to know for sure that a function
> result is provided as a list, regardless of how many elements it contains
> (because then you can write `res[0]` without risking the danger to throw an
> exception). Does anyone have a good suggestion for this one?

Well, no.  If the list is empty res[0] will throw an error.  You need
to know both that it is indexable (note: not iterable...an object can be
iterable without being indexable) and that it is non-empty.  Well behaved
objects should I think pass an isinstance check against collections.Sequence.
(I can't think of a good way to check for indexability without the abc.)

> Enough complaining. Here comes my prayer to the python gods: **Please**
> 
>  - add a good `isiterable` function

That would be spelled isinstance(x, collections.Iterable), it seems.

>  - add a `size` attribute to all objects (I wouldn't mind if this is None
> in case you don't really know how to define the size of something, but it
> would be good to have it, so that `anything.size` would never throw an error

Why?  What is the definition of 'size' that makes it useful outside
of numpy?

>  - add an `isscalar` function which would at least try to test if something
> is a scalar (meaning a single entity). Note that this might give different
> results compared to `isiterable`, because one would consider a scalar
> string as a scalar even though it is iterable. And if `isscalar` would
> throw exceptions in cases where it doesn't know what to do: fine - this can
> be easily captured.

This I sort of agree with.  I've often enough wanted to know if something
is a non-string iterable.  But you'd have to decide if bytes/bytearray
is a sequence of integers or a scaler...

>  - enable the `len()` function for scalar variables such as integers or
> floats. I would tend to think that 1 is a natural answer to what the length
> of a number is.

That would screw up the ABC type hierarchy...the existence of len
indicates that an iterable is indexable.

--David

From pje at telecommunity.com  Fri Jun 14 23:03:14 2013
From: pje at telecommunity.com (PJ Eby)
Date: Fri, 14 Jun 2013 17:03:14 -0400
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BB5CBF.5070308@gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
Message-ID: <CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>

On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam <ron3200 at gmail.com> wrote:
>
>
> On 06/14/2013 10:36 AM, Guido van Rossum wrote:
>>
>> Not a bug. The same is done for file input -- CRLF is changed to LF before
>> tokenizing.
>
>
>
> Should this be the same?
>
>
> python3 -c 'print(bytes("""\r\n""", "utf8"))'
> b'\r\n'
>
>
>>>> eval('print(bytes("""\r\n""", "utf8"))')
> b'\n'

No, but:

eval(r'print(bytes("""\r\n""", "utf8"))')

should be.  (And is.)

What I believe you and Walter are missing is that the \r\n in the eval
strings are converted early if you don't make the enclosing string
raw.  So what you're eval-ing is not what you think you are eval-ing,
hence the confusion.

From robert.kern at gmail.com  Sat Jun 15 00:31:47 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 14 Jun 2013 23:31:47 +0100
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <20130614205534.A595A250BE9@webabinitio.net>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
Message-ID: <kpg5kd$6cu$1@ger.gmane.org>

On 2013-06-14 21:55, R. David Murray wrote:
> On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz <maschu09 at gmail.com> wrote:
>> 2. Testing for empty lists or empty ndarrays:
>>
>>   In principle, `len(x) == 0` will do the trick. **BUT** there are several
>> caveats here:
>>     - `len(scalar)` raises a TypeError, so you will have to use try and
>> except or find some other way of testing for a scalar value
>>     - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
>> a TypeError ("unsized object")
>>     - `len([[]])` returns a length of 1, which is somehow understandable,
>> but - I would argue - perhaps not what one might expect initially
>>
>>   Alternatively, numpy arrays have a size attribute, and
>> `numpy.array([]).size`, `numpy.array(8.).size`, and
>> `numpy.array([8.]).size` all return what you would expect. And even
>> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
>> a numpy array, this might work. But have you ever tried to assign a list of
>> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
>> will fail, even though the list inside is perfectly fine as a list.
>
> In general you test whether nor not something is empty in Python by
> testing its truth value.  Empty things are False.  Numpy seems to
> follow this using size, from the limited examples you have given
>
>     >>> bool(numpy.array([[]])
>     False
>     >>> bool(numpy.array([[1]])
>     True

numpy does not do so. Empty arrays are extremely rare and testing for them rarer 
(rarer still is testing for emptiness not knowing if it is an array or some 
other sequence). What people usually want from bool(some_array) is either 
some_array.all() or some_array.any(). In the face of this ambiguity, numpy 
refuses the temptation to guess and raises an exception explaining matters.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robert.kern at gmail.com  Sat Jun 15 00:53:53 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 14 Jun 2013 23:53:53 +0100
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <kpg5kd$6cu$1@ger.gmane.org>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpg5kd$6cu$1@ger.gmane.org>
Message-ID: <kpg6tr$ipo$1@ger.gmane.org>

On 2013-06-14 23:31, Robert Kern wrote:
> On 2013-06-14 21:55, R. David Murray wrote:
>> On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz <maschu09 at gmail.com> wrote:
>>> 2. Testing for empty lists or empty ndarrays:
>>>
>>>   In principle, `len(x) == 0` will do the trick. **BUT** there are several
>>> caveats here:
>>>     - `len(scalar)` raises a TypeError, so you will have to use try and
>>> except or find some other way of testing for a scalar value
>>>     - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises
>>> a TypeError ("unsized object")
>>>     - `len([[]])` returns a length of 1, which is somehow understandable,
>>> but - I would argue - perhaps not what one might expect initially
>>>
>>>   Alternatively, numpy arrays have a size attribute, and
>>> `numpy.array([]).size`, `numpy.array(8.).size`, and
>>> `numpy.array([8.]).size` all return what you would expect. And even
>>> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to
>>> a numpy array, this might work. But have you ever tried to assign a list of
>>> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])`
>>> will fail, even though the list inside is perfectly fine as a list.
>>
>> In general you test whether nor not something is empty in Python by
>> testing its truth value.  Empty things are False.  Numpy seems to
>> follow this using size, from the limited examples you have given
>>
>>     >>> bool(numpy.array([[]])
>>     False
>>     >>> bool(numpy.array([[1]])
>>     True
>
> numpy does not do so. Empty arrays are extremely rare and testing for them rarer
> (rarer still is testing for emptiness not knowing if it is an array or some
> other sequence). What people usually want from bool(some_array) is either
> some_array.all() or some_array.any(). In the face of this ambiguity, numpy
> refuses the temptation to guess and raises an exception explaining matters.

Actually, that's a bit of a lie. In the empty case and the one-element case, we 
do return a bool, False for empty and bool(element) for whatever that one 
element is. Anything else raises the exception since we don't know whether it is 
all() or any() that was desired.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From brett at python.org  Sat Jun 15 00:57:01 2013
From: brett at python.org (Brett Cannon)
Date: Fri, 14 Jun 2013 18:57:01 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Closes issue 17947.
 Adds PEP-0435 (Enum, IntEnum) to the stdlib.
In-Reply-To: <3bWtm22z9dz7Ljj@mail.python.org>
References: <3bWtm22z9dz7Ljj@mail.python.org>
Message-ID: <CAP1=2W7EtxgQDrmjoKjY8kM649xQ8NYrBSfZFOaFaXSkcCK4Qw@mail.gmail.com>

Ethan, did you forget to run ``hg add`` before committing? If not then why
the heck did we argue over enums for so long if this was all it took to
make everyone happy? =)


On Fri, Jun 14, 2013 at 3:31 AM, ethan.furman <python-checkins at python.org>wrote:

> http://hg.python.org/cpython/rev/fae92309c3be
> changeset:   84117:fae92309c3be
> parent:      84115:af27c661d4fb
> user:        Ethan Furman <ethan at stoneleaf.us>
> date:        Fri Jun 14 00:30:27 2013 -0700
> summary:
>   Closes issue 17947.  Adds PEP-0435 (Enum, IntEnum) to the stdlib.
>
> files:
>   Doc/library/datatypes.rst |  1 +
>   1 files changed, 1 insertions(+), 0 deletions(-)
>
>
> diff --git a/Doc/library/datatypes.rst b/Doc/library/datatypes.rst
> --- a/Doc/library/datatypes.rst
> +++ b/Doc/library/datatypes.rst
> @@ -30,3 +30,4 @@
>     copy.rst
>     pprint.rst
>     reprlib.rst
> +   enum.rst
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130614/f5ad9abd/attachment.html>

From victor.stinner at gmail.com  Sat Jun 15 01:03:42 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 15 Jun 2013 01:03:42 +0200
Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held
 in Python 3.4
In-Reply-To: <EFE3877620384242A686D52278B7CCD35394B5BE@RKV-IT-EXCH104.ccp.ad.local>
References: <CAMpsgwbcnR7=JgfYTcxVZJrm_qZednDr-YnRO7FoWL4jJ8645w@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35394B5BE@RKV-IT-EXCH104.ccp.ad.local>
Message-ID: <CAMpsgwaf=rg=j+NpEdNwwnkpMKa8+rFEkDZrVvihiRSgJ4Jr+w@mail.gmail.com>

I commited the new API (little bit different than my last patch on issue #3329):
http://hg.python.org/cpython/rev/6661a8154eb3

The documentation will be available in a few minutes at:
http://docs.python.org/3/c-api/memory.html

2013/6/14 Kristj?n Valur J?nsson <kristjan at ccpgames.com>:
>> Removing the GIL restriction would help to replace direct calls to
>> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application
>> would be able to replace memory allocators, and these allocators would be
>> used "everywhere".
>> => see http://bugs.python.org/issue18203
>
> To keep this interesting, I have a somewhat different opinion to Victor :)
>   have put comments in the original defect, but would like to repeat them here.
> IMHO, keeping the GIL restriction on PyMem_MALLOC is useful.
> 1) It allows it to be replaced with PyObject_MALLOC(). Or anything else.  In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details.  Requiring it to be GIL free severely limits what it can do.  For example, it would be forbidden to delegate to PyObject_MALLOC when debugging.

For my own pytracemalloc tool, holding the GIL while calling
PyMem_Malloc() is required to be able to retrieve the Python filename
and line number of the caller. So you convinced me :-) I am also
worried by the backward compatibility, even if I expect that only a
very few developers replaced Python memory allocators. A custom memory
allocator may not be thread-safe, so the GIL can also be convinient.

I added new functions in the "final" API: PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree(). These functions are just wrapper
for malloc(), realloc() and free(). The GIL does not need to be hold.
No process is done before/after at all. Behaviour of
PyMem_RawMalloc(0) is platform depend for example. "size >
PY_SSIZE_T_MAX" check is not done neither, but it may be interesting
to add this check for security reasons (it is already in place for
PyMem_Malloc and PyObject_Malloc).

Using these new functions instead of malloc/realloc/free is
interesting because the internal functions can be replaced with
PyMem_SetRawAllocators() and many checks are added in debug mode (ex:
check for buffer under- and overflow).

PyObject_Malloc() was not documented, so I did not document
PyObject_SetAllocators().

In the final API, I added a new PyMemAllocators structure to simplify
the API. I also made _PyObject_SetArenaAllocators() private because I
don't like its API (it is not homogenous with PyMem_SetAllocators) and
it is concerned by less use cases. I prefer to wait a little before
making this API public.

I didn't use "#ifndef Py_LIMITED_API", so all new functions are part
of the stable API. Is it correct?

Victor

From solipsis at pitrou.net  Sat Jun 15 03:04:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 03:04:31 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
References: <3bXH1R0dqGz7LkT@mail.python.org>
Message-ID: <20130615030431.679bfc1b@fsol>

On Sat, 15 Jun 2013 00:44:11 +0200 (CEST)
victor.stinner <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/6661a8154eb3
> changeset:   84127:6661a8154eb3
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Sat Jun 15 00:37:46 2013 +0200
> summary:
>   Issue #3329: Add new APIs to customize memory allocators
> 
> * Add a new PyMemAllocators structure
> * New functions:
> 
>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>     allocator functions
>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>   - PyMem_SetupDebugHooks()
>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()

My two cents, but I would prefer if this whole changeset was reverted.
I think it adds too much complexity in the memory allocation APIs,
for a pretty specialized benefit. IMHO, we should be able to get by with
less allocation APIs (why the new _Raw APIs) and less hook-setting
functions.

Regards

Antoine.



From victor.stinner at gmail.com  Sat Jun 15 03:54:50 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 15 Jun 2013 03:54:50 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130615030431.679bfc1b@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
Message-ID: <CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>

2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
>> http://hg.python.org/cpython/rev/6661a8154eb3
>> ...
>>   Issue #3329: Add new APIs to customize memory allocators
>>
>> * Add a new PyMemAllocators structure
>> * New functions:
>>
>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>>     allocator functions
>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>   - PyMem_SetupDebugHooks()
>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>
> My two cents, but I would prefer if this whole changeset was reverted.
> I think it adds too much complexity in the memory allocation APIs,
> for a pretty specialized benefit. IMHO, we should be able to get by with
> less allocation APIs (why the new _Raw APIs) and less hook-setting
> functions.

Ok, I reverted my commit.

I posted my initial patch 3 months ago on the bug tracker. I got some
reviews and discussed with Kristj?n Valur J?nsson who heavily modified
Python for his game at CCP. I started two threads on python-dev this
week (ok, only two days ago). I thaugh that the last known issues were
fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
GIL-free allocator).

I will work on a PEP to explain all these new functions and their use cases.

**

The addition of PyMem_RawMalloc() is motivated by the issue #18203
(Replace calls to malloc() with PyMem_Malloc()). The goal is to be
able to setup a custom allocator for *all* allocation made by Python,
so malloc() should not be called directly. PyMem_RawMalloc() is
required in places where the GIL is not held (ex: in os.getcwd() on
Windows).

PyMem_Malloc() is misused (called without the GIL held) in different
places. Examples: the readline modules and functions called at Python
startup, including main(). Replacing PyMem_Malloc() with malloc()
would not allow to use the custom allocator everywhere, so
PyMem_RawMalloc() is also required here.

The last point is an extension of the issue #18203: some external
libraries like zlib or OpenSSL are also calling malloc() directly. But
Python can configure these libraries to use a custom memory allocator.
I plan to configure external libraries to use PyMem_GetRawAllocators()
if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not
simply malloc) and if setting a custom allocator only affect a
function and not the whole library.

Victor

From greg.ewing at canterbury.ac.nz  Sat Jun 15 06:08:21 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 15 Jun 2013 16:08:21 +1200
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
Message-ID: <51BBE8B5.1080706@canterbury.ac.nz>

Guido van Rossum wrote:
> Not a bug. The same is done for file input -- CRLF is changed to LF 
> before tokenizing.

I'm not convinced it's reasonable behaviour to re-scan the
string as though it's being read from a file. It's a Python
string, so it's already been through whatever line-ending
transformation is appropriate to get it into memory.

-- 
Greg

From ncoghlan at gmail.com  Sat Jun 15 06:57:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jun 2013 14:57:49 +1000
Subject: [Python-Dev] PEP 442 accepted
In-Reply-To: <20130614193441.04b219c8@fsol>
References: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
	<20130614193441.04b219c8@fsol>
Message-ID: <CADiSq7e9xdVBNV5mHh0HEHOcV8iD=EAbR3AVMa4LmCR+nfyBwg@mail.gmail.com>

On 15 June 2013 03:34, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 5 Jun 2013 09:10:54 -0700
> Benjamin Peterson <benjamin at python.org> wrote:
>> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
>> condition that finalizers are only ever called once globally.
>
> Ok, so there's an issue with that condition: it can't be upholded on
> non-GC objects. Creating a non-GC object is quite obscure and rare,
> though, since it requires basically a class with no __dict__ and an
> empty __slots__:
>
> class C:
>     __slots__ = ()
>     survivors = []
>
>     def __del__(self):
>         self.survivors.append(self)
>
>
> In this case, a C instance's __del__ will be called every time
> destruction is attempted, not only once. Is that a realistic problem?

So, to trigger that __del__() method a second time, such an object
would have to be:

1. Defined in the first place (the use cases for stateless objects
with destructors seem rare...)
2. Hanging off a reference cycle
3. Which then gets resurrected

If it isn't easy to fix by changing the way such classes are
constructed by typeobject.c (i.e. force them to be GC-aware when they
define a __del__ method, since they may still be hanging off the edge
of a reference cycle, even if they can't form one themselves), I'd say
chalk it up as a CPython wart and not worry about it for now.

As far as the test_subtype_resurrection failure goes, I think the
associated comment says it all: "(On non-CPython implementations,
however, __del__ is typically not called again.)"

The test should just be rewritten to check that __del__ is only called
once in the face of resurrection.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Jun 15 07:18:01 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jun 2013 15:18:01 +1000
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BBE8B5.1080706@canterbury.ac.nz>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BBE8B5.1080706@canterbury.ac.nz>
Message-ID: <CADiSq7cEqzxJT9w=1AA5SM5YWW_zf37rCjTWZ8daOmGvFpwZZA@mail.gmail.com>

On 15 June 2013 14:08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>>
>> Not a bug. The same is done for file input -- CRLF is changed to LF before
>> tokenizing.
>
>
> I'm not convinced it's reasonable behaviour to re-scan the
> string as though it's being read from a file. It's a Python
> string, so it's already been through whatever line-ending
> transformation is appropriate to get it into memory.

No, that's not the way the Python compiler works. The transformation
Guido is talking about is the way the tokenizer identifiers "NEWLINE"
tokens:

>>> list(tokenize.tokenize((l for l in (b"""'\r\n'""", b"")).__next__))[2]
TokenInfo(type=4 (NEWLINE), string='\r\n', start=(1, 1), end=(1, 3),
line="'\r\n'")

This long predates universal newlines mode - it's part of the
compilation process, not part of the IO system. The compiler then sees
the NEWLINE token in the tokenizer output, and inserts a "\n" into the
triple-quoted string.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Jun 15 07:50:58 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jun 2013 15:50:58 +1000
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
Message-ID: <CADiSq7cy=kkEuoN6E2=nJh080BY6cCP3p5GS_jiQRyEsjYwrzA@mail.gmail.com>

On 15 June 2013 11:54, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
>>> http://hg.python.org/cpython/rev/6661a8154eb3
>>> ...
>>>   Issue #3329: Add new APIs to customize memory allocators
>>>
>>> * Add a new PyMemAllocators structure
>>> * New functions:
>>>
>>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
>>>     allocator functions
>>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
>>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
>>>   - PyMem_SetupDebugHooks()
>>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
>>
>> My two cents, but I would prefer if this whole changeset was reverted.
>> I think it adds too much complexity in the memory allocation APIs,
>> for a pretty specialized benefit. IMHO, we should be able to get by with
>> less allocation APIs (why the new _Raw APIs) and less hook-setting
>> functions.
>
> Ok, I reverted my commit.
>
> I posted my initial patch 3 months ago on the bug tracker. I got some
> reviews and discussed with Kristj?n Valur J?nsson who heavily modified
> Python for his game at CCP. I started two threads on python-dev this
> week (ok, only two days ago). I thaugh that the last known issues were
> fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> GIL-free allocator).
>
> I will work on a PEP to explain all these new functions and their use cases.

I think the new APIs are mostly valid and well-justified, but agree a
PEP is a good idea.

Yes, it's a complex solution, but it's solving a complex problem that
arises when embedding CPython inside executables that need to run on
non-traditional platforms where the simple C defined
malloc/realloc/free trio is inadequate.

This is a complementary effort to PEP 432 - that aims to simplify
embedding CPython in general, while Victor's efforts here specifically
focus on situations where it is necessary to better map CPython to an
underlying memory model that differs from the traditional C one. While
the "single heap" model of memory enshrined in the C standard is
certainly the most common model, it's far from being the only one, and
these days CPython also gets used in those other environments.

About the only simplification I can see is that PyMem_RawMalloc(),
PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
preprocessor macros instead of permitting runtime reconfiguration.
Allowing the memory allocations for the CPython runtime to be handled
separately from those for arbitrary C libraries loaded into the
process seems reasonable, though.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From greg at krypto.org  Sat Jun 15 09:54:10 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 15 Jun 2013 00:54:10 -0700
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CADiSq7cy=kkEuoN6E2=nJh080BY6cCP3p5GS_jiQRyEsjYwrzA@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<CADiSq7cy=kkEuoN6E2=nJh080BY6cCP3p5GS_jiQRyEsjYwrzA@mail.gmail.com>
Message-ID: <CAGE7PNJKNkX=+b6hsyTpj6zX6=Sdnj1W2tcscGP7Bq9QNOqtrg@mail.gmail.com>

fwiw i'm also supportive of adding these apis.  Lets PEP away to iron out
any details or document disagreements but overall I'd also like to see
something a lot like these go in.

-gps


On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 June 2013 11:54, Victor Stinner <victor.stinner at gmail.com> wrote:
> > 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> >>> http://hg.python.org/cpython/rev/6661a8154eb3
> >>> ...
> >>>   Issue #3329: Add new APIs to customize memory allocators
> >>>
> >>> * Add a new PyMemAllocators structure
> >>> * New functions:
> >>>
> >>>   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free
> memory
> >>>     allocator functions
> >>>   - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
> >>>   - PyMem_GetAllocators(), PyMem_SetAllocators()
> >>>   - PyMem_SetupDebugHooks()
> >>>   - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
> >>
> >> My two cents, but I would prefer if this whole changeset was reverted.
> >> I think it adds too much complexity in the memory allocation APIs,
> >> for a pretty specialized benefit. IMHO, we should be able to get by with
> >> less allocation APIs (why the new _Raw APIs) and less hook-setting
> >> functions.
> >
> > Ok, I reverted my commit.
> >
> > I posted my initial patch 3 months ago on the bug tracker. I got some
> > reviews and discussed with Kristj?n Valur J?nsson who heavily modified
> > Python for his game at CCP. I started two threads on python-dev this
> > week (ok, only two days ago). I thaugh that the last known issues were
> > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an
> > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a
> > GIL-free allocator).
> >
> > I will work on a PEP to explain all these new functions and their use
> cases.
>
> I think the new APIs are mostly valid and well-justified, but agree a
> PEP is a good idea.
>
> Yes, it's a complex solution, but it's solving a complex problem that
> arises when embedding CPython inside executables that need to run on
> non-traditional platforms where the simple C defined
> malloc/realloc/free trio is inadequate.
>
> This is a complementary effort to PEP 432 - that aims to simplify
> embedding CPython in general, while Victor's efforts here specifically
> focus on situations where it is necessary to better map CPython to an
> underlying memory model that differs from the traditional C one. While
> the "single heap" model of memory enshrined in the C standard is
> certainly the most common model, it's far from being the only one, and
> these days CPython also gets used in those other environments.
>
> About the only simplification I can see is that PyMem_RawMalloc(),
> PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with
> preprocessor macros instead of permitting runtime reconfiguration.
> Allowing the memory allocations for the CPython runtime to be handled
> separately from those for arbitrary C libraries loaded into the
> process seems reasonable, though.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/23063972/attachment-0001.html>

From solipsis at pitrou.net  Sat Jun 15 12:55:57 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 12:55:57 +0200
Subject: [Python-Dev] PEP 442 accepted
In-Reply-To: <CADiSq7e9xdVBNV5mHh0HEHOcV8iD=EAbR3AVMa4LmCR+nfyBwg@mail.gmail.com>
References: <CAPZV6o8=pBT2PEYSzVjjKZx0MxwSeVbErrD9uOWz9aj_yyGJqQ@mail.gmail.com>
	<20130614193441.04b219c8@fsol>
	<CADiSq7e9xdVBNV5mHh0HEHOcV8iD=EAbR3AVMa4LmCR+nfyBwg@mail.gmail.com>
Message-ID: <20130615125557.6455f6a5@fsol>

On Sat, 15 Jun 2013 14:57:49 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 June 2013 03:34, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Wed, 5 Jun 2013 09:10:54 -0700
> > Benjamin Peterson <benjamin at python.org> wrote:
> >> I (and Guido) are accepting PEP 442 (Safe object finalization) on the
> >> condition that finalizers are only ever called once globally.
> >
> > Ok, so there's an issue with that condition: it can't be upholded on
> > non-GC objects. Creating a non-GC object is quite obscure and rare,
> > though, since it requires basically a class with no __dict__ and an
> > empty __slots__:
> >
> > class C:
> >     __slots__ = ()
> >     survivors = []
> >
> >     def __del__(self):
> >         self.survivors.append(self)
> >
> >
> > In this case, a C instance's __del__ will be called every time
> > destruction is attempted, not only once. Is that a realistic problem?
> 
> So, to trigger that __del__() method a second time, such an object
> would have to be:
> 
> 1. Defined in the first place (the use cases for stateless objects
> with destructors seem rare...)
> 2. Hanging off a reference cycle
> 3. Which then gets resurrected

They don't need to hang off a reference cycle. You can resurrect a
non-cyclic object from __del__ too.

But, yeah, stateless objects must be pretty rare, since by definition
they cannot represent anything (except perhaps "nothing").

> i.e. force them to be GC-aware when they
> define a __del__ method, since they may still be hanging off the edge
> of a reference cycle, even if they can't form one themselves

Objects which hang off the edge of a reference cycle don't need to be
GC-aware, since traversing them isn't needed to be break the cycle.
I think "chalking it up as a CPython wart" is a reasonable solution, I
just wanted to ask.

Regards

Antoine.

From solipsis at pitrou.net  Sat Jun 15 13:01:25 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 13:01:25 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
Message-ID: <20130615130125.1d4ef1f7@fsol>

On Sat, 15 Jun 2013 03:54:50 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> The addition of PyMem_RawMalloc() is motivated by the issue #18203
> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
> able to setup a custom allocator for *all* allocation made by Python,
> so malloc() should not be called directly. PyMem_RawMalloc() is
> required in places where the GIL is not held (ex: in os.getcwd() on
> Windows).

We already had this discussion on IRC and this argument isn't very
convincing to me. If os.getcwd() doesn't hold the GIL while allocating
memory, then you should fix it to hold the GIL while allocating memory.

I don't like the idea of adding of third layer of allocation APIs. The
dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
gratuitous (i.e. not motivated by any actual real-world concern, as
far as I can tell).

As for the debug functions you added: PyMem_GetRawAllocators(),
PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
_PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
you try to make that 2 or 3?

Regards

Antoine.

From ncoghlan at gmail.com  Sat Jun 15 14:22:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jun 2013 22:22:33 +1000
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130615130125.1d4ef1f7@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
Message-ID: <CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>

On 15 June 2013 21:01, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 15 Jun 2013 03:54:50 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>> The addition of PyMem_RawMalloc() is motivated by the issue #18203
>> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
>> able to setup a custom allocator for *all* allocation made by Python,
>> so malloc() should not be called directly. PyMem_RawMalloc() is
>> required in places where the GIL is not held (ex: in os.getcwd() on
>> Windows).
>
> We already had this discussion on IRC and this argument isn't very
> convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> memory, then you should fix it to hold the GIL while allocating memory.
>
> I don't like the idea of adding of third layer of allocation APIs. The
> dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> gratuitous (i.e. not motivated by any actual real-world concern, as
> far as I can tell).

The only reason for the small object allocator to exist is because
operating system allocators generally aren't optimised for frequent
allocation and deallocation of small objects. You can gain a *lot* of
speed from handling those inside the application. As the allocations
grow in size, though, the application level allocator just becomes
useless overhead, so it's better to delegate those operations directly
to the OS.

However, it's still desirable to be able to monitor those direct
allocations in debug mode, thus it makes sense to have a GIL protected
direct allocation API as well. You could try to hide the existence of
the latter behaviour and treat it as a private API, but why? For
custom allocators, it's useful to be able to *ensure* you can bypass
CPython's small object allocator, rather than having to rely on it
being bypassed for allocations above a certain size.

> As for the debug functions you added: PyMem_GetRawAllocators(),
> PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> you try to make that 2 or 3?

Faux simplicity that is achieved only by failing to model a complex
problem domain correctly is a bad idea (if we were satisfied with
that, we could stick with the status quo).

The only question mark in my mind is over the GIL-free raw allocation
APIs. I think it makes sense to at least conditionally define those as
macros so an embedding application can redirect *just* the allocations
made by the CPython runtime (rather than having to redefine
malloc/realloc/free when building Python), but I don't believe the
case has been adequately made for making the raw APIs configurable at
runtime. Dropping that aspect would at least eliminate the
PyMem_(Get|Set)RawAllocators() APIs.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat Jun 15 14:41:15 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 14:41:15 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
Message-ID: <20130615144115.0a7e3176@fsol>

On Sat, 15 Jun 2013 22:22:33 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 June 2013 21:01, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 15 Jun 2013 03:54:50 +0200
> > Victor Stinner <victor.stinner at gmail.com> wrote:
> >> The addition of PyMem_RawMalloc() is motivated by the issue #18203
> >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
> >> able to setup a custom allocator for *all* allocation made by Python,
> >> so malloc() should not be called directly. PyMem_RawMalloc() is
> >> required in places where the GIL is not held (ex: in os.getcwd() on
> >> Windows).
> >
> > We already had this discussion on IRC and this argument isn't very
> > convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> > memory, then you should fix it to hold the GIL while allocating memory.
> >
> > I don't like the idea of adding of third layer of allocation APIs. The
> > dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> > gratuitous (i.e. not motivated by any actual real-world concern, as
> > far as I can tell).
> 
> The only reason for the small object allocator to exist is because
> operating system allocators generally aren't optimised for frequent
> allocation and deallocation of small objects. You can gain a *lot* of
> speed from handling those inside the application. As the allocations
> grow in size, though, the application level allocator just becomes
> useless overhead, so it's better to delegate those operations directly
> to the OS.

The small object allocator *already* delegates those operations
directly to the OS. You don't need a separate API to do it by hand.

> For
> custom allocators, it's useful to be able to *ensure* you can bypass
> CPython's small object allocator, rather than having to rely on it
> being bypassed for allocations above a certain size.

Which custom allocators?

> > As for the debug functions you added: PyMem_GetRawAllocators(),
> > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> > you try to make that 2 or 3?
> 
> Faux simplicity that is achieved only by failing to model a complex
> problem domain correctly is a bad idea (if we were satisfied with
> that, we could stick with the status quo).

Actually, I'm sure almost everyone *is* satisfied with the status quo
here (witness the total absence of bug reports on the matter). Victor's
patch addresses a rare concern compared to the common use cases of
CPython.

And I'm not even sure what "faux simplicity" you are talking about.
What is the supposed complexity that this API is supposed to address?
Why do we need two different pairs of hook-setting functions, rather
than letting each function set / get all hooks at once? And why the
private API functions for setting arena allocators?

Memory allocation APIs are a fundamental part of the C API that many
extension writers have to understand and deal with. I'm opposed to
gratuitous complication when the use cases are not compelling.

Regards

Antoine.

From tseaver at palladion.com  Sat Jun 15 14:53:07 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 15 Jun 2013 08:53:07 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <20130614205534.A595A250BE9@webabinitio.net>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
Message-ID: <kpho3d$ao1$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/14/2013 04:55 PM, R. David Murray wrote:
> This I sort of agree with.  I've often enough wanted to know if
> something is a non-string iterable.  But you'd have to decide if
> bytes/bytearray is a sequence of integers or a scaler...

In fifteen years of Python programming, I have literally *never* wanted
to iterate over 'str' (or now 'bytes').  I've always considered the fact
that Python made them iterable by default (rather than e.g. defining a
method / property to get to an iterable "view" on the underlying string)
a wart and a serious bug magnet.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H
R4cAn2+ijAkebHuF92txeddBq99L/zqn
=bLkO
-----END PGP SIGNATURE-----


From victor.stinner at gmail.com  Sat Jun 15 14:57:12 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 15 Jun 2013 14:57:12 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
Message-ID: <CAMpsgwaTem9OqwVDatix9ttQCbHMKpbwwHEzoWO4bCpN-uxSmQ@mail.gmail.com>

Le 15 juin 2013 03:54, "Victor Stinner"
<victor.stinner at gmail.com<javascript:_e({}, 'cvml',
'victor.stinner at gmail.com');>>
a ?crit :

> Ok, I reverted my commit.
>
> I will work on a PEP to explain all these new functions and their use
cases.

I created the PEP 445 to reserve the number. It is ready for a review but
already contains some explanation of the new API.

http://www.python.org/dev/peps/pep-0445/

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/1ad89789/attachment.html>

From donald at stufft.io  Sat Jun 15 15:15:12 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 15 Jun 2013 09:15:12 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
	and list/ndarray? A prayer to the gods of Python
In-Reply-To: <kpho3d$ao1$1@ger.gmane.org>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpho3d$ao1$1@ger.gmane.org>
Message-ID: <9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io>

I never want to iterate, but I love slice syntax and indexing. Don't think you can have that w/o being able to loop over it can you? Maybe I'm just thinking slow since I just woke up. 

On Jun 15, 2013, at 8:53 AM, Tres Seaver <tseaver at palladion.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 06/14/2013 04:55 PM, R. David Murray wrote:
>> This I sort of agree with.  I've often enough wanted to know if
>> something is a non-string iterable.  But you'd have to decide if
>> bytes/bytearray is a sequence of integers or a scaler...
> 
> In fifteen years of Python programming, I have literally *never* wanted
> to iterate over 'str' (or now 'bytes').  I've always considered the fact
> that Python made them iterable by default (rather than e.g. defining a
> method / property to get to an iterable "view" on the underlying string)
> a wart and a serious bug magnet.
> 
> 
> Tres.
> - -- 
> ===================================================================
> Tres Seaver          +1 540-429-0999          tseaver at palladion.com
> Palladion Software   "Excellence by Design"    http://palladion.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H
> R4cAn2+ijAkebHuF92txeddBq99L/zqn
> =bLkO
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

From tseaver at palladion.com  Sat Jun 15 15:47:06 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 15 Jun 2013 09:47:06 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpho3d$ao1$1@ger.gmane.org>
	<9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io>
Message-ID: <kphr8k$7lg$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/15/2013 09:15 AM, Donald Stufft wrote:
> I never want to iterate, but I love slice syntax and indexing. Don't 
> think you can have that w/o being able to loop over it can you? Maybe 
> I'm just thinking slow since I just woke up.

You could if '__iter__' raised an error ('NotIterable', maybe) which
defeated the '__getitem__'/'__len__'-based fallback.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG8cFoACgkQ+gerLs4ltQ7KcgCfdFCEkp2gBeuUgn/KooY7F5HX
Jm8An2bwq8QoplwJ1MqIbS76m+xdl/Mk
=6hA9
-----END PGP SIGNATURE-----


From ncoghlan at gmail.com  Sat Jun 15 16:12:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Jun 2013 00:12:02 +1000
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130615144115.0a7e3176@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<20130615144115.0a7e3176@fsol>
Message-ID: <CADiSq7f5b3z-fjYUkAp2e06oYd8hBpoJKF5uG91mXZ-ido8CnQ@mail.gmail.com>

On 15 June 2013 22:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 15 Jun 2013 22:22:33 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> For
>> custom allocators, it's useful to be able to *ensure* you can bypass
>> CPython's small object allocator, rather than having to rely on it
>> being bypassed for allocations above a certain size.
>
> Which custom allocators?

Those used by companies like Dropbox to speed up frequent allocations
(look up their PyCon 2011 keynote). If we don't provide suitable APIs
that we can still hook in debug mode, they'll bypass our
infrastructure completely and we'll miss significant memory accesses.

>> > As for the debug functions you added: PyMem_GetRawAllocators(),
>> > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
>> > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
>> > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
>> > you try to make that 2 or 3?
>>
>> Faux simplicity that is achieved only by failing to model a complex
>> problem domain correctly is a bad idea (if we were satisfied with
>> that, we could stick with the status quo).
>
> Actually, I'm sure almost everyone *is* satisfied with the status quo
> here (witness the total absence of bug reports on the matter). Victor's
> patch addresses a rare concern compared to the common use cases of
> CPython.

Indeed, but they're use cases I care about, Victor cares about,
Kristjan cares about, Greg cares about. It's OK that you don't care
about them, just as 99% of the Python programmers on the planet won't
care about PEP 432 or the various arcane metaclass changes we've made
over the years.

issue 3329 (the one where Victor implemented this) was actually filed
by the folks working on the Symbian port. The last comment on that
issue before Victor restarted was from you, in reply to someone asking
if we had implemented it yet.

> And I'm not even sure what "faux simplicity" you are talking about.
> What is the supposed complexity that this API is supposed to address?

The fact that there is more to the world than x86/x86_64 and the very
simplistic C memory model. Python is growing more popular in
non-traditional execution environments, and we finally have someone
(Victor) interested in doing the work to support them properly. That
should be celebrated, not blocked because it isn't meaningful for the
more common systems where the C memory model is fine.

> Why do we need two different pairs of hook-setting functions, rather
> than letting each function set / get all hooks at once?

I've already said I don't think the raw allocators should be
configurable at runtime. The other is because it's likely people will
only want to replace the lower level allocators and leave the small
object allocator alone. However, they should be able to completely
replace the small object allocator if they want. Making the more
common case more complicated to avoid adding a more appropriate two
level API is the kind of thing I mean by "faux simplicity" - it's
almost certainly going to be harder to use in practice, so trading
multiple functions for fewer functions each taking more parameters
isn't actually a win.

> And why the
> private API functions for setting arena allocators?

Because they're in a different compilation unit...

> Memory allocation APIs are a fundamental part of the C API that many
> extension writers have to understand and deal with. I'm opposed to
> gratuitous complication when the use cases are not compelling.

That's a documentation problem. C extension authors shouldn't be
touching these, and most people embedding CPython shouldn't be
touching them either. They're the C level equivalent of metaclasses:
if you're not sure you need them, you don't need them.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat Jun 15 16:38:39 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 16:38:39 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CADiSq7f5b3z-fjYUkAp2e06oYd8hBpoJKF5uG91mXZ-ido8CnQ@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<20130615144115.0a7e3176@fsol>
	<CADiSq7f5b3z-fjYUkAp2e06oYd8hBpoJKF5uG91mXZ-ido8CnQ@mail.gmail.com>
Message-ID: <20130615163839.61a2d669@fsol>

On Sun, 16 Jun 2013 00:12:02 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 15 June 2013 22:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 15 Jun 2013 22:22:33 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> For
> >> custom allocators, it's useful to be able to *ensure* you can bypass
> >> CPython's small object allocator, rather than having to rely on it
> >> being bypassed for allocations above a certain size.
> >
> > Which custom allocators?
> 
> Those used by companies like Dropbox to speed up frequent allocations
> (look up their PyCon 2011 keynote). If we don't provide suitable APIs
> that we can still hook in debug mode, they'll bypass our
> infrastructure completely and we'll miss significant memory accesses.

I don't understand the concern. People can ignore the Python
allocators, and then use their own debugging infrastructure. This is
what happens everytime you want to use your own infrastructure instead
of a 3rd party-provided one.

> > And I'm not even sure what "faux simplicity" you are talking about.
> > What is the supposed complexity that this API is supposed to address?
> 
> The fact that there is more to the world than x86/x86_64 and the very
> simplistic C memory model.

Then I think it needs a PEP to properly address it and explain it to
everyone.

Moreover, I think you are conflating two issues: the ability to add
memory allocation hooks (for tracing/debugging purposes), and the
adaptation to "non-traditional" memory models (whatever that means).
Those concerns don't necessarily come together.

Regards

Antoine.

From oscar.j.benjamin at gmail.com  Sat Jun 15 17:12:38 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sat, 15 Jun 2013 16:12:38 +0100
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
Message-ID: <CAHVvXxSCAp8SihBrB0rPR3936q55LC2OwVYDvvjKz=hBKnbWbg@mail.gmail.com>

Your questions/suggestions are off-topic for this list. This belongs
on python-ideas.

On 14 June 2013 20:12, Martin Schultz <maschu09 at gmail.com> wrote:
>
> 2. Testing for empty lists or empty ndarrays:
> 4. Finding the number of elements in an object:
> 6. Detecting None values in a list:

Each of the problems above relates to wanting to use generic Python
containers interchangeably with numpy arrays. The solution is simply
not to do this; there are too many semantic inconsistencies for this
interchangeability to be safe. Here are just a few of them:

>>> import numpy as np
>>> [1, 2, 3] + [4, 5, 6]
[1, 2, 3, 4, 5, 6]
>>> np.array([1, 2, 3]) + np.array([4, 5, 6])
array([5, 7, 9])
>>> [1, 2, 3] * 3
[1, 2, 3, 1, 2, 3, 1, 2, 3]
>>> np.array([1, 2, 3]) * 3
array([3, 6, 9])
>>> [[1, 2, 3], [4, 5, 6]] + [7, 8, 9]
[[1, 2, 3], [4, 5, 6], 7, 8, 9]
>>> np.array([[1, 2, 3], [4, 5, 6]]) + np.array([7, 8, 9])
array([[ 8, 10, 12],
       [11, 13, 15]])
>>> bool([1, 2, 3])
True
>>> bool(np.array([1, 2, 3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

Numpy provides a very convenient function that will convert objects of
all the mentioned types to ndarray objects: the numpy.array function.
If you have a function and want to accept lists (of lists ...) and
ndarrays interchangeably, simply convert your input with numpy.array
at the start of the function i.e.:

def square(numbers):
    numbers = numpy.array(numbers)
    print(numbers.size, numbers.shape)  # etc.
    return numbers ** 2  # Would be an error for a list of lists

> 3. Testing for scalar:
> 5. Forcing a scalar to become a 1-element list:

numbers = np.array(possibly_scalar_input)
if not numbers.shape:
    print('Numbers was scalar: coercing...')
    numbers = np.array([numbers])

Usually it is bad design to have an API that interchangeably accepts
scalars and sequences. There are some situations where it does make
sense such as numpy's ufuncs. These are scalar mathematical functions
that can operate distributively on lists of lists, ndarrays etc. and
numpy provides a convenient way for you to define these yourself:

>>> def square(x):
...     return x ** 2
...
>>> square = np.frompyfunc(square, 1, 1)
>>> square(2)
4
>>> square([2])
array([4], dtype=object)
>>> square([2, 3, 4])
array([4, 9, 16], dtype=object)
>>> square([[2, 3, 4]])
array([[4, 9, 16]], dtype=object)
>>> square([[]])
array([], shape=(1, 0), dtype=object)

The frompyfunc function unfortunately doesn't quite work like a
decorator but you can easily wrap it do so:

def ufunc(nin, nout):
    def wrapper_factory(func):
        return np.frompyfunc(func, nin, nout)
    return wrapper_factory

@ufunc(1, 1)
def square(x):
    return x ** 2


Oscar

From christian at python.org  Sat Jun 15 17:28:25 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 15 Jun 2013 17:28:25 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
	memory allocators
In-Reply-To: <CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
Message-ID: <51BC8819.5060106@python.org>

Am 15.06.2013 14:22, schrieb Nick Coghlan:
> However, it's still desirable to be able to monitor those direct
> allocations in debug mode, thus it makes sense to have a GIL protected
> direct allocation API as well. You could try to hide the existence of
> the latter behaviour and treat it as a private API, but why? For
> custom allocators, it's useful to be able to *ensure* you can bypass
> CPython's small object allocator, rather than having to rely on it
> being bypassed for allocations above a certain size.


There is even more to it. We like to keep track of memory allocations in
libraries that are wrapped by Python's extension modules, e.g. expat,
openssl etc. Almost every library has a hook to set a custom memory
allocator, either globally (CRYPTO_set_mem_functions) or for each object
(XML_ParserCreate_MM's XML_Memory_Handling_Suite).

Python releases the GIL around IO or CPU critical sections of the
library. But these sections may call the memory management functions. If
these memory functions use the GIL, then some speed ups of releasing the
GIL in the first place are lost. It might even be possible to walk into
dead lock situations.

For that reason it makes sense to have a set of low level memory
management functions, that don't rely on the GIL for locking. These
functions can still impose their own locking if they need to modify a
global state (e.g. allocation statistics). In normal release mode and on
most platforms the raw memory allocators should be thin wrappers around
malloc(), realloc() and free() -- or perhaps just macros.

Eventually I would like to ban direct usage of malloc() from Python's
core and patch all memory management through our API.

Christian

From christian at python.org  Sat Jun 15 17:49:41 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 15 Jun 2013 17:49:41 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
	memory allocators
In-Reply-To: <CAMpsgwaTem9OqwVDatix9ttQCbHMKpbwwHEzoWO4bCpN-uxSmQ@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<CAMpsgwaTem9OqwVDatix9ttQCbHMKpbwwHEzoWO4bCpN-uxSmQ@mail.gmail.com>
Message-ID: <51BC8D15.1050707@python.org>

Am 15.06.2013 14:57, schrieb Victor Stinner:
> Le 15 juin 2013 03:54, "Victor Stinner" <victor.stinner at gmail.com
> <javascript:_e({}, 'cvml', 'victor.stinner at gmail.com');>> a ?crit :
> 
>> Ok, I reverted my commit.
>>
>> I will work on a PEP to explain all these new functions and their use
> cases.
> 
> I created the PEP 445 to reserve the number. It is ready for a review
> but already containssome explanation of the new API.

+1

How about you compare your approach with some other libraries, too?
Other projects have dealt with similar issues in the past, too. libxml2
has an extensive API for memory management and debugging.

I have assembled a list of APIs for you:

OpenSSL has CRYPTO_set_mem_functions() to set memory management
functions globally
http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124

expat has a per-instance memory handler:
http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l717

libtiff has three global hooks _TIFFmalloc(), _TIFFrealloc() and
_TIFFfree() that are used instead of malloc() in its core.
http://trac.imagemagick.org/browser/tiff/trunk/libtiff/tif_unix.c#L258

libxml2 has http://xmlsoft.org/html/libxml-xmlmemory.html

Christian

From ron3200 at gmail.com  Sat Jun 15 21:58:33 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 15 Jun 2013 14:58:33 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
Message-ID: <51BCC769.8070904@gmail.com>



On 06/14/2013 04:03 PM, PJ Eby wrote:
>> >Should this be the same?
>> >
>> >
>> >python3 -c 'print(bytes("""\r\n""", "utf8"))'
>> >b'\r\n'
>> >
>> >
>>>>> >>>>eval('print(bytes("""\r\n""", "utf8"))')
>> >b'\n'
> No, but:
>
> eval(r'print(bytes("""\r\n""", "utf8"))')
>
> should be.  (And is.)
>
> What I believe you and Walter are missing is that the \r\n in the eval
> strings are converted early if you don't make the enclosing string
> raw.  So what you're eval-ing is not what you think you are eval-ing,
> hence the confusion.

Yes thanks, seems like an easy mistake to make.

To be clear...

The string to eval is parsed when the eval line is tokenized in the scope 
containing the eval() function.  The eval function then parses the 
resulting string object it receives as it's input.

There is no mention of using raw strings in the docs on evel and exec.  I 
think there should be, because the intention (in most cases) is for eval to 
parse the string, and not for it to be parsed or changed before it's 
evaluated by eval or exec.

An example using a string with escape characters might make it clearer.

Cheers,
    Ron

From ron3200 at gmail.com  Sat Jun 15 21:58:33 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 15 Jun 2013 14:58:33 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
Message-ID: <51BCC769.8070904@gmail.com>



On 06/14/2013 04:03 PM, PJ Eby wrote:
>> >Should this be the same?
>> >
>> >
>> >python3 -c 'print(bytes("""\r\n""", "utf8"))'
>> >b'\r\n'
>> >
>> >
>>>>> >>>>eval('print(bytes("""\r\n""", "utf8"))')
>> >b'\n'
> No, but:
>
> eval(r'print(bytes("""\r\n""", "utf8"))')
>
> should be.  (And is.)
>
> What I believe you and Walter are missing is that the \r\n in the eval
> strings are converted early if you don't make the enclosing string
> raw.  So what you're eval-ing is not what you think you are eval-ing,
> hence the confusion.

Yes thanks, seems like an easy mistake to make.

To be clear...

The string to eval is parsed when the eval line is tokenized in the scope 
containing the eval() function.  The eval function then parses the 
resulting string object it receives as it's input.

There is no mention of using raw strings in the docs on evel and exec.  I 
think there should be, because the intention (in most cases) is for eval to 
parse the string, and not for it to be parsed or changed before it's 
evaluated by eval or exec.

An example using a string with escape characters might make it clearer.

Cheers,
    Ron


From tjreedy at udel.edu  Sat Jun 15 22:11:59 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 15 Jun 2013 16:11:59 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <kpho3d$ao1$1@ger.gmane.org>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpho3d$ao1$1@ger.gmane.org>
Message-ID: <kpihqb$l1m$1@ger.gmane.org>

On 6/15/2013 8:53 AM, Tres Seaver wrote:

> In fifteen years of Python programming, I have literally *never* wanted
> to iterate over 'str' (or now 'bytes').

If so, it is because you have always been able to use pre-written 
methods and functions that internally do the iteration for you.

 > I've always considered the fact
> that Python made them iterable by default (rather than e.g. defining a
> method / property to get to an iterable "view" on the underlying string)

.__iter__ is that method.

But this is off-topic for pydev.

-- 
Terry Jan Reedy


From guido at python.org  Sat Jun 15 22:23:46 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 15 Jun 2013 13:23:46 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BCC769.8070904@gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BCC769.8070904@gmail.com>
Message-ID: <CAP7+vJ+yPAW9ZRv2OdF0507tC9LCURvvN4eTVVGTcBiHueS5uA@mail.gmail.com>

The semantics of raw strings are clear. I don't see that they should be
called out especially in any context. (Except for regexps.) Usually exec()
is not used with a literal anyway (what would be the point).

--Guido van Rossum (sent from Android phone)
On Jun 15, 2013 1:03 PM, "Ron Adam" <ron3200 at gmail.com> wrote:

>
>
> On 06/14/2013 04:03 PM, PJ Eby wrote:
>
>> >Should this be the same?
>>> >
>>> >
>>> >python3 -c 'print(bytes("""\r\n""", "utf8"))'
>>> >b'\r\n'
>>> >
>>> >
>>>
>>>> >>>>eval('print(bytes("""\r\n"**"", "utf8"))')
>>>>>>
>>>>> >b'\n'
>>>
>> No, but:
>>
>> eval(r'print(bytes("""\r\n""", "utf8"))')
>>
>> should be.  (And is.)
>>
>> What I believe you and Walter are missing is that the \r\n in the eval
>> strings are converted early if you don't make the enclosing string
>> raw.  So what you're eval-ing is not what you think you are eval-ing,
>> hence the confusion.
>>
>
> Yes thanks, seems like an easy mistake to make.
>
> To be clear...
>
> The string to eval is parsed when the eval line is tokenized in the scope
> containing the eval() function.  The eval function then parses the
> resulting string object it receives as it's input.
>
> There is no mention of using raw strings in the docs on evel and exec.  I
> think there should be, because the intention (in most cases) is for eval to
> parse the string, and not for it to be parsed or changed before it's
> evaluated by eval or exec.
>
> An example using a string with escape characters might make it clearer.
>
> Cheers,
>    Ron
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> guido%40python.org<http://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/cd5895e2/attachment.html>

From ethan at stoneleaf.us  Sat Jun 15 21:46:32 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 15 Jun 2013 12:46:32 -0700
Subject: [Python-Dev] backported Enum
Message-ID: <51BCC498.1030100@stoneleaf.us>

So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series.

I would like to put this on PyPI, but the `enum` name is already taken.

Would it be inappropriate to call it `stdlib.enum`?

--
~Ethan~

From solipsis at pitrou.net  Sat Jun 15 22:53:54 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 Jun 2013 22:53:54 +0200
Subject: [Python-Dev] backported Enum
References: <51BCC498.1030100@stoneleaf.us>
Message-ID: <20130615225354.33ab8e09@fsol>

On Sat, 15 Jun 2013 12:46:32 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series.
> 
> I would like to put this on PyPI, but the `enum` name is already taken.
> 
> Would it be inappropriate to call it `stdlib.enum`?

`backport.enum`?



From barry at python.org  Sat Jun 15 23:14:07 2013
From: barry at python.org (Barry Warsaw)
Date: Sat, 15 Jun 2013 17:14:07 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <51BCC498.1030100@stoneleaf.us>
References: <51BCC498.1030100@stoneleaf.us>
Message-ID: <20130615171407.0e2c827d@anarchist>

On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:

>So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4
>in the 2.x series.
>
>I would like to put this on PyPI, but the `enum` name is already taken.
>
>Would it be inappropriate to call it `stdlib.enum`?

The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it and
wouldn't mind giving up the enum PyPI name?

-Barry

From tseaver at palladion.com  Sat Jun 15 23:45:33 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 15 Jun 2013 17:45:33 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <kpihqb$l1m$1@ger.gmane.org>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpho3d$ao1$1@ger.gmane.org> <kpihqb$l1m$1@ger.gmane.org>
Message-ID: <kpin9n$80b$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/15/2013 04:11 PM, Terry Reedy wrote:

> On 6/15/2013 8:53 AM, Tres Seaver wrote:
> 
>> In fifteen years of Python programming, I have literally *never*
>> wanted to iterate over 'str' (or now 'bytes').
> 
> If so, it is because you have always been able to use pre-written 
> methods and functions that internally do the iteration for you.

Given that strings are implemented in C, there is no real "iteration"
happing (in the Python sense) in their methods.  What stdlib code can you
point to that does iterate over them in Python?  I.e.:

  for c in stringval:
      ...

Even if there were, aren't you proving my point?  The fact that strings
are implicitly iterable injects all kinds of fur into methods which take
either a single value or an iterable as an argument, e.g.:

  def api_function(value_or_values):
      if isinstance(value_or_values, STRING_TYPES):
         value_or_values = [value_or_values]
      elif isinstance(value_or_values, collections.abc.Iterable):
         value_or_values = list(value_or_values)
      else:
         value_or_values = [value_or_values]

The bugs that leaving the first test out yields pop up all over the place.


>> I've always considered the fact
>>> that Python made them iterable by default (rather than e.g.
>>> defining a method / property to get to an iterable "view" on the
>>> underlying string)
> 
> .__iter__ is that method.

But it is *on be default*:  I was describing something which has to be
requested in ways that don't get triggered by syntax, and doesn't make
strings look "non-scalar" for APIs like the one above.			


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG84H0ACgkQ+gerLs4ltQ56RQCgks8R52f41RwJ+v9oteOBC3qY
kIIAoIHmg+zcmJpV3v/gAhkKJfbNKufj
=ZeRB
-----END PGP SIGNATURE-----


From tseaver at palladion.com  Sat Jun 15 23:47:25 2013
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 15 Jun 2013 17:47:25 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <20130615171407.0e2c827d@anarchist>
References: <51BCC498.1030100@stoneleaf.us> <20130615171407.0e2c827d@anarchist>
Message-ID: <kpind7$80b$2@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/15/2013 05:14 PM, Barry Warsaw wrote:
> On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:
> 
>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and
>> back to 2.4 in the 2.x series.
>> 
>> I would like to put this on PyPI, but the `enum` name is already
>> taken.
>> 
>> Would it be inappropriate to call it `stdlib.enum`?
> 
> The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it
> and wouldn't mind giving up the enum PyPI name?

That would screw anybody already using the existing distributions pretty
badly.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG84O0ACgkQ+gerLs4ltQ71qwCgo4uubYXVw/qvARESfzPLzFYZ
Fb8AnjB5ZcwupMoQ6SP6r7Xl26Hg3wpQ
=u3L7
-----END PGP SIGNATURE-----


From matthew.lefavor at nasa.gov  Sat Jun 15 23:58:24 2013
From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC])
Date: Sat, 15 Jun 2013 16:58:24 -0500
Subject: [Python-Dev] backported Enum
In-Reply-To: <20130615171407.0e2c827d@anarchist>
References: <51BCC498.1030100@stoneleaf.us> <20130615171407.0e2c827d@anarchist>
Message-ID: <8897D17F-8C15-49D8-A477-E4C1693891B9@nasa.gov>

What if there's some obscure PyPi module that requires that `enum` package, or some other project (not hosted on PyPI) that requires Ben Finney's original `enum` package? Is there anyway to get usage data to make sure nobody's been using it recently?

ML

On Jun 15, 2013, at 5:14 PM, Barry Warsaw wrote:

On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote:

So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4
in the 2.x series.

I would like to put this on PyPI, but the `enum` name is already taken.

Would it be inappropriate to call it `stdlib.enum`?

The last upload was on 2009-08-25.  Maybe Ben Finney's abandoned it and
wouldn't mind giving up the enum PyPI name?

-Barry
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/matthew.lefavor%40nasa.gov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/8e6aebf3/attachment.html>

From ethan at stoneleaf.us  Sat Jun 15 23:43:36 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 15 Jun 2013 14:43:36 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <20130615225354.33ab8e09@fsol>
References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol>
Message-ID: <51BCE008.5070308@stoneleaf.us>

On 06/15/2013 01:53 PM, Antoine Pitrou wrote:
> On Sat, 15 Jun 2013 12:46:32 -0700
> Ethan Furman <ethan at stoneleaf.us> wrote:
>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series.
>>
>> I would like to put this on PyPI, but the `enum` name is already taken.
>>
>> Would it be inappropriate to call it `stdlib.enum`?
>
> `backport.enum`?

Some other well-meaning soul has already claimed that name, but that one only covers 2.6 and 2.7.

--
~Ethan~

From greg at krypto.org  Sun Jun 16 00:43:10 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 15 Jun 2013 15:43:10 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <51BCC498.1030100@stoneleaf.us>
References: <51BCC498.1030100@stoneleaf.us>
Message-ID: <CAGE7PNJPMMvWAV2GT9GTQ_KdS-WkozjBhMufaCwW5Z==vgXxiw@mail.gmail.com>

I tend to just pick a name and stick with it.  subprocess32 is subprocess
backported from 3.2 (with the 3.3 timeout feature also in it).  unittest2
is unittest from 2.7.

It tends to work.  and it also emphasizes that i'm unlikely to backport
future non-bugfix updates beyond the release mentioned and merely focus on
keeping it stable and available for use on older pythons.

A "backport" namespace is a neat idea but unless someone's going to create
a system for backports to register which versions they are backports from
and automagically have sub-module imports from backport pick the backported
code or the standard library version when the Python VM is recent enough to
not need a backport I wouldn't bother claiming that name.  (and even that
feels like overengineering)

-gps



On Sat, Jun 15, 2013 at 12:46 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to
> 2.4 in the 2.x series.
>
> I would like to put this on PyPI, but the `enum` name is already taken.
>
> Would it be inappropriate to call it `stdlib.enum`?
>
> --
> ~Ethan~
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> greg%40krypto.org<http://mail.python.org/mailman/options/python-dev/greg%40krypto.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/805c09a8/attachment.html>

From victor.stinner at gmail.com  Sun Jun 16 01:48:06 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 16 Jun 2013 01:48:06 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <51BC8819.5060106@python.org>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<51BC8819.5060106@python.org>
Message-ID: <CAMpsgwaVjG7tgM2i=Nt+d495Jcn7rU8COx51C-jeVn-=hyG39w@mail.gmail.com>

2013/6/15 Christian Heimes <christian at python.org>:
> Am 15.06.2013 14:22, schrieb Nick Coghlan:
>> However, it's still desirable to be able to monitor those direct
>> allocations in debug mode, thus it makes sense to have a GIL protected
>> direct allocation API as well. You could try to hide the existence of
>> the latter behaviour and treat it as a private API, but why? For
>> custom allocators, it's useful to be able to *ensure* you can bypass
>> CPython's small object allocator, rather than having to rely on it
>> being bypassed for allocations above a certain size.
>
> There is even more to it. We like to keep track of memory allocations in
> libraries that are wrapped by Python's extension modules, e.g. expat,
> openssl etc. Almost every library has a hook to set a custom memory
> allocator, either globally (CRYPTO_set_mem_functions) or for each object
> (XML_ParserCreate_MM's XML_Memory_Handling_Suite).

I just create the issue http://bugs.python.org/issue18227: "Use Python
memory allocators in external libraries like zlib or OpenSSL".

Is it possible to detect if Python is used as a standalone application
(the classic "python" program) or if Python is embedded? If it is
possible, we can modify the "global" memory allocators of a library.
Otherwise, it is more tricky. Should Python sets its "own" memory
allocators? Maybe only if PyMem_SetRawAllocators() was called?

> Eventually I would like to ban direct usage of malloc() from Python's
> core and patch all memory management through our API.

I already create issue http://bugs.python.org/issue18203 for this part.

Victor

From victor.stinner at gmail.com  Sun Jun 16 02:18:32 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 16 Jun 2013 02:18:32 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130615130125.1d4ef1f7@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
Message-ID: <CAMpsgwY=NbiEowtaM+b8QopTU7t--g=vAV8kH6O-VD9RyfGBdA@mail.gmail.com>

2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> On Sat, 15 Jun 2013 03:54:50 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>> The addition of PyMem_RawMalloc() is motivated by the issue #18203
>> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
>> able to setup a custom allocator for *all* allocation made by Python,
>> so malloc() should not be called directly. PyMem_RawMalloc() is
>> required in places where the GIL is not held (ex: in os.getcwd() on
>> Windows).
>
> We already had this discussion on IRC and this argument isn't very
> convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> memory, then you should fix it to hold the GIL while allocating memory.

The GIL is released for best performances, holding the GIL would have
an impact on performances.

PyMem_RawMalloc() is needed when PyMem_Malloc() cannot be used because
the GIL was released. For example, for the issue #18227 (reuse the
custom allocator in external libraries), PyMem_Malloc() is usually not
appropriate. PyMem_RawMalloc() should also be used instead of
PyMem_Malloc() in the Python startup sequence, because PyMem_Malloc()
requires the GIL whereas the GIL does not exist yet.

PyMem_RawMalloc() also provides more accurate memory usage if it can
be replaced or hooked (with PyMem_SetRawAllocators).

The issue #18203 explains why I would like to replace direct call to
malloc() with PyMem_Malloc() or PyMem_RawMalloc().

> I don't like the idea of adding of third layer of allocation APIs. The
> dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit
> gratuitous (i.e. not motivated by any actual real-world concern, as
> far as I can tell).

In Python 3.3, PyMem_Malloc() cannot be used instead of malloc() where
the GIL is not held. Instead of adding PyMem_RawMalloc(), an
alternative is to remove the "the GIL must be held" restriction from
PyMem_Malloc() by changing PyMem_Malloc() to make it always call
malloc() (instead of PyObject_Malloc() in debug mode).

With such change, a debug hook cannot rely on the GIL anymore: it
cannot inspect Python objects, get a frame or traceback, etc. To still
get accurate debug report, PyMem_Malloc() should be replaced with
PyObject_Malloc().

I don't understand yet the effect of such change on backport
compatibility. May it break applications?

> As for the debug functions you added: PyMem_GetRawAllocators(),
> PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(),
> PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(),
> _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't
> you try to make that 2 or 3?

Get/SetAllocators of PyMem, PyMem_Raw and PyObject can be grouped into
2 functions (get and set) with an argument to select the API.

It is what I proposed initially. I changed this when I had to choose a
name for the name of the argument ("api", "domain",  something else?)
because there were only two choices. With 3 family of functions
(PyMem, PyMem_Raw and PyObject), it becomes again interesting to have
generic functions.

The arena case is different: pymalloc only uses two functions to
allocate areneas: void* alloc(size_t) and void release(void*, size_t).
The release function has a size argument, which is unusual, but
require to implement it using munmap(). VirtualFree() on Windows
requires also the size.

An application can choose to replace PyObject_Malloc() with its own
allocator, but in my experience, it has an important impact on
performance (Python is slower). To benefit of pymalloc with a custom
memory allocator, _PyObject_SetArenaAllocators() can be used.

I kept _PyObject_SetArenaAllocators() private because I don't like its
API, it is not homogenous with the other SetAllocators functions. I'm
not sure that it would be used, so I prefer to keep it private until
it is tested by some projects.

"Private" functions can be used by applications, it's just that Python
doesn't give any backward compatibility warranty. Am I right?

Victor

From victor.stinner at gmail.com  Sun Jun 16 02:25:16 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 16 Jun 2013 02:25:16 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
Message-ID: <CAMpsgwYFjGBjJorqjEuJ4PSyf29kR3ideMPvNUdi_PvtV7Fvvg@mail.gmail.com>

2013/6/15 Nick Coghlan <ncoghlan at gmail.com>:
> The only reason for the small object allocator to exist is because
> operating system allocators generally aren't optimised for frequent
> allocation and deallocation of small objects. You can gain a *lot* of
> speed from handling those inside the application. As the allocations
> grow in size, though, the application level allocator just becomes
> useless overhead, so it's better to delegate those operations directly
> to the OS.

Why not using PyObject_Malloc() for all allocations? PyObject_Malloc()
fallbacks to malloc() if the size is larger than a threshold (512
bytes in Python 3.4).

Are PyObject_Realloc() and PyObject_Free() more expensive than
realloc() and free() (when the memory was allocated by malloc)?

> The only question mark in my mind is over the GIL-free raw allocation
> APIs. I think it makes sense to at least conditionally define those as
> macros so an embedding application can redirect *just* the allocations
> made by the CPython runtime (rather than having to redefine
> malloc/realloc/free when building Python), but I don't believe the
> case has been adequately made for making the raw APIs configurable at
> runtime. Dropping that aspect would at least eliminate the
> PyMem_(Get|Set)RawAllocators() APIs.

PyMem_SetRawAllocators() is required for the two use cases: use a
custom memory allocator (embedded device and Python embedded in an
application) and setup an hook for debug purpose.

Without PyMem_SetRawAllocators(), allocations made by
PyMem_RawMalloc() would go to the same place than the rest of the
"Python memory", nor seen by debug tools. It becomes worse with large
allocations kept for a long time.

Victor

From victor.stinner at gmail.com  Sun Jun 16 02:51:29 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 16 Jun 2013 02:51:29 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130615163839.61a2d669@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<20130615144115.0a7e3176@fsol>
	<CADiSq7f5b3z-fjYUkAp2e06oYd8hBpoJKF5uG91mXZ-ido8CnQ@mail.gmail.com>
	<20130615163839.61a2d669@fsol>
Message-ID: <CAMpsgwYPqsZwaORv7ne88u=JWKsJUm1nBCHZ8D1o-pZbJ5S=sw@mail.gmail.com>

2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> Moreover, I think you are conflating two issues: the ability to add
> memory allocation hooks (for tracing/debugging purposes), and the
> adaptation to "non-traditional" memory models (whatever that means).
> Those concerns don't necessarily come together.

In my implementation, both uses case use the same API:
PyMem_SetAllocators(), except that hooks need also
PyMem_GetAllocators().

Victor

From donald at stufft.io  Sun Jun 16 03:50:37 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 15 Jun 2013 21:50:37 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <51BCE008.5070308@stoneleaf.us>
References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol>
	<51BCE008.5070308@stoneleaf.us>
Message-ID: <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io>


On Jun 15, 2013, at 5:43 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 06/15/2013 01:53 PM, Antoine Pitrou wrote:
>> On Sat, 15 Jun 2013 12:46:32 -0700
>> Ethan Furman <ethan at stoneleaf.us> wrote:
>>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series.
>>> 
>>> I would like to put this on PyPI, but the `enum` name is already taken.
>>> 
>>> Would it be inappropriate to call it `stdlib.enum`?
>> 
>> `backport.enum`?
> 
> Some other well-meaning soul has already claimed that name, but that one only covers 2.6 and 2.7.
> 
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I claimed backport.enum, but you're welcome to the name. I was going to try and backport this PEP under that name anyways.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/aff652fa/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130615/aff652fa/attachment.pgp>

From ron3200 at gmail.com  Sun Jun 16 04:00:13 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 15 Jun 2013 21:00:13 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJ+yPAW9ZRv2OdF0507tC9LCURvvN4eTVVGTcBiHueS5uA@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BCC769.8070904@gmail.com>
	<CAP7+vJ+yPAW9ZRv2OdF0507tC9LCURvvN4eTVVGTcBiHueS5uA@mail.gmail.com>
Message-ID: <51BD1C2D.2050101@gmail.com>



On 06/15/2013 03:23 PM, Guido van Rossum wrote:
> The semantics of raw strings are clear. I don't see that they should be
> called out especially in any context. (Except for regexps.) Usually exec()
> is not used with a literal anyway (what would be the point).


There are about a hundred instances of eval/exec(some_string_literal) in 
pythons library.  Most of them in the tests, and maybe about half of those 
testing the compiler, eval, and exec.

egrep -owr --include="*.py" "(eval|exec)\(('.*'|\".*\")\)" * | wc -l
114


I have no idea in how many places a string literal is assigned to a name 
first and then used later in eval or exec.  It's harder to grep for but 
would be less than...

egrep -owr --include="*.py" "(eval|exec)\(.*\)" * | wc -l
438

That's overstated because some of those are comments, and some may be 
functions with the name ending with eval or exec.


I do think that eval and exec is a similar case to regexps.  And possibly 
often enough, the string may contain a raw string, regular expression, or a 
file/path name.

Only a short note needed in the docs for eval, nothing more.  And not even 
that if no thinks it's an issue.

cheers,
    Ron










From ron3200 at gmail.com  Sun Jun 16 04:00:13 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 15 Jun 2013 21:00:13 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJ+yPAW9ZRv2OdF0507tC9LCURvvN4eTVVGTcBiHueS5uA@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BCC769.8070904@gmail.com>
	<CAP7+vJ+yPAW9ZRv2OdF0507tC9LCURvvN4eTVVGTcBiHueS5uA@mail.gmail.com>
Message-ID: <51BD1C2D.2050101@gmail.com>



On 06/15/2013 03:23 PM, Guido van Rossum wrote:
> The semantics of raw strings are clear. I don't see that they should be
> called out especially in any context. (Except for regexps.) Usually exec()
> is not used with a literal anyway (what would be the point).


There are about a hundred instances of eval/exec(some_string_literal) in 
pythons library.  Most of them in the tests, and maybe about half of those 
testing the compiler, eval, and exec.

egrep -owr --include="*.py" "(eval|exec)\(('.*'|\".*\")\)" * | wc -l
114


I have no idea in how many places a string literal is assigned to a name 
first and then used later in eval or exec.  It's harder to grep for but 
would be less than...

egrep -owr --include="*.py" "(eval|exec)\(.*\)" * | wc -l
438

That's overstated because some of those are comments, and some may be 
functions with the name ending with eval or exec.


I do think that eval and exec is a similar case to regexps.  And possibly 
often enough, the string may contain a raw string, regular expression, or a 
file/path name.

Only a short note needed in the docs for eval, nothing more.  And not even 
that if no thinks it's an issue.

cheers,
    Ron











From ncoghlan at gmail.com  Sun Jun 16 04:23:47 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Jun 2013 12:23:47 +1000
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwYPqsZwaORv7ne88u=JWKsJUm1nBCHZ8D1o-pZbJ5S=sw@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<20130615144115.0a7e3176@fsol>
	<CADiSq7f5b3z-fjYUkAp2e06oYd8hBpoJKF5uG91mXZ-ido8CnQ@mail.gmail.com>
	<20130615163839.61a2d669@fsol>
	<CAMpsgwYPqsZwaORv7ne88u=JWKsJUm1nBCHZ8D1o-pZbJ5S=sw@mail.gmail.com>
Message-ID: <CADiSq7esB_h1m4=KVgk8UKguiccs_k_GHC6kzXbs4s22FYZZjw@mail.gmail.com>

On 16 Jun 2013 10:54, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> > Moreover, I think you are conflating two issues: the ability to add
> > memory allocation hooks (for tracing/debugging purposes), and the
> > adaptation to "non-traditional" memory models (whatever that means).
> > Those concerns don't necessarily come together.
>
> In my implementation, both uses case use the same API:
> PyMem_SetAllocators(), except that hooks need also
> PyMem_GetAllocators().

Right - they're different use cases that share a technical solution, so it
makes sense to consider them together.

Cheers,
Nick.

>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130616/5054a3cd/attachment.html>

From ben+python at benfinney.id.au  Sun Jun 16 04:45:58 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 16 Jun 2013 12:45:58 +1000
Subject: [Python-Dev] backported Enum
References: <51BCC498.1030100@stoneleaf.us>
Message-ID: <7w8v2aepsp.fsf@benfinney.id.au>

Ethan Furman <ethan at stoneleaf.us> writes:

> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back
> to 2.4 in the 2.x series.
>
> I would like to put this on PyPI, but the `enum` name is already
> taken.

I have for a long time approved of ?flufl.enum? becoming the One Obvious
Way to do enumerations, and am glad to see PEP 435 bring it into the
standard library.

<URL:http://mail.python.org/pipermail/python-ideas/2011-July/010811.html>

There are some people still contacting me about their use of the ?enum?
package on PyPI, so I think that to just replace it with a different
code base under the same name would not be helpful.

> Would it be inappropriate to call it `stdlib.enum`?

That sounds good to me.

Is there anything I can do to keep the ?enum? package online for
continuity but make it clear, to automated tools, that this is
end-of-life and obsoleted by another package?

-- 
 \       ?Free thought is a necessary, but not a sufficient, condition |
  `\                                       for democracy.? ?Carl Sagan |
_o__)                                                                  |
Ben Finney


From donald at stufft.io  Sun Jun 16 05:53:35 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 15 Jun 2013 23:53:35 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <7w8v2aepsp.fsf@benfinney.id.au>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
Message-ID: <C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>



On Jun 15, 2013, at 10:45 PM, Ben Finney <ben+python at benfinney.id.au> wrote:

> Ethan Furman <ethan at stoneleaf.us> writes:
> 
>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back
>> to 2.4 in the 2.x series.
>> 
>> I would like to put this on PyPI, but the `enum` name is already
>> taken.
> 
> I have for a long time approved of ?flufl.enum? becoming the One Obvious
> Way to do enumerations, and am glad to see PEP 435 bring it into the
> standard library.
> 
> <URL:http://mail.python.org/pipermail/python-ideas/2011-July/010811.html>
> 
> There are some people still contacting me about their use of the ?enum?
> package on PyPI, so I think that to just replace it with a different
> code base under the same name would not be helpful.
> 
>> Would it be inappropriate to call it `stdlib.enum`?
> 
> That sounds good to me.
> 
> Is there anything I can do to keep the ?enum? package online for
> continuity but make it clear, to automated tools, that this is
> end-of-life and obsoleted by another package?

Right now the best you can do is make a note of it. Pep 426 let's you do what you want. 

> 
> -- 
> \       ?Free thought is a necessary, but not a sufficient, condition |
>  `\                                       for democracy.? ?Carl Sagan |
> _o__)                                                                  |
> Ben Finney
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io

From tjreedy at udel.edu  Sun Jun 16 05:55:43 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 15 Jun 2013 23:55:43 -0400
Subject: [Python-Dev] Clean way in python to test for None, empty, scalar,
 and list/ndarray? A prayer to the gods of Python
In-Reply-To: <kpin9n$80b$1@ger.gmane.org>
References: <CABiiyJLhGNJo5iMCO6s=4HSdSq+=XkCJ6SpoNqamVpJorjftNA@mail.gmail.com>
	<20130614205534.A595A250BE9@webabinitio.net>
	<kpho3d$ao1$1@ger.gmane.org> <kpihqb$l1m$1@ger.gmane.org>
	<kpin9n$80b$1@ger.gmane.org>
Message-ID: <kpjcvr$l8s$1@ger.gmane.org>

On 6/15/2013 5:45 PM, Tres Seaver wrote:

> Given that strings are implemented in C,

That is a current implementation detail. String functions were 
originally written in python in string.py. Some used 'for c in s:'. The 
functions only because methods after 2.2. I presume Pypy starts from 
Python code again.

 > there is no real "iteration"
> happing (in the Python sense) in their methods.

I disagree, but not relevant.

>  What stdlib code can you
> point to that does iterate over them in Python?  I.e.:
>
>    for c in stringval:

Using Idle's grep (Find in files) for ...Python33/Lib/*.py:
'for c in' 193 hits
'for ch in' 30 hits
'for chr in' 0 hits
'for char in' 14 hits.

Some, especially in the 193, are false positives, but most are not. 
There are at least a few other 'for name in string' uses: for instance, 
'for a in _hexdig for b in _hexdig' where _hexdig is 
'0123456789ABCDEFabcdef'.

-- 
Terry Jan Reedy


From ben+python at benfinney.id.au  Sun Jun 16 07:52:33 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 16 Jun 2013 15:52:33 +1000
Subject: [Python-Dev] backported Enum
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
Message-ID: <7wd2rmeh5q.fsf@benfinney.id.au>

Donald Stufft <donald at stufft.io> writes:

> On Jun 15, 2013, at 10:45 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > Is there anything I can do to keep the ?enum? package online for
> > continuity but make it clear, to automated tools, that this is
> > end-of-life and obsoleted by another package?
>
> Right now the best you can do is make a note of it. Pep 426 let's you
> do what you want.

Thanks. What specifically in PEP 426 do you mean; how would I make a
note of ?this package is end-of-life as is, please migrate to
?flufl.enum? instead? using PEP 426 metadata?

-- 
 \        ?There are no significant bugs in our released software that |
  `\         any significant number of users want fixed.? ?Bill Gates, |
_o__)                                                       1995-10-23 |
Ben Finney


From donald at stufft.io  Sun Jun 16 07:58:55 2013
From: donald at stufft.io (Donald Stufft)
Date: Sun, 16 Jun 2013 01:58:55 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <7wd2rmeh5q.fsf@benfinney.id.au>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au>
Message-ID: <CFAE88D1-00B6-45A4-9790-EA9FFB323620@stufft.io>


On Jun 16, 2013, at 1:52 AM, Ben Finney <ben+python at benfinney.id.au> wrote:

> Donald Stufft <donald at stufft.io> writes:
> 
>> On Jun 15, 2013, at 10:45 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
>>> Is there anything I can do to keep the ?enum? package online for
>>> continuity but make it clear, to automated tools, that this is
>>> end-of-life and obsoleted by another package?
>> 
>> Right now the best you can do is make a note of it. Pep 426 let's you
>> do what you want.
> 
> Thanks. What specifically in PEP 426 do you mean; how would I make a
> note of ?this package is end-of-life as is, please migrate to
> ?flufl.enum? instead? using PEP 426 metadata?

http://www.python.org/dev/peps/pep-0426/#obsoleted-by

Note PEP426 isn't completed and isn't implemented :)

> 
> -- 
> \        ?There are no significant bugs in our released software that |
>  `\         any significant number of users want fixed.? ?Bill Gates, |
> _o__)                                                       1995-10-23 |
> Ben Finney
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130616/fc14579d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130616/fc14579d/attachment.pgp>

From ncoghlan at gmail.com  Sun Jun 16 09:31:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Jun 2013 17:31:29 +1000
Subject: [Python-Dev] backported Enum
In-Reply-To: <CFAE88D1-00B6-45A4-9790-EA9FFB323620@stufft.io>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au>
	<CFAE88D1-00B6-45A4-9790-EA9FFB323620@stufft.io>
Message-ID: <CADiSq7ewgNFyPiQpeO1oR-1xVeSiaqbcquQ5p1m=M4Zb9aebtg@mail.gmail.com>

On 16 June 2013 15:58, Donald Stufft <donald at stufft.io> wrote:
>
> On Jun 16, 2013, at 1:52 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
>
> Donald Stufft <donald at stufft.io> writes:
>
> On Jun 15, 2013, at 10:45 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
>
> Is there anything I can do to keep the ?enum? package online for
> continuity but make it clear, to automated tools, that this is
> end-of-life and obsoleted by another package?
>
>
> Right now the best you can do is make a note of it. Pep 426 let's you
> do what you want.
>
>
> Thanks. What specifically in PEP 426 do you mean; how would I make a
> note of ?this package is end-of-life as is, please migrate to
> ?flufl.enum? instead? using PEP 426 metadata?
>
>
> http://www.python.org/dev/peps/pep-0426/#obsoleted-by
>
> Note PEP426 isn't completed and isn't implemented :)

That specific bit is stable, though :)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sun Jun 16 08:48:47 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 15 Jun 2013 23:48:47 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io>
References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol>
	<51BCE008.5070308@stoneleaf.us>
	<644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io>
Message-ID: <51BD5FCF.1040203@stoneleaf.us>

On 06/15/2013 06:50 PM, Donald Stufft wrote:
>
> I claimed backport.enum, but you're welcome to the name. I was going to try and backport this PEP under that name anyways.

Thank you for the offer, but I think I'll go with GPS's idea of calling it enum34.

--
~Ethan~

From ethan at stoneleaf.us  Sun Jun 16 08:53:16 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 15 Jun 2013 23:53:16 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <7wd2rmeh5q.fsf@benfinney.id.au>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au>
Message-ID: <51BD60DC.40701@stoneleaf.us>

On 06/15/2013 10:52 PM, Ben Finney wrote:
> Donald Stufft <donald at stufft.io> writes:
>> On Jun 15, 2013, at 10:45 PM, Ben Finney wrote:
>>>
>>> Is there anything I can do to keep the ?enum? package online for
>>> continuity but make it clear, to automated tools, that this is
>>> end-of-life and obsoleted by another package?
>>
>> Right now the best you can do is make a note of it. Pep 426 let's you
>> do what you want.
>
> Thanks. What specifically in PEP 426 do you mean; how would I make a
> note of ?this package is end-of-life as is, please migrate to
> ?flufl.enum? instead? using PEP 426 metadata?


Note that while flufl.enum was the inspiration, it is not the version that went into 3.4.


Differences include:

   Enum does not allow subclassing when members have been defined

   int(Enum.member) is an error


Similarities include:

   different Enum enumerations do not compare to each other

   members in an Enum enumeration are not ordered


There may be (and probably are) other differences from flufl.enum, but it's late and I'm not remembering them.

So to cut a long story short, your note should say: "migrate to enum34 instead."

Oh, and I'm still learning the intricacies of distutils -- you should be able to ignore any errors if you try to install 
using `easy_install enum34`; I'll hopefully get those kinks worked out in the next few days.

--
~Ethan~

From lukasz at langa.pl  Sun Jun 16 11:30:41 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 16 Jun 2013 11:30:41 +0200
Subject: [Python-Dev] Backports galore
Message-ID: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl>

Now we have (at least) the following libraries backported from 3.2+ to
older versions of Python by members of the core team:

- configparser
- enum34
- singledispatch
- subprocess32
- unittest2

There are also unofficial backports like billiard (multiprocessing).

I would be happy if all those were more discoverable by the community
at large. Having a single namespace for backports would be great but
my spidey sense forebodes large flocks of bike sheds flying that way.

Can we put links to those backports in the docs of older versions of
Python? Most users would be better off using the updated packages
while still deploying on an older release of Python.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev


From solipsis at pitrou.net  Sun Jun 16 11:50:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 16 Jun 2013 11:50:31 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwaVjG7tgM2i=Nt+d495Jcn7rU8COx51C-jeVn-=hyG39w@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<51BC8819.5060106@python.org>
	<CAMpsgwaVjG7tgM2i=Nt+d495Jcn7rU8COx51C-jeVn-=hyG39w@mail.gmail.com>
Message-ID: <20130616115031.6924f071@fsol>

On Sun, 16 Jun 2013 01:48:06 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/6/15 Christian Heimes <christian at python.org>:
> > Am 15.06.2013 14:22, schrieb Nick Coghlan:
> >> However, it's still desirable to be able to monitor those direct
> >> allocations in debug mode, thus it makes sense to have a GIL protected
> >> direct allocation API as well. You could try to hide the existence of
> >> the latter behaviour and treat it as a private API, but why? For
> >> custom allocators, it's useful to be able to *ensure* you can bypass
> >> CPython's small object allocator, rather than having to rely on it
> >> being bypassed for allocations above a certain size.
> >
> > There is even more to it. We like to keep track of memory allocations in
> > libraries that are wrapped by Python's extension modules, e.g. expat,
> > openssl etc. Almost every library has a hook to set a custom memory
> > allocator, either globally (CRYPTO_set_mem_functions) or for each object
> > (XML_ParserCreate_MM's XML_Memory_Handling_Suite).
> 
> I just create the issue http://bugs.python.org/issue18227: "Use Python
> memory allocators in external libraries like zlib or OpenSSL".
> 
> Is it possible to detect if Python is used as a standalone application
> (the classic "python" program) or if Python is embedded? If it is
> possible, we can modify the "global" memory allocators of a library.

The question is why you want to do so, not how/whether to do it.

Regards

Antoine.

From solipsis at pitrou.net  Sun Jun 16 11:56:24 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 16 Jun 2013 11:56:24 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <CAMpsgwY=NbiEowtaM+b8QopTU7t--g=vAV8kH6O-VD9RyfGBdA@mail.gmail.com>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CAMpsgwY=NbiEowtaM+b8QopTU7t--g=vAV8kH6O-VD9RyfGBdA@mail.gmail.com>
Message-ID: <20130616115624.027a8945@fsol>

On Sun, 16 Jun 2013 02:18:32 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:
> > On Sat, 15 Jun 2013 03:54:50 +0200
> > Victor Stinner <victor.stinner at gmail.com> wrote:
> >> The addition of PyMem_RawMalloc() is motivated by the issue #18203
> >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be
> >> able to setup a custom allocator for *all* allocation made by Python,
> >> so malloc() should not be called directly. PyMem_RawMalloc() is
> >> required in places where the GIL is not held (ex: in os.getcwd() on
> >> Windows).
> >
> > We already had this discussion on IRC and this argument isn't very
> > convincing to me. If os.getcwd() doesn't hold the GIL while allocating
> > memory, then you should fix it to hold the GIL while allocating memory.
> 
> The GIL is released for best performances, holding the GIL would have
> an impact on performances.

Well, do you have benchmark numbers? Do you have a workload where
getcwd() is performance-critical to the point that a single
GIL-protected allocation may slow down your program?

> "Private" functions can be used by applications, it's just that Python
> doesn't give any backward compatibility warranty. Am I right?

Anyone "can" use anything obviously, but when it's private, it can be
changed or removed in any release.
If the only goal for these functions is to be used by applications,
though, it's quite a bad idea to make them private.

Regards

Antoine.

From ncoghlan at gmail.com  Sun Jun 16 12:06:03 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Jun 2013 20:06:03 +1000
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130616115031.6924f071@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<51BC8819.5060106@python.org>
	<CAMpsgwaVjG7tgM2i=Nt+d495Jcn7rU8COx51C-jeVn-=hyG39w@mail.gmail.com>
	<20130616115031.6924f071@fsol>
Message-ID: <CADiSq7cfegwkSYK05XJgFMeijYZ_ynnb4R9a9qJxn9f-RzdtAw@mail.gmail.com>

On 16 June 2013 19:50, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 16 Jun 2013 01:48:06 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>
>> 2013/6/15 Christian Heimes <christian at python.org>:
>> > Am 15.06.2013 14:22, schrieb Nick Coghlan:
>> >> However, it's still desirable to be able to monitor those direct
>> >> allocations in debug mode, thus it makes sense to have a GIL protected
>> >> direct allocation API as well. You could try to hide the existence of
>> >> the latter behaviour and treat it as a private API, but why? For
>> >> custom allocators, it's useful to be able to *ensure* you can bypass
>> >> CPython's small object allocator, rather than having to rely on it
>> >> being bypassed for allocations above a certain size.
>> >
>> > There is even more to it. We like to keep track of memory allocations in
>> > libraries that are wrapped by Python's extension modules, e.g. expat,
>> > openssl etc. Almost every library has a hook to set a custom memory
>> > allocator, either globally (CRYPTO_set_mem_functions) or for each object
>> > (XML_ParserCreate_MM's XML_Memory_Handling_Suite).
>>
>> I just create the issue http://bugs.python.org/issue18227: "Use Python
>> memory allocators in external libraries like zlib or OpenSSL".
>>
>> Is it possible to detect if Python is used as a standalone application
>> (the classic "python" program) or if Python is embedded? If it is
>> possible, we can modify the "global" memory allocators of a library.
>
> The question is why you want to do so, not how/whether to do it.

I don't think we should be doing that ourselves - it's up to system
integrators/embedders to configure those libraries if they want to, we
shouldn't be doing it on their behalf.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jun 16 12:32:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 Jun 2013 20:32:02 +1000
Subject: [Python-Dev] Backports galore
In-Reply-To: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl>
References: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl>
Message-ID: <CADiSq7dSzLqoyoeGonOjzBcAGRm08Mch+E9OeBbAA_Q+jbSeyg@mail.gmail.com>

On 16 June 2013 19:30, ?ukasz Langa <lukasz at langa.pl> wrote:
> Now we have (at least) the following libraries backported from 3.2+ to
> older versions of Python by members of the core team:
>
> - configparser
> - enum34
> - singledispatch
> - subprocess32
> - unittest2
>
> There are also unofficial backports like billiard (multiprocessing).

Step one is compiling a list. I started a page for that here:
http://wiki.python.org/moin/StandardLibraryBackports

(It's reachable from the front page of the wiki via the Useful Modules page)

> I would be happy if all those were more discoverable by the community
> at large. Having a single namespace for backports would be great but
> my spidey sense forebodes large flocks of bike sheds flying that way.
>
> Can we put links to those backports in the docs of older versions of
> Python? Most users would be better off using the updated packages
> while still deploying on an older release of Python.

It's probably better to start with a list anyone can edit, rather than
links that can only be updated by core developers.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ben+python at benfinney.id.au  Sun Jun 16 15:17:31 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 16 Jun 2013 23:17:31 +1000
Subject: [Python-Dev] backported Enum
References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol>
	<51BCE008.5070308@stoneleaf.us>
	<644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io>
	<51BD5FCF.1040203@stoneleaf.us>
Message-ID: <7wehc2w5xw.fsf@benfinney.id.au>

Ethan Furman <ethan at stoneleaf.us> writes:

> Thank you for the offer, but I think I'll go with GPS's idea of
> calling it enum34.

That name will be confusing once Python 3.5 exists (even as a
development version). I'd argue that it is confusing even now, since
this is a backport for Python version *earlier* than Python 3.4.

The name ?stdlib.enum? avoids these problems, do you think that is
suitable?

-- 
 \       ?If trees could scream, would we be so cavalier about cutting |
  `\   them down? We might, if they screamed all the time, for no good |
_o__)                                            reason.? ?Jack Handey |
Ben Finney


From barry at python.org  Sun Jun 16 17:19:12 2013
From: barry at python.org (Barry Warsaw)
Date: Sun, 16 Jun 2013 11:19:12 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <51BD60DC.40701@stoneleaf.us>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au> <51BD60DC.40701@stoneleaf.us>
Message-ID: <20130616111912.1a59b0fe@anarchist>

On Jun 15, 2013, at 11:53 PM, Ethan Furman wrote:

>Note that while flufl.enum was the inspiration, it is not the version that
>went into 3.4.

I haven't yet decided whether to continue with development of flufl.enum or
not.  I think there's a lot of value in promoting a standard enum library and
having PEP 435 in both the 3.4 stdlib and in a good backport may outweigh the
differences.  I mostly agree with everything in PEP 435 <wink>.

One big test will be porting Mailman 3 to enum34.  I expect that to mostly
Just Work and I don't think the differences will affect it (I've already
adjusted for the lack of inheritance - mostly I have to see what effect it
will have on the database layer).

OTOH, if folks really depend on the differences in flufl.enum, I will probably
at least continue to maintain it, but not add any new features.

-Barry

From victor.stinner at gmail.com  Sun Jun 16 20:59:26 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 16 Jun 2013 20:59:26 +0200
Subject: [Python-Dev] Backports galore
In-Reply-To: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl>
References: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl>
Message-ID: <CAMpsgwaUh314f3PQzSXjnP9WYhgh86vMFWeU=kg7iZZywm63ug@mail.gmail.com>

There is also faulthandler on PyPI. It is not really a backport since the
project developement started on PyPI.

Victor

Le dimanche 16 juin 2013, ?ukasz Langa a ?crit :

> Now we have (at least) the following libraries backported from 3.2+ to
> older versions of Python by members of the core team:
>
> - configparser
> - enum34
> - singledispatch
> - subprocess32
> - unittest2
>
> There are also unofficial backports like billiard (multiprocessing).
>
> I would be happy if all those were more discoverable by the community
> at large. Having a single namespace for backports would be great but
> my spidey sense forebodes large flocks of bike sheds flying that way.
>
> Can we put links to those backports in the docs of older versions of
> Python? Most users would be better off using the updated packages
> while still deploying on an older release of Python.
>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org <javascript:;>
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130616/f2327462/attachment.html>

From eliben at gmail.com  Sun Jun 16 22:13:00 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 16 Jun 2013 13:13:00 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <20130616111912.1a59b0fe@anarchist>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au>
	<51BD60DC.40701@stoneleaf.us> <20130616111912.1a59b0fe@anarchist>
Message-ID: <CAF-Rda9iagHg+WYmcZqDDKUZ7ka4MjuDiotpasPRed660B_+Vw@mail.gmail.com>

On Sun, Jun 16, 2013 at 8:19 AM, Barry Warsaw <barry at python.org> wrote:

> On Jun 15, 2013, at 11:53 PM, Ethan Furman wrote:
>
> >Note that while flufl.enum was the inspiration, it is not the version that
> >went into 3.4.
>
> I haven't yet decided whether to continue with development of flufl.enum or
> not.  I think there's a lot of value in promoting a standard enum library
> and
> having PEP 435 in both the 3.4 stdlib and in a good backport may outweigh
> the
> differences.  I mostly agree with everything in PEP 435 <wink>.
>
> One big test will be porting Mailman 3 to enum34.  I expect that to mostly
> Just Work and I don't think the differences will affect it (I've already
> adjusted for the lack of inheritance - mostly I have to see what effect it
> will have on the database layer).
>
> OTOH, if folks really depend on the differences in flufl.enum, I will
> probably
> at least continue to maintain it, but not add any new features.
>

If you write down the process of porting mailmain from flufl to stdlib.enum
in some place, it can make the process much easier for others, and even
encourage them to follow the same path.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130616/ea001ee9/attachment.html>

From vinay_sajip at yahoo.co.uk  Mon Jun 17 18:27:45 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 17 Jun 2013 16:27:45 +0000 (UTC)
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
Message-ID: <loom.20130617T180748-379@post.gmane.org>

Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on
Windows with the pydoc script provided with venvs created by pyvenv. On
POSIX, the script is named pydoc and causes no problems: on Windows, it is
called pydoc.py and this causes problems because it shadows the stdlib pydoc
module.

Possible solutions:

1. Remove the pydoc script altogether from created venvs, on Windows but
also on POSIX (for consistency).
2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and
pydocs respectively).
3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
launcher pydoc.exe adjacent to it (which is how setuptools and distlib
handle installed scripts).

The first two approaches are backwards-incompatible, while the third is less
likely to lead to breakage, but involves adding a Windows script launcher to
Python. While this is a bigger change, I think any built-in Python installer
functionality should include such a launcher (as setuptools and distlib do).
Still, that's probably a discussion for another day.

Does anyone have any comments? Approach #2 seems the most appropriate. I
assume it would be reasonable to implement this in both 3.3 and 3.4, as it's
not a change in core Python APIs.

In the absence of adverse feedback here, I propose to implement approach #2
on both 3.3 and 3.4.

Regards,

Vinay Sajip


From adityaavinash143 at gmail.com  Mon Jun 17 18:42:12 2013
From: adityaavinash143 at gmail.com (Aditya Avinash)
Date: Mon, 17 Jun 2013 22:12:12 +0530
Subject: [Python-Dev] Writing Extensions for Python 3 in Windows
Message-ID: <CABocMVrqP4FUyHKD2DJZsTok734egYomJrC5xsPnD68O9g__NA@mail.gmail.com>

Hi. This is the last place where I want to ask a question. I have searched
for lots of tutorials and documentation on the web but, didn't find a
decent one to develop extensions for Python 3 using a custom compiler
(mingw32, nvcc). Please help me.
PS: Don't point me to Python Documentation. It is not good for beginners.
It doesn't elaborate about calls and implementation.

-- 
Aditya Avinash Atluri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130617/be78934c/attachment.html>

From walter at livinglogic.de  Mon Jun 17 19:04:41 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Mon, 17 Jun 2013 19:04:41 +0200
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
Message-ID: <51BF41A9.4040400@livinglogic.de>

On 14.06.13 23:03, PJ Eby wrote:
> On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam <ron3200 at gmail.com> wrote:
>>
>>
>> On 06/14/2013 10:36 AM, Guido van Rossum wrote:
>>>
>>> Not a bug. The same is done for file input -- CRLF is changed to LF before
>>> tokenizing.
>>
>>
>>
>> Should this be the same?
>>
>>
>> python3 -c 'print(bytes("""\r\n""", "utf8"))'
>> b'\r\n'
>>
>>
>>>>> eval('print(bytes("""\r\n""", "utf8"))')
>> b'\n'
>
> No, but:
>
> eval(r'print(bytes("""\r\n""", "utf8"))')
>
> should be.  (And is.)
>
> What I believe you and Walter are missing is that the \r\n in the eval
> strings are converted early if you don't make the enclosing string
> raw.  So what you're eval-ing is not what you think you are eval-ing,
> hence the confusion.

I expected that eval()ing a string that contains the characters

    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+000D: CR
    U+000A: LR
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE

to return a string containing the characters:

    U+000D: CR
    U+000A: LR

Making the string raw, of course turns it into:

    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+005C: REVERSE SOLIDUS
    U+0072: LATIN SMALL LETTER R
    U+005C: REVERSE SOLIDUS
    U+006E: LATIN SMALL LETTER N
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE
    U+0027: APOSTROPHE

and eval()ing that does indeed give "\r\n" as expected.

Hmm, it seems that codecs.unicode_escape_decode() does what I want:

 >>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a")
('\r\n\r\n\r\n\r\n', 26)

Servus,
    Walter


From brett at python.org  Mon Jun 17 19:05:35 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 17 Jun 2013 13:05:35 -0400
Subject: [Python-Dev] Writing Extensions for Python 3 in Windows
In-Reply-To: <CABocMVrqP4FUyHKD2DJZsTok734egYomJrC5xsPnD68O9g__NA@mail.gmail.com>
References: <CABocMVrqP4FUyHKD2DJZsTok734egYomJrC5xsPnD68O9g__NA@mail.gmail.com>
Message-ID: <CAP1=2W6dPjFR6Nhi3u7XNNeeYjaF1i9nAkisYkvWtrUCAgNTcw@mail.gmail.com>

So the best place to ask this question is python-list and not python-dev as
this is about developing *for* Python and not *with* Python.


On Mon, Jun 17, 2013 at 12:42 PM, Aditya Avinash <adityaavinash143 at gmail.com
> wrote:

> Hi. This is the last place where I want to ask a question. I have searched
> for lots of tutorials and documentation on the web but, didn't find a
> decent one to develop extensions for Python 3 using a custom compiler
> (mingw32, nvcc). Please help me.
> PS: Don't point me to Python Documentation. It is not good for beginners.
> It doesn't elaborate about calls and implementation.
>
> --
> Aditya Avinash Atluri
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130617/e86809ea/attachment.html>

From shibturn at gmail.com  Mon Jun 17 19:12:49 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Mon, 17 Jun 2013 18:12:49 +0100
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
In-Reply-To: <loom.20130617T180748-379@post.gmane.org>
References: <loom.20130617T180748-379@post.gmane.org>
Message-ID: <kpng2d$2ja$1@ger.gmane.org>

On 17/06/2013 5:27pm, Vinay Sajip wrote:
> 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
> launcher pydoc.exe adjacent to it (which is how setuptools and distlib
> handle installed scripts).

Can't a batch file pydoc.bat be used?

-- 
Richard


From walter at livinglogic.de  Mon Jun 17 19:14:52 2013
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Mon, 17 Jun 2013 19:14:52 +0200
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF41A9.4040400@livinglogic.de>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
Message-ID: <51BF440C.1090900@livinglogic.de>

On 17.06.13 19:04, Walter D?rwald wrote:

> Hmm, it seems that codecs.unicode_escape_decode() does what I want:
>
>  >>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a")
> ('\r\n\r\n\r\n\r\n', 26)

Hmm, no it doesn't:

 >>> codecs.unicode_escape_decode("\u1234")
('?\x88?', 3)

Servus,
    Walter


From guido at python.org  Mon Jun 17 19:22:01 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Jun 2013 10:22:01 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF41A9.4040400@livinglogic.de>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
Message-ID: <CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>

On Mon, Jun 17, 2013 at 10:04 AM, Walter D?rwald <walter at livinglogic.de> wrote:
> I expected that eval()ing a string that contains the characters
>
>    U+0027: APOSTROPHE
>    U+0027: APOSTROPHE
>    U+0027: APOSTROPHE
>    U+000D: CR
>    U+000A: LR
>    U+0027: APOSTROPHE
>    U+0027: APOSTROPHE
>    U+0027: APOSTROPHE
>
> to return a string containing the characters:
>
>    U+000D: CR
>    U+000A: LR

No. Executing a file containing those exact characters produces a
string containing only '\n' and exec/eval is meant to behave the same
way. The string may not have originated from a file, so the universal
newlines behavior of the io module is irrelevant here -- the parser
must implement its own equivalent processing, and it does.

-- 
--Guido van Rossum (python.org/~guido)

From vinay_sajip at yahoo.co.uk  Mon Jun 17 20:37:49 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 17 Jun 2013 18:37:49 +0000 (UTC)
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
References: <loom.20130617T180748-379@post.gmane.org>
	<kpng2d$2ja$1@ger.gmane.org>
Message-ID: <loom.20130617T203010-289@post.gmane.org>

Richard Oudkerk <shibturn <at> gmail.com> writes:

> 
> Can't a batch file pydoc.bat be used?
> 

I generally find .bat files so limiting that I never thought of that. In
this case, it makes perfect sense to use one. Thanks!

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"

should do it.

Regards,

Vinay Sajip


From rdmurray at bitdance.com  Mon Jun 17 20:45:05 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 17 Jun 2013 14:45:05 -0400
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
In-Reply-To: <loom.20130617T180748-379@post.gmane.org>
References: <loom.20130617T180748-379@post.gmane.org>
Message-ID: <20130617184506.27624250169@webabinitio.net>

On Mon, 17 Jun 2013 16:27:45 -0000, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on
> Windows with the pydoc script provided with venvs created by pyvenv. On
> POSIX, the script is named pydoc and causes no problems: on Windows, it is
> called pydoc.py and this causes problems because it shadows the stdlib pydoc
> module.
> 
> Possible solutions:
> 
> 1. Remove the pydoc script altogether from created venvs, on Windows but
> also on POSIX (for consistency).
> 2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and
> pydocs respectively).
> 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe
> launcher pydoc.exe adjacent to it (which is how setuptools and distlib
> handle installed scripts).
> 
> The first two approaches are backwards-incompatible, while the third is less
> likely to lead to breakage, but involves adding a Windows script launcher to
> Python. While this is a bigger change, I think any built-in Python installer
> functionality should include such a launcher (as setuptools and distlib do).
> Still, that's probably a discussion for another day.
> 
> Does anyone have any comments? Approach #2 seems the most appropriate. I
> assume it would be reasonable to implement this in both 3.3 and 3.4, as it's
> not a change in core Python APIs.
> 
> In the absence of adverse feedback here, I propose to implement approach #2
> on both 3.3 and 3.4.

I don't think renaming pydoc to pydocs is appropriate on posix.  Too many
people likely have 'pydoc' in their fingers and brains as the command name.

The .bat file suggestion seems better, IMO.

--David

From phd at phdru.name  Mon Jun 17 21:15:12 2013
From: phd at phdru.name (Oleg Broytman)
Date: Mon, 17 Jun 2013 23:15:12 +0400
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
In-Reply-To: <loom.20130617T203010-289@post.gmane.org>
References: <loom.20130617T180748-379@post.gmane.org>
	<kpng2d$2ja$1@ger.gmane.org>
	<loom.20130617T203010-289@post.gmane.org>
Message-ID: <20130617191512.GA12889@iskra.aviel.ru>

On Mon, Jun 17, 2013 at 06:37:49PM +0000, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> @echo off
> __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"

   I think you want to pass command line arguments:

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4 %5 %6 %7 %8 %9

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From vinay_sajip at yahoo.co.uk  Mon Jun 17 22:30:54 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 17 Jun 2013 20:30:54 +0000 (UTC)
Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs
References: <loom.20130617T180748-379@post.gmane.org>
	<kpng2d$2ja$1@ger.gmane.org>
	<loom.20130617T203010-289@post.gmane.org>
	<20130617191512.GA12889@iskra.aviel.ru>
Message-ID: <loom.20130617T222842-313@post.gmane.org>

Oleg Broytman <phd <at> phdru.name> writes:

> 
> On Mon, Jun 17, 2013 at 06:37:49PM +0000, Vinay Sajip <vinay_sajip <at>
yahoo.co.uk> wrote:
> >  <at> echo off
> > __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())"
> 
>    I think you want to pass command line arguments:
> 
>  <at> echo off
> __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4
%5 %6 %7 %8 %9
> 

That was off-the-cuff, what I actually implemented (in my sandbox repo) was

@echo off
__VENV_PYTHON__ -c "import sys, pydoc; sys.argv[0] = 'pydoc';
sys.exit(pydoc.cli())" %*

Regards,

Vinay Sajip



From ron3200 at gmail.com  Mon Jun 17 23:52:48 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 17 Jun 2013 16:52:48 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF41A9.4040400@livinglogic.de>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
Message-ID: <kpo0fa$rmk$1@ger.gmane.org>



On 06/17/2013 12:04 PM, Walter D?rwald wrote:
> Making the string raw, of course turns it into:
>
>     U+0027: APOSTROPHE
>     U+0027: APOSTROPHE
>     U+0027: APOSTROPHE
>     U+005C: REVERSE SOLIDUS
>     U+0072: LATIN SMALL LETTER R
>     U+005C: REVERSE SOLIDUS
>     U+006E: LATIN SMALL LETTER N
>     U+0027: APOSTROPHE
>     U+0027: APOSTROPHE
>     U+0027: APOSTROPHE
>
> and eval()ing that does indeed give "\r\n" as expected.

You can also escape the reverse slashes in a regular string to get the same 
result.


 >>> s1 = "'''\\r\\n'''"
 >>> list(s1)
["'", "'", "'", '\\', 'r', '\\', 'n', "'", "'", "'"]

 >>> s2 = eval(s1)
 >>> list(s2)
['\r', '\n']

 >>> s3 = "'''%s'''" % s2
 >>> list(s3)
["'", "'", "'", '\r', '\n', "'", "'", "'"]

 >>> s4 = eval(s3)
 >>> list(s4)
['\n']


When a standard string literal used with eval, it's evaluated first to a 
string object in the same scope as the eval function is called from, then 
the eval function is called with that string object and it's evaluated 
again.  So it's really being parsed twice.  (That's the part that got me.)

The transformation between s1 and s2 is what Phillip is referring to, and
Guido is referring to the transformation from s2 to s4.   (s3 is needed to 
avoid the end of line error of evaluating a single quoted string with \n in 
it.)

When a sting literal is used directly with eval, it looks like it is 
evaluated from s1 to s4 in one step, but that isn't what is happening.

Cheers,  Ron


(ps: Was informed my posts where showing up twice.. hopefully I got that 
fixed now.)















From greg.ewing at canterbury.ac.nz  Tue Jun 18 00:18:24 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 18 Jun 2013 10:18:24 +1200
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
Message-ID: <51BF8B30.70707@canterbury.ac.nz>

Guido van Rossum wrote:
> No. Executing a file containing those exact characters produces a
> string containing only '\n' and exec/eval is meant to behave the same
> way. The string may not have originated from a file, so the universal
> newlines behavior of the io module is irrelevant here -- the parser
> must implement its own equivalent processing, and it does.

I'm still not convinced that this is necessary or desirable
behaviour. I can understand the parser doing this as a
workaround before we had universal newlines, but now that
we do, I'd expect any Python string to already have newlines
converted to their canonical representation, and that any CRs
it contains are meant to be there. The parser shouldn't need
to do newline translation a second time.

-- 
Greg

From guido at python.org  Tue Jun 18 00:35:12 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Jun 2013 15:35:12 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF8B30.70707@canterbury.ac.nz>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
Message-ID: <CAP7+vJLcHV6hAMs=Rq=GRUwQMmt2S6vh8pii1yregrhswDP2gQ@mail.gmail.com>

On Mon, Jun 17, 2013 at 3:18 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>>
>> No. Executing a file containing those exact characters produces a
>> string containing only '\n' and exec/eval is meant to behave the same
>> way. The string may not have originated from a file, so the universal
>> newlines behavior of the io module is irrelevant here -- the parser
>> must implement its own equivalent processing, and it does.

> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.

There are other ways to get a string besides reading it from a file.

Anyway, I think that if you want a string literal that contains \r\n
as its line endings, you should use a syntactic solution, and the
syntax ought to be the same regardless of whether you are reading it
from a file or from a string literal. That syntactic solution is very
clear:

"""line one\r
line two\r
line three\r
"""

This works everywhere.

-- 
--Guido van Rossum (python.org/~guido)

From benjamin at python.org  Tue Jun 18 01:40:01 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 17 Jun 2013 16:40:01 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF8B30.70707@canterbury.ac.nz>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
Message-ID: <CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>

2013/6/17 Greg Ewing <greg.ewing at canterbury.ac.nz>:
> Guido van Rossum wrote:
>>
>> No. Executing a file containing those exact characters produces a
>> string containing only '\n' and exec/eval is meant to behave the same
>> way. The string may not have originated from a file, so the universal
>> newlines behavior of the io module is irrelevant here -- the parser
>> must implement its own equivalent processing, and it does.
>
>
> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.

It used to be that way until 2.7. People like to do things like

    with open("myfile.py", "rb") as fp:
        exec fp.read() in ns

which used to fail with CRLF newlines because binary mode doesn't have
them. I think this is actually the correct way to execute Python
sources because the parser then handles the somewhat complicated
process of decoding Python source for you.



--
Regards,
Benjamin

From guido at python.org  Tue Jun 18 01:52:21 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Jun 2013 16:52:21 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
Message-ID: <CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>

On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2013/6/17 Greg Ewing <greg.ewing at canterbury.ac.nz>:
>> Guido van Rossum wrote:
>>>
>>> No. Executing a file containing those exact characters produces a
>>> string containing only '\n' and exec/eval is meant to behave the same
>>> way. The string may not have originated from a file, so the universal
>>> newlines behavior of the io module is irrelevant here -- the parser
>>> must implement its own equivalent processing, and it does.
>>
>>
>> I'm still not convinced that this is necessary or desirable
>> behaviour. I can understand the parser doing this as a
>> workaround before we had universal newlines, but now that
>> we do, I'd expect any Python string to already have newlines
>> converted to their canonical representation, and that any CRs
>> it contains are meant to be there. The parser shouldn't need
>> to do newline translation a second time.
>
> It used to be that way until 2.7. People like to do things like
>
>     with open("myfile.py", "rb") as fp:
>         exec fp.read() in ns
>
> which used to fail with CRLF newlines because binary mode doesn't have
> them. I think this is actually the correct way to execute Python
> sources because the parser then handles the somewhat complicated
> process of decoding Python source for you.

What exactly does the parser handles better than the io module? Is it
just the coding cookies? I suppose that works as long as the file is
encoded using as ASCII superset like the Latin-N variants or UTF-8. It
would fail pretty badly if it was UTF-16 (and yes, that's an
abominable encoding for other reasons :-).
-- 
--Guido van Rossum (python.org/~guido)

From benjamin at python.org  Tue Jun 18 02:02:26 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 17 Jun 2013 17:02:26 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
Message-ID: <CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>

2013/6/17 Guido van Rossum <guido at python.org>:
> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2013/6/17 Greg Ewing <greg.ewing at canterbury.ac.nz>:
>>> Guido van Rossum wrote:
>>>>
>>>> No. Executing a file containing those exact characters produces a
>>>> string containing only '\n' and exec/eval is meant to behave the same
>>>> way. The string may not have originated from a file, so the universal
>>>> newlines behavior of the io module is irrelevant here -- the parser
>>>> must implement its own equivalent processing, and it does.
>>>
>>>
>>> I'm still not convinced that this is necessary or desirable
>>> behaviour. I can understand the parser doing this as a
>>> workaround before we had universal newlines, but now that
>>> we do, I'd expect any Python string to already have newlines
>>> converted to their canonical representation, and that any CRs
>>> it contains are meant to be there. The parser shouldn't need
>>> to do newline translation a second time.
>>
>> It used to be that way until 2.7. People like to do things like
>>
>>     with open("myfile.py", "rb") as fp:
>>         exec fp.read() in ns
>>
>> which used to fail with CRLF newlines because binary mode doesn't have
>> them. I think this is actually the correct way to execute Python
>> sources because the parser then handles the somewhat complicated
>> process of decoding Python source for you.
>
> What exactly does the parser handles better than the io module? Is it
> just the coding cookies? I suppose that works as long as the file is
> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
> would fail pretty badly if it was UTF-16 (and yes, that's an
> abominable encoding for other reasons :-).

The coding cookie is the main one. In fact, if you can't parse that,
you don't really know what encoding to open the file with at all.
There's also small things like BOM handling (you have to use the
utf-16-sig encoding with TextIO to get it removed) and defaulting to
UTF-8 (which the io module doesn't do) which is better left to the
parser.


--
Regards,
Benjamin

From guido at python.org  Tue Jun 18 02:06:03 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Jun 2013 17:06:03 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
	<CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
Message-ID: <CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>

On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2013/6/17 Guido van Rossum <guido at python.org>:
>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2013/6/17 Greg Ewing <greg.ewing at canterbury.ac.nz>:
>>>> Guido van Rossum wrote:
>>>>>
>>>>> No. Executing a file containing those exact characters produces a
>>>>> string containing only '\n' and exec/eval is meant to behave the same
>>>>> way. The string may not have originated from a file, so the universal
>>>>> newlines behavior of the io module is irrelevant here -- the parser
>>>>> must implement its own equivalent processing, and it does.
>>>>
>>>>
>>>> I'm still not convinced that this is necessary or desirable
>>>> behaviour. I can understand the parser doing this as a
>>>> workaround before we had universal newlines, but now that
>>>> we do, I'd expect any Python string to already have newlines
>>>> converted to their canonical representation, and that any CRs
>>>> it contains are meant to be there. The parser shouldn't need
>>>> to do newline translation a second time.
>>>
>>> It used to be that way until 2.7. People like to do things like
>>>
>>>     with open("myfile.py", "rb") as fp:
>>>         exec fp.read() in ns
>>>
>>> which used to fail with CRLF newlines because binary mode doesn't have
>>> them. I think this is actually the correct way to execute Python
>>> sources because the parser then handles the somewhat complicated
>>> process of decoding Python source for you.
>>
>> What exactly does the parser handles better than the io module? Is it
>> just the coding cookies? I suppose that works as long as the file is
>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>> would fail pretty badly if it was UTF-16 (and yes, that's an
>> abominable encoding for other reasons :-).
>
> The coding cookie is the main one. In fact, if you can't parse that,
> you don't really know what encoding to open the file with at all.
> There's also small things like BOM handling (you have to use the
> utf-16-sig encoding with TextIO to get it removed) and defaulting to
> UTF-8 (which the io module doesn't do) which is better left to the
> parser.

Maybe there are some lessons here that the TextIO module could learn?

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Tue Jun 18 02:22:18 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 18 Jun 2013 02:22:18 +0200
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
	<CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
	<CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
Message-ID: <CAMpsgwYFGhEKki2cB8Ga+VypsmXg-UjtzUge7tBFOz8EPWwN2w@mail.gmail.com>

It may be possible to implement parsing the codec cookie as a Python codec :-)

Victor

2013/6/18 Guido van Rossum <guido at python.org>:
> On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2013/6/17 Guido van Rossum <guido at python.org>:
>>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>>> 2013/6/17 Greg Ewing <greg.ewing at canterbury.ac.nz>:
>>>>> Guido van Rossum wrote:
>>>>>>
>>>>>> No. Executing a file containing those exact characters produces a
>>>>>> string containing only '\n' and exec/eval is meant to behave the same
>>>>>> way. The string may not have originated from a file, so the universal
>>>>>> newlines behavior of the io module is irrelevant here -- the parser
>>>>>> must implement its own equivalent processing, and it does.
>>>>>
>>>>>
>>>>> I'm still not convinced that this is necessary or desirable
>>>>> behaviour. I can understand the parser doing this as a
>>>>> workaround before we had universal newlines, but now that
>>>>> we do, I'd expect any Python string to already have newlines
>>>>> converted to their canonical representation, and that any CRs
>>>>> it contains are meant to be there. The parser shouldn't need
>>>>> to do newline translation a second time.
>>>>
>>>> It used to be that way until 2.7. People like to do things like
>>>>
>>>>     with open("myfile.py", "rb") as fp:
>>>>         exec fp.read() in ns
>>>>
>>>> which used to fail with CRLF newlines because binary mode doesn't have
>>>> them. I think this is actually the correct way to execute Python
>>>> sources because the parser then handles the somewhat complicated
>>>> process of decoding Python source for you.
>>>
>>> What exactly does the parser handles better than the io module? Is it
>>> just the coding cookies? I suppose that works as long as the file is
>>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>>> would fail pretty badly if it was UTF-16 (and yes, that's an
>>> abominable encoding for other reasons :-).
>>
>> The coding cookie is the main one. In fact, if you can't parse that,
>> you don't really know what encoding to open the file with at all.
>> There's also small things like BOM handling (you have to use the
>> utf-16-sig encoding with TextIO to get it removed) and defaulting to
>> UTF-8 (which the io module doesn't do) which is better left to the
>> parser.
>
> Maybe there are some lessons here that the TextIO module could learn?
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From ron3200 at gmail.com  Tue Jun 18 02:44:21 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 17 Jun 2013 19:44:21 -0500
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BF8B30.70707@canterbury.ac.nz>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
Message-ID: <kpoagv$qk2$1@ger.gmane.org>



On 06/17/2013 05:18 PM, Greg Ewing wrote:
> I'm still not convinced that this is necessary or desirable
> behaviour. I can understand the parser doing this as a
> workaround before we had universal newlines, but now that
> we do, I'd expect any Python string to already have newlines
> converted to their canonical representation, and that any CRs
> it contains are meant to be there. The parser shouldn't need
> to do newline translation a second time.


It's the other way around.

Eval and exec should generate the same results as pythons compiler with the 
same input, including errors and exceptions.  The only way we can have that 
is if eval and exec parses everything the same way.

It's the first parsing that needs to be avoided or compensated for in these 
cases.  Raw strings (my preference) works for string literals, or you can 
escape the escape codes so they are still individual characters after the 
first translation.  Or read the code directly from a file rather than 
importing it.

For example, if you wrote your own python console program, you would want 
all the errors and exceptions to come from eval, including those for bad 
strings.  You would still need to feed the bad strings to eval.  If you 
don't then you won't get the same output from eval as the compiler does.

Cheers,
    Ron


From benjamin at python.org  Tue Jun 18 02:49:10 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 17 Jun 2013 17:49:10 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
	<CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
	<CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
Message-ID: <CAPZV6o-0B1dBBza69zJoLVvDpa35DOW+Z2306dvnrx9g9-XBAA@mail.gmail.com>

2013/6/17 Guido van Rossum <guido at python.org>:
> On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2013/6/17 Guido van Rossum <guido at python.org>:
>>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> What exactly does the parser handles better than the io module? Is it
>>> just the coding cookies? I suppose that works as long as the file is
>>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It
>>> would fail pretty badly if it was UTF-16 (and yes, that's an
>>> abominable encoding for other reasons :-).
>>
>> The coding cookie is the main one. In fact, if you can't parse that,
>> you don't really know what encoding to open the file with at all.
>> There's also small things like BOM handling (you have to use the
>> utf-16-sig encoding with TextIO to get it removed) and defaulting to
>> UTF-8 (which the io module doesn't do) which is better left to the
>> parser.
>
> Maybe there are some lessons here that the TextIO module could learn?

UTF-8 by default would be great, but that ship has sailed.

Reading Python coding cookies is outside the purview of TextIOWrapper.
However, it would be good to have a function in the stdlib to read a
python source file to Unicode; I've definitely implemented that
several times.



--
Regards,
Benjamin

From victor.stinner at gmail.com  Tue Jun 18 03:03:07 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 18 Jun 2013 03:03:07 +0200
Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize
 memory allocators
In-Reply-To: <20130616115031.6924f071@fsol>
References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol>
	<CAMpsgwbMB+9xKxjCy1-S-vLSxU8zL7ejoWZQK-UO032VdyGt-g@mail.gmail.com>
	<20130615130125.1d4ef1f7@fsol>
	<CADiSq7eVPTmVbN79fAUHB0jbr3anC96-hgH4Qj_sN4KXc+mDPg@mail.gmail.com>
	<51BC8819.5060106@python.org>
	<CAMpsgwaVjG7tgM2i=Nt+d495Jcn7rU8COx51C-jeVn-=hyG39w@mail.gmail.com>
	<20130616115031.6924f071@fsol>
Message-ID: <CAMpsgwZ126KKYRP83kpfSZSxQzfv-3Cne8W4yttf7dsFKdagdQ@mail.gmail.com>

2013/6/16 Antoine Pitrou <solipsis at pitrou.net>:
> On Sun, 16 Jun 2013 01:48:06 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>> I just create the issue http://bugs.python.org/issue18227: "Use Python
>> memory allocators in external libraries like zlib or OpenSSL".
>>
>> Is it possible to detect if Python is used as a standalone application
>> (the classic "python" program) or if Python is embedded? If it is
>> possible, we can modify the "global" memory allocators of a library.
>
> The question is why you want to do so, not how/whether to do it.

I want to be able to track the memory usage of all Python memory, even
in external libraries, and/or use a custom memory allocator, even in
external libraries.

Victor

From merwok at netwok.org  Tue Jun 18 03:07:42 2013
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 17 Jun 2013 21:07:42 -0400
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <CAPZV6o-0B1dBBza69zJoLVvDpa35DOW+Z2306dvnrx9g9-XBAA@mail.gmail.com>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
	<CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
	<CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
	<CAPZV6o-0B1dBBza69zJoLVvDpa35DOW+Z2306dvnrx9g9-XBAA@mail.gmail.com>
Message-ID: <51BFB2DE.2050403@netwok.org>

Le 17/06/2013 20:49, Benjamin Peterson a ?crit :
> Reading Python coding cookies is outside the purview of TextIOWrapper.
> However, it would be good to have a function in the stdlib to read a
> python source file to Unicode; I've definitely implemented that
> several times.

IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open
(3.2+).

Regards

From benjamin at python.org  Tue Jun 18 03:28:59 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 17 Jun 2013 18:28:59 -0700
Subject: [Python-Dev] eval and triple quoted strings
In-Reply-To: <51BFB2DE.2050403@netwok.org>
References: <51BB2EDC.80604@livinglogic.de>
	<CAP7+vJK6aAWpvnvWcyDQtuOqf-UiyvNYf9dzWpVd3yFjYkoQ5g@mail.gmail.com>
	<51BB5CBF.5070308@gmail.com>
	<CALeMXf5cdT9bf01FjfHK69KcJ_9sHPpdBXH-x=WK9tG4JQf19g@mail.gmail.com>
	<51BF41A9.4040400@livinglogic.de>
	<CAP7+vJKy11xS-h9v2jwBuPeZ4AQbqTrDJa1k7zhbA5fU3Gm_-A@mail.gmail.com>
	<51BF8B30.70707@canterbury.ac.nz>
	<CAPZV6o_JCUYLxUk=j_YH0W=w15fFW9ZNh-Ju-+a2Q=2f9eMNCw@mail.gmail.com>
	<CAP7+vJ+i6sEscypvCuBX1juQPg_OxqBsziZLkw2psyiSN46oZg@mail.gmail.com>
	<CAPZV6o8DQi5DNciJEB6_an_AuxVEkzJEcG_VM-DqUCPd109PxQ@mail.gmail.com>
	<CAP7+vJ+Y++4esuGwCUaRc9vT2hHZP07cuxf_RNTYpXdRzMWdiQ@mail.gmail.com>
	<CAPZV6o-0B1dBBza69zJoLVvDpa35DOW+Z2306dvnrx9g9-XBAA@mail.gmail.com>
	<51BFB2DE.2050403@netwok.org>
Message-ID: <CAPZV6o_8q7azD7o5HRtDG-cfMjKjOPcYh7OY16oc5=QksCUGPA@mail.gmail.com>

2013/6/17 ?ric Araujo <merwok at netwok.org>:
> Le 17/06/2013 20:49, Benjamin Peterson a ?crit :
>> Reading Python coding cookies is outside the purview of TextIOWrapper.
>> However, it would be good to have a function in the stdlib to read a
>> python source file to Unicode; I've definitely implemented that
>> several times.
>
> IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open
> (3.2+).

Yep. :)



--
Regards,
Benjamin

From eliben at gmail.com  Tue Jun 18 06:05:53 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 17 Jun 2013 21:05:53 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <kpf6u5$826$1@ger.gmane.org>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
Message-ID: <CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>

On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka <storchaka at gmail.com>wrote:

> 14.06.13 11:46, Antoine Pitrou ???????(??):
>
>> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
>> raymond.hettinger <python-checkins at python.org> wrote:
>>
>>> http://hg.python.org/cpython/**rev/5accb0ac8bfb<http://hg.python.org/cpython/rev/5accb0ac8bfb>
>>> changeset:   84116:5accb0ac8bfb
>>>    Fix comment blocks.  Adjust blocksize to a power-of-two for better
>>> divmod computations.
>>>
>>
>> Is there any rationale for changing the heuristic from "fits in a whole
>> number of cachelines" to "allows fast division by the blocksize"?
>>
>> I personally would prefer if such changes were justified a bit more
>> than by a one-liner changeset message without any corresponding open
>> issue.
>>
>
> I share the doubts of Antoine and I was going to write the same comment. I
> thought there were good reasons for previous code. What has changed?
>

I agree it would be interesting to hear about the reasons for the change.
Raymond?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130617/f87c2c70/attachment.html>

From ethan at stoneleaf.us  Tue Jun 18 08:08:22 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 17 Jun 2013 23:08:22 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
Message-ID: <51BFF956.7040403@stoneleaf.us>

On 06/17/2013 09:05 PM, Eli Bendersky wrote:
>
>
>
> On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka <storchaka at gmail.com <mailto:storchaka at gmail.com>> wrote:
>
>     14.06.13 11:46, Antoine Pitrou ???????(??):
>
>         On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
>         raymond.hettinger <python-checkins at python.org <mailto:python-checkins at python.org>> wrote:
>
>             http://hg.python.org/cpython/__rev/5accb0ac8bfb <http://hg.python.org/cpython/rev/5accb0ac8bfb>
>             changeset:   84116:5accb0ac8bfb
>                 Fix comment blocks.  Adjust blocksize to a power-of-two for better divmod computations.
>
>
>         Is there any rationale for changing the heuristic from "fits in a whole
>         number of cachelines" to "allows fast division by the blocksize"?
>
>         I personally would prefer if such changes were justified a bit more
>         than by a one-liner changeset message without any corresponding open
>         issue.
>
>
>     I share the doubts of Antoine and I was going to write the same comment. I thought there were good reasons for
>     previous code. What has changed?
>
>
> I agree it would be interesting to hear about the reasons for the change. Raymond?

Asking as a learner:  are such non-bugfix changes appropriate for the 2.7 line?

--
~Ethan~

From greg at krypto.org  Tue Jun 18 09:00:53 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 18 Jun 2013 00:00:53 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <51BFF956.7040403@stoneleaf.us>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
Message-ID: <CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>

Raymond -

Why did you do this in the 2.7 branch?

It hasn't been done in 3.3 or default and it isn't not the sort of change
we make in a stable release branch without justification.  What issue was
this for?  What problem were you solving?

-gps



On Mon, Jun 17, 2013 at 11:08 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 06/17/2013 09:05 PM, Eli Bendersky wrote:
>
>>
>>
>>
>> On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka <storchaka at gmail.com<mailto:
>> storchaka at gmail.com>> wrote:
>>
>>     14.06.13 11:46, Antoine Pitrou ???????(??):
>>
>>         On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
>>         raymond.hettinger <python-checkins at python.org <mailto:
>> python-checkins@**python.org <python-checkins at python.org>>> wrote:
>>
>>             http://hg.python.org/cpython/_**_rev/5accb0ac8bfb<http://hg.python.org/cpython/__rev/5accb0ac8bfb><
>> http://hg.python.org/cpython/**rev/5accb0ac8bfb<http://hg.python.org/cpython/rev/5accb0ac8bfb>
>> >
>>
>>             changeset:   84116:5accb0ac8bfb
>>                 Fix comment blocks.  Adjust blocksize to a power-of-two
>> for better divmod computations.
>>
>>
>>         Is there any rationale for changing the heuristic from "fits in a
>> whole
>>         number of cachelines" to "allows fast division by the blocksize"?
>>
>>         I personally would prefer if such changes were justified a bit
>> more
>>         than by a one-liner changeset message without any corresponding
>> open
>>         issue.
>>
>>
>>     I share the doubts of Antoine and I was going to write the same
>> comment. I thought there were good reasons for
>>     previous code. What has changed?
>>
>>
>> I agree it would be interesting to hear about the reasons for the change.
>> Raymond?
>>
>
> Asking as a learner:  are such non-bugfix changes appropriate for the 2.7
> line?
>
> --
> ~Ethan~
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> greg%40krypto.org<http://mail.python.org/mailman/options/python-dev/greg%40krypto.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130618/77af5e6e/attachment.html>

From jeremy.kloth at gmail.com  Tue Jun 18 12:56:37 2013
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Tue, 18 Jun 2013 04:56:37 -0600
Subject: [Python-Dev] [Python-checkins] cpython (3.3): ctypes: AIX needs
 an explicit #include <alloca.h> to get alloca()
In-Reply-To: <3bZ3HS2pvhzQxV@mail.python.org>
References: <3bZ3HS2pvhzQxV@mail.python.org>
Message-ID: <CAGvrs3JXKUB5eRcNsTn+FC_PvVCau1-O4ajDHY2YPp652-iNTA@mail.gmail.com>

On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner
<python-checkins at python.org> wrote:
> diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
> --- a/Modules/_ctypes/callproc.c
> +++ b/Modules/_ctypes/callproc.c
> @@ -70,6 +70,7 @@
>
>  #include <ffi.h>
>  #include "ctypes.h"
> +#include <alloca.h>

This header is not present on Windows, thus breaking all the Windows
buildbots.  Perhaps it should be wrapped in an AIX-specific #ifdef?

-- 
Jeremy Kloth

From christian at python.org  Tue Jun 18 13:28:13 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 18 Jun 2013 13:28:13 +0200
Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit
 #include <alloca.h> to get alloca()
In-Reply-To: <CAGvrs3JXKUB5eRcNsTn+FC_PvVCau1-O4ajDHY2YPp652-iNTA@mail.gmail.com>
References: <3bZ3HS2pvhzQxV@mail.python.org>
	<CAGvrs3JXKUB5eRcNsTn+FC_PvVCau1-O4ajDHY2YPp652-iNTA@mail.gmail.com>
Message-ID: <51C0444D.8030402@python.org>

Am 18.06.2013 12:56, schrieb Jeremy Kloth:
> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner
> <python-checkins at python.org> wrote:
>> diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
>> --- a/Modules/_ctypes/callproc.c
>> +++ b/Modules/_ctypes/callproc.c
>> @@ -70,6 +70,7 @@
>>
>>  #include <ffi.h>
>>  #include "ctypes.h"
>> +#include <alloca.h>
> 
> This header is not present on Windows, thus breaking all the Windows
> buildbots.  Perhaps it should be wrapped in an AIX-specific #ifdef?

I have added HAVE_ALLOCA_H to configure:

http://hg.python.org/cpython/rev/7b6ae19dd116

Christian

From victor.stinner at gmail.com  Tue Jun 18 13:32:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 18 Jun 2013 13:32:14 +0200
Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit
 #include <alloca.h> to get alloca()
In-Reply-To: <51C0444D.8030402@python.org>
References: <3bZ3HS2pvhzQxV@mail.python.org>
	<CAGvrs3JXKUB5eRcNsTn+FC_PvVCau1-O4ajDHY2YPp652-iNTA@mail.gmail.com>
	<51C0444D.8030402@python.org>
Message-ID: <CAMpsgwb=5Gtc2MHUhfZC5trofVXxJASCbaZQC=Oz7K-aG4GdBg@mail.gmail.com>

2013/6/18 Christian Heimes <christian at python.org>:
> Am 18.06.2013 12:56, schrieb Jeremy Kloth:
>> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner
>>> +#include <alloca.h>
>>
>> This header is not present on Windows, thus breaking all the Windows
>> buildbots.  Perhaps it should be wrapped in an AIX-specific #ifdef?

Oh really? Portability is complex :-) I only tested on Linux, but I
expected this header to be part of the C standard...

> I have added HAVE_ALLOCA_H to configure:
>
> http://hg.python.org/cpython/rev/7b6ae19dd116

Cool, thanks.

Victor

From christian at python.org  Tue Jun 18 13:45:05 2013
From: christian at python.org (Christian Heimes)
Date: Tue, 18 Jun 2013 13:45:05 +0200
Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit
 #include <alloca.h> to get alloca()
In-Reply-To: <CAMpsgwb=5Gtc2MHUhfZC5trofVXxJASCbaZQC=Oz7K-aG4GdBg@mail.gmail.com>
References: <3bZ3HS2pvhzQxV@mail.python.org>
	<CAGvrs3JXKUB5eRcNsTn+FC_PvVCau1-O4ajDHY2YPp652-iNTA@mail.gmail.com>
	<51C0444D.8030402@python.org>
	<CAMpsgwb=5Gtc2MHUhfZC5trofVXxJASCbaZQC=Oz7K-aG4GdBg@mail.gmail.com>
Message-ID: <51C04841.8080302@python.org>

Am 18.06.2013 13:32, schrieb Victor Stinner:
> 2013/6/18 Christian Heimes <christian at python.org>:
>> Am 18.06.2013 12:56, schrieb Jeremy Kloth:
>>> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner
>>>> +#include <alloca.h>
>>>
>>> This header is not present on Windows, thus breaking all the Windows
>>> buildbots.  Perhaps it should be wrapped in an AIX-specific #ifdef?
> 
> Oh really? Portability is complex :-) I only tested on Linux, but I
> expected this header to be part of the C standard...

It's neither C99 nor POSIX.1-2001. I guess it's just too fragile and not
portable enough. http://c-faq.com/malloc/alloca.html

> 
>> I have added HAVE_ALLOCA_H to configure:
>>
>> http://hg.python.org/cpython/rev/7b6ae19dd116
> 
> Cool, thanks.

You are welcome. alloca() is documented to require <alloca.h>. It merely
works with GCC because the compiler translates the function call to
inline code.

Christian


From victor.stinner at gmail.com  Tue Jun 18 22:40:49 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 18 Jun 2013 22:40:49 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory
	allocators
Message-ID: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>

If you prefer the HTML version:
http://www.python.org/dev/peps/pep-0445/

PEP: 445
Title: Add new APIs to customize Python memory allocators
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-june-2013
Python-Version: 3.4

Abstract
========

Add new APIs to customize Python memory allocators.


Rationale
=========

Use cases:

* Application embedding Python may want to isolate Python memory from
  the memory of the application, or may want to use a different memory
  allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU.
  A custom memory allocator may be required to use efficiently the
  memory and/or to be able to use all the memory of the device.
* Debug tool to:

  - track the memory usage (memory leaks)
  - get the Python filename and line number where an object was
    allocated
  - detect buffer underflow, buffer overflow and detect misuse of Python
    allocator APIs (builtin Python debug hooks)
  - force allocation to fail to test handling of ``MemoryError``
    exception


Proposal
========

API changes
-----------

* Add new GIL-free (no need to hold the GIL) memory allocator functions:

  - ``void* PyMem_RawMalloc(size_t size)``
  - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
  - ``void PyMem_RawFree(void *ptr)``
  - the behaviour of requesting zero bytes is not defined: return *NULL*
    or a distinct non-*NULL* pointer depending on the platform.

* Add a new ``PyMemBlockAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, void *ptr);
    } PyMemBlockAllocator;

* Add new functions to get and set internal functions of
  ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()`` and ``PyMem_RawFree()``:

  - ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
  - ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
  - default allocator: ``malloc()``, ``realloc()``, ``free()``

* Add new functions to get and set internal functions of
  ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``:

  - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
  - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
  - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
    *NULL*: it would be treated as an error.
  - default allocator: ``malloc()``, ``realloc()``, ``free()``;
    ``PyMem_Malloc(0)`` calls ``malloc(1)``
    and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)``

* Add new functions to get and set internal functions of
  ``PyObject_Malloc()``, ``PyObject_Realloc()`` and
  ``PyObject_Free()``:

  - ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
  - ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
  - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
    *NULL*: it would be treated as an error.
  - default allocator: the *pymalloc* allocator

* Add a new ``PyMemMappingAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 2 functions */
        void *ctx;

        /* allocate a memory mapping */
        void* (*alloc) (void *ctx, size_t size);

        /* release a memory mapping */
        void (*free) (void *ctx, void *ptr, size_t size);
    } PyMemMappingAllocator;

* Add a new function to get and set the memory mapping allocator:

  - ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)``
  - ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)``
  - Currently, this allocator is only used internally by *pymalloc* to
    allocate arenas.

* Add a new function to setup the builtin Python debug hooks when memory
  allocators are replaced:

  - ``void PyMem_SetupDebugHooks(void)``
  - the function does nothing is Python is compiled not compiled in
    debug mode

* The following memory allocators always returns *NULL* if size is
  greater than ``PY_SSIZE_T_MAX`` (check before calling the internal
  function): ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``,
  ``PyMem_Malloc()``, ``PyMem_Realloc()``, ``PyObject_Malloc()``,
  ``PyObject_Realloc()``.

The builtin Python debug hooks were introduced in Python 2.3 and
implement the following checks:

* Newly allocated memory is filled with the byte ``0xCB``, freed memory
  is filled with the byte ``0xDB``.
* Detect API violations, ex: ``PyObject_Free()`` called on a memory
  block allocated by ``PyMem_Malloc()``
* Detect write before the start of the buffer (buffer underflow)
* Detect write after the end of the buffer (buffer overflow)


Other changes
-------------

* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()``
  and ``realloc()``, instead of calling ``PyObject_Malloc()`` and
  ``PyObject_Realloc()`` in debug mode

* ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
  ``malloc()`` if size is greater or equal than
  ``SMALL_REQUEST_THRESHOLD`` (512 bytes), and ``PyObject_Realloc()``
  falls back on ``PyMem_Realloc()`` instead of ``realloc()``

* Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
  ``PyMem_RawMalloc()`` if the GIL is not held

* Configure external libraries like zlib or OpenSSL to allocate memory
  using ``PyMem_RawMalloc()``


Examples
========

Use case 1: Replace Memory Allocator, keep pymalloc
----------------------------------------------------

Dummy example wasting 2 bytes per memory block,
and 10 bytes per memory mapping::

    #include <stdlib.h>

    int block_padding = 2;
    int mapping_padding = 10;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void* my_alloc_mapping(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void my_free_mapping(void *ctx, void *ptr, size_t size)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemBlockAllocator block;
        PyMemMappingAllocator mapping;

        block.ctx = &block_padding;
        block.malloc = my_malloc;
        block.realloc = my_realloc;
        block.free = my_free;

        PyMem_SetRawAllocator(&block);
        PyMem_SetAllocator(&block);

        mapping.ctx = &mapping_padding;
        mapping.alloc = my_alloc_mapping;
        mapping.free = my_free_mapping;
        PyMem_SetMappingAllocator(mapping);

        PyMem_SetupDebugHooks();
    }

.. warning::
   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new
   allocator are not thread-safe.


Use case 2: Replace Memory Allocator, override pymalloc
--------------------------------------------------------

If your allocator is optimized for allocation of small objects (less
than 512 bytes) with a short lifetime, pymalloc can be overriden
(replace ``PyObject_Malloc()``).

Dummy example wasting 2 bytes per memory block::

    #include <stdlib.h>

    int padding = 2;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemBlockAllocator alloc;
        alloc.ctx = &padding;
        alloc.malloc = my_malloc;
        alloc.realloc = my_realloc;
        alloc.free = my_free;

        PyMem_SetRawAllocator(&alloc);
        PyMem_SetAllocator(&alloc);
        PyObject_SetAllocator(&alloc);

        PyMem_SetupDebugHooks();
    }

.. warning::
   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new
   allocator are not thread-safe.



Use case 3: Setup Allocator Hooks
---------------------------------

Example to setup hooks on all memory allocators::

    struct {
        PyMemBlockAllocator raw;
        PyMemBlockAllocator mem;
        PyMemBlockAllocator obj;
        /* ... */
    } hook;

    static void* hook_malloc(void *ctx, size_t size)
    {
        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
        /* ... */
        ptr = alloc->malloc(alloc->ctx, size);
        /* ... */
        return ptr;
    }

    static void* hook_realloc(void *ctx, void *ptr, size_t new_size)
    {
        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
        void *ptr2;
        /* ... */
        ptr2 = alloc->realloc(alloc->ctx, ptr, new_size);
        /* ... */
        return ptr2;
    }

    static void hook_free(void *ctx, void *ptr)
    {
        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
        /* ... */
        alloc->free(alloc->ctx, ptr);
        /* ... */
    }

    void setup_hooks(void)
    {
        PyMemBlockAllocator alloc;
        static int installed = 0;

        if (installed)
            return;
        installed = 1;

        alloc.malloc = hook_malloc;
        alloc.realloc = hook_realloc;
        alloc.free = hook_free;

        PyMem_GetRawAllocator(&hook.raw);
        alloc.ctx = &hook.raw;
        PyMem_SetRawAllocator(&alloc);

        PyMem_GetAllocator(&hook.mem);
        alloc.ctx = &hook.mem;
        PyMem_SetAllocator(&alloc);

        PyObject_GetAllocator(&hook.obj);
        alloc.ctx = &hook.obj;
        PyObject_SetAllocator(&alloc);
    }

.. warning::
   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if hooks are not
   thread-safe.

.. note::
   ``PyMem_SetupDebugHooks()`` does not need to be called: Python debug
   hooks are installed automatically at startup.


Performances
============

Results of the `Python benchmarks suite
<http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x
faster, some tests are 1.04 slower, significant is between 115 and -191.

Results of pybench benchmark: "+0.1%" slower globally (diff between
-4.9% and +5.6%).

The full reports are attached to the issue #3329.


Alternatives
============

Only one get/set function for block allocators
----------------------------------------------

Replace the 6 functions:

* ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
* ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
* ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
* ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
* ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
* ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``

with 2 functions with an additional *domain* argument:

* ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator *allocator)``
* ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator *allocator)``

These functions return 0 on success, or -1 if the domain is unknown.

where domain is one of these values:

* ``PYALLOC_PYMEM``
* ``PYALLOC_PYMEM_RAW``
* ``PYALLOC_PYOBJECT``

Drawback: the caller has to check if the result is 0, or handle the error.


Make PyMem_Malloc() reuse PyMem_RawMalloc() by default
------------------------------------------------------

``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So
calling ``PyMem_SetRawAllocator()`` would also also patch
``PyMem_Malloc()`` indirectly.

.. note::

   In the implementation of this PEP (issue #3329),
   ``PyMem_RawMalloc(0)`` calls ``malloc(0)``,
   whereas ``PyMem_Malloc(0)`` calls ``malloc(1)``.


Add a new PYDEBUGMALLOC environment variable
--------------------------------------------

To be able to use the Python builtin debug hooks even when a custom
memory allocator replaces the default Python allocator, an environment
variable ``PYDEBUGMALLOC`` can be added to setup these debug function
hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``.
If the environment variable is present, ``PyMem_SetRawAllocator()``,
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall
automatically the hook on top of the new allocator.

An new environment variable would make the Python initialization even
more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_
tries to simply the CPython startup sequence.


Use macros to get customizable allocators
-----------------------------------------

To have no overhead in the default configuration, customizable
allocators would be an optional feature enabled by a configuration
option or by macros.

Not having to recompile Python makes debug hooks easier to use in
practice.  Extensions modules don't have to be recompiled with macros.


Pass the C filename and line number
-----------------------------------

Define allocator functions as macros using ``__FILE__`` and ``__LINE__``
to get the C filename and line number of a memory allocation.

Example of ``PyMem_Malloc`` macro with the modified
``PyMemBlockAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, const char *filename, int lineno,
                         size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, const char *filename, int lineno,
                          void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, const char *filename, int lineno,
                      void *ptr);
    } PyMemBlockAllocator;

    void* _PyMem_MallocTrace(const char *filename, int lineno,
                             size_t size);

    /* need also a function for the Python stable ABI */
    void* PyMem_Malloc(size_t size);

    #define PyMem_Malloc(size) \
            _PyMem_MallocTrace(__FILE__, __LINE__, size)

Passing a filename and a line number to each allocator makes the API more
complex: pass 3 new arguments, instead of just a context argument, to each
allocator function. The GC allocator functions should also be patched.
For example, ``_PyObject_GC_Malloc()`` is used in many C functions and so
objects of differenet types would have the same allocation location. Such
changes add too much complexity for a little gain.


GIL-free PyMem_Malloc()
-----------------------

When Python is compiled in debug mode, ``PyMem_Malloc()`` calls
indirectly ``PyObject_Malloc()`` which requires the GIL to be held.
That's why ``PyMem_Malloc()`` must be called with the GIL held.

This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call
``malloc()``.  So the "GIL must be held" restriction may be removed from
``PyMem_Malloc()``.

Allowing to call ``PyMem_Malloc()`` without holding the GIL might break
applications which setup their own allocators or allocator hooks.
Holding the GIL is convinient to develop a custom allocator: no need to
care of other threads. It is also convinient for a debug allocator hook:
Python internal objects can be safetly inspected.

Calling ``PyGILState_Ensure()`` in a memory allocator may have
unexpected behaviour, especially at Python startup and at creation of a
new Python thread state.


Don't add PyMem_RawMalloc()
---------------------------

Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is
held.  Otherwise, keep ``malloc()`` unchanged.

The ``PyMem_Malloc()`` is used without the GIL held in some Python
functions.  For example, the ``main()`` and ``Py_Main()`` functions of
Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this
case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or
``PyMem_RawMalloc()``).

If an hook is used to the track memory usage, the ``malloc()`` memory
will not be seen. Remaining ``malloc()`` may allocate a lot of memory
and so would be missed in reports.


Use existing debug tools to analyze the memory
----------------------------------------------

There are many existing debug tools to analyze the memory. Some
examples: `Valgrind <http://valgrind.org/>`_, `Purify
<http://ibm.com/software/awdtools/purify/>`_, `Clang AddressSanitizer
<http://code.google.com/p/address-sanitizer/>`_, `failmalloc
<http://www.nongnu.org/failmalloc/>`_, etc.

The problem is to retrieve the Python object related to a memory pointer
to read its type and/or content. Another issue is to retrieve the
location of the memory allocation: the C backtrace is usually useless
(same reasoning than macros using ``__FILE__`` and ``__LINE__``), the
Python filename and line number (or even the Python traceback) is more
useful.

Classic tools are unable to introspect Python internals to collect such
information. Being able to setup a hook on allocators called with the
GIL held allow to collect a lot of useful data from Python internals.


Add msize()
-----------

Add another field to ``PyMemBlockAllocator`` and
``PyMemMappingAllocator``::

    size_t msize(void *ptr);

This function returns the size of a memory block or a memory mapping.
Return (size_t)-1 if the function is not implemented or if the pointer
is unknown (ex: NULL pointer).

On Windows, this function can be implemented using ``_msize()`` and
``VirtualQuery()``.


No context argument
-------------------

Simplify the signature of allocator functions, remove the context
argument:

* ``void* malloc(size_t size)``
* ``void* realloc(void *ptr, size_t new_size)``
* ``void free(void *ptr)``

It is likely for an allocator hook to be reused for
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
``PyMem_SetRawAllocator()``, but the hook must call a different function
depending on the allocator. The context is a convenient way to reuse the
same custom allocator or hook for different Python allocators.


External libraries
==================

Python should try to reuse the same prototypes for allocator functions
than other libraries.

Libraries used by Python:

* OpenSSL: `CRYPTO_set_mem_functions()
  <http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124>`_
  to set memory management functions globally
* expat: `parserCreate()
  <http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l724>`_
  has a per-instance memory handler

Other libraries:

* glib: `g_mem_set_vtable()
  <http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable>`_
* libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
  global

See also the `GNU libc: Memory Allocation Hooks
<http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_.


Memory allocators
=================

The C standard library provides the well known ``malloc()`` function.
Its implementation depends on the platform and of the C library. The GNU
C library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
(dlmalloc). FreeBSD uses `jemalloc
<http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which
is part of `gperftools <http://code.google.com/p/gperftools/>`_.

``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256
KB), whereas the heap is used for small allocations.

On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on
Linux, and it is contiguous.  On Windows, the heap is handled by
``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by
``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be
discontiguous.

Releasing a memory mapping gives back immediatly the memory to the
system. On UNIX, heap memory is only given back to the system if it is
at the end of the heap. Otherwise, the memory will only be given back to
the system when all the memory located after the released memory are
also released.

To allocate memory in the heap, the allocator tries to reuse free space.
If there is no contiguous space big enough, the heap must be increased,
even if we have more free space than required size.  This issue is
called the "memory fragmentation": the memory usage seen by the system
may be much higher than real usage. On Windows, ``HeapAlloc()`` creates
a new memory mapping with ``VirtualAlloc()`` if there is not enough free
contiguous memory.

CPython has a *pymalloc* allocator for allocations smaller than 512
bytes. This allocator is optimized for small objects with a short
lifetime. It uses memory mappings called "arenas" with a fixed size of
256 KB.

Other allocators:

* Windows provides a `Low-fragmentation Heap
  <http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%29.aspx>`_.

* The Linux kernel uses `slab allocation
  <http://en.wikipedia.org/wiki/Slab_allocation>`_.

* The glib library has a `Memory Slice API
  <https://developer.gnome.org/glib/unstable/glib-Memory-Slices.html>`_:
  efficient way to allocate groups of equal-sized chunks of memory


Links
=====

CPython issues related to memory allocation:

* `Issue #3329: Add new APIs to customize memory allocators
  <http://bugs.python.org/issue3329>`_
* `Issue #13483: Use VirtualAlloc to allocate memory arenas
  <http://bugs.python.org/issue13483>`_
* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
  isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
  PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_
* `Issue #18227: Use Python memory allocators in external libraries like
  zlib or OpenSSL <http://bugs.python.org/issue18227>`_

Projects analyzing the memory usage of Python applications:

* `pytracemalloc
  <https://pypi.python.org/pypi/pytracemalloc>`_
* `Meliae: Python Memory Usage Analyzer
  <https://pypi.python.org/pypi/meliae>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
  <http://guppy-pe.sourceforge.net/>`_
* `PySizer (developed for Python 2.4)
  <http://pysizer.8325.org/>`_

From victor.stinner at gmail.com  Tue Jun 18 23:20:04 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 18 Jun 2013 23:20:04 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
	memory allocators
In-Reply-To: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
Message-ID: <CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>

    typedef struct {
        /* user context passed as the first argument
           to the 2 functions */
        void *ctx;

        /* allocate a memory mapping */
        void* (*alloc) (void *ctx, size_t size);

        /* release a memory mapping */
        void (*free) (void *ctx, void *ptr, size_t size);
    } PyMemMappingAllocator;

The PyMemMappingAllocator structure is very specific to the pymalloc
allocator. There is no "resize", "lock" nor "protect" method. There is
no way to configure protection or flags of the mapping. The
PyMem_SetMappingAllocator() function was initially called
_PyObject_SetArenaAllocator(). I'm not sure that the structure and the
2 related functions should be public. Can an extension module call
private (_Py*) functions or use a private structure?

Or the structure might be renamed to indicate that it is specific to arenas?

What do you think?

Victor

From scott+python-dev at scottdial.com  Wed Jun 19 01:23:55 2013
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Tue, 18 Jun 2013 19:23:55 -0400
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
Message-ID: <51C0EC0B.5040704@scottdial.com>

On 6/18/2013 4:40 PM, Victor Stinner wrote:
> No context argument
> -------------------
> 
> Simplify the signature of allocator functions, remove the context
> argument:
> 
> * ``void* malloc(size_t size)``
> * ``void* realloc(void *ptr, size_t new_size)``
> * ``void free(void *ptr)``
> 
> It is likely for an allocator hook to be reused for
> ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
> ``PyMem_SetRawAllocator()``, but the hook must call a different function
> depending on the allocator. The context is a convenient way to reuse the
> same custom allocator or hook for different Python allocators.

I think there is a lack of justification for the extra argument, and the
extra argument is not free. The typical use-case for doing this
continuation-passing style is when the set of contexts is either
unknown, arbitrarily large, or infinite. In other words, when it would
be either impossible or impractical to enumerate all of the contexts.
However, in this case, we have only 3.

Your proposal already puts forward having 3 pairs of Get/Set functions,
so there is no distinct advantage in having a single typedef instance
that you pass in to all 3 of them. And, having all 3 pairs use the same
typedef is a bit of an attractive nuisance, in that one could pass the
wrong allocators to the wrong setter. With that, I could argue that
there should be 3 typedefs to prevent coding errors.

Nevertheless, the ctx argument buys the implementer nothing if they have
to begin their alloc function with "if(ctx == X)". In other words, there
is nothing simpler about:

"""
void *_alloc(void *ctx, size_t size) {
  if(ctx == PYALLOC_PYMEM)
    return _alloc_pymem(size);
  else if(ctx == PYALLOC_PYMEM_RAW)
    return _alloc_pymem_raw(size);
  else if(ctx == PYALLOC_PYOBJECT)
    return _alloc_pyobject(size);
  else
    abort();
}

PyMemBlockAllocator pymem_allocator =
  {.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pymem_raw_allocator =
  {.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pyobject_allocator =
  {.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free};
"""

In comparison to:

"""
PyMemBlockAllocator pymem_allocator =
  {.alloc=&_alloc_pymem, .free=&_free_pymem};
PyMemBlockAllocator pymem_raw_allocator =
  {.alloc=&_alloc_pymem_raw, .free=&_free_pymem};
PyMemBlockAllocator pyobject_allocator =
  {.alloc=&_alloc_pyobject, .free=&_free_pyobject};
"""

And in the latter case, there is no extra indirect branching in the
hot-path of the allocators.

Also, none of the external libraries cited introduce this CPS/ctx stuff.

-- 
Scott Dial
scott at scottdial.com

From barry at python.org  Wed Jun 19 05:04:03 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 18 Jun 2013 23:04:03 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <CAF-Rda9iagHg+WYmcZqDDKUZ7ka4MjuDiotpasPRed660B_+Vw@mail.gmail.com>
References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au>
	<C0C022D7-BFB1-482F-850F-2F83E05BF3D1@stufft.io>
	<7wd2rmeh5q.fsf@benfinney.id.au> <51BD60DC.40701@stoneleaf.us>
	<20130616111912.1a59b0fe@anarchist>
	<CAF-Rda9iagHg+WYmcZqDDKUZ7ka4MjuDiotpasPRed660B_+Vw@mail.gmail.com>
Message-ID: <20130618230403.4cf9e657@anarchist>

On Jun 16, 2013, at 01:13 PM, Eli Bendersky wrote:

>If you write down the process of porting mailmain from flufl to stdlib.enum
>in some place, it can make the process much easier for others, and even
>encourage them to follow the same path.

Let's write it down here!  It was mostly mechanical, and some probably would
have been done to fix deprecations in flufl.enum.  Here's the list.

Switching from call syntax to getitem syntax for looking up an enum member by
name, e.g.

    -        delivery_mode = DeliveryMode(data['delivery_mode'])
    +        delivery_mode = DeliveryMode[data['delivery_mode']]

Switching from getitem syntax to call syntax for looking up an enum member by
value, e.g.

    -        return self._enum[value]
    +        return self._enum(value)

Interesting that these two were exactly opposite from flufl.enum.

Switching from int() to .value to get the integer value of an enum member,
e.g.

    -    return (member.list_id, member.address.email, int(member.role))
    +    return (member.list_id, member.address.email, member.role.value)

Changing the imports (obviously), e.g.

    -from flufl.enum import Enum
    +from enum import Enum

Adapting to the different repr as seen in a doctest, e.g.

    -    <EnumValue: Response.hold [int=1]>
    +    <Response.hold: 1>

Fixing some type tests.  I have one case that was testing for an enum member
to do value type conversion.  Since enum members are now instances of their
enum class, and because members no longer have a .enum attribute (because you
can just ask for its __class__), the type test actually became simpler:

    -        elif hasattr(obj, 'enum') and issubclass(obj.enum, Enum):
    +        elif isinstance(obj, Enum):

An unexpected difference is that failing name lookups raise a KeyError instead
of a ValueError.  There are many cases where I catch failing lookups in my
REST API.  Fixing this was mostly easy...

    -    except ValueError:
    +    except KeyError:

...except in one case where I was previously allowing the ValueError to
percolate up to signal to a higher level that some exception handling had to
be done.  Here, I had to convert the exception:

    -        # This will raise a ValueError if the enum value is unknown.  Let
    -        # that percolate up.
    -        return self._enum_class[enum_value]
    +        # This will raise a KeyError if the enum value is unknown.  The
    +        # Validator API requires turning this into a ValueError.
    +        try:
    +            return self._enum_class[enum_value]
    +        except KeyError as exception:
    +            # Retain the error message.
    +            raise ValueError(exception.message)

Interestingly, I found an existing hidden bug in an upgrade script where I was
not converting a value from an integer to an enum.  Nice to catch that one.

For now, I'll hold out on merging this branch to trunk until I can get enum34
packaged up for Debian.  Other than that, it looks pretty good!

Merge proposal: http://tinyurl.com/l2fq38l
Branch: https://code.launchpad.net/~barry/mailman/pep435

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130618/bec98f6f/attachment-0001.pgp>

From ncoghlan at gmail.com  Wed Jun 19 05:32:01 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 Jun 2013 13:32:01 +1000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <51C0EC0B.5040704@scottdial.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<51C0EC0B.5040704@scottdial.com>
Message-ID: <CADiSq7e76dUOKQfB2SgMy1rw8qSneLYx-FJPsyKDPVaxpwmGqQ@mail.gmail.com>

On 19 June 2013 09:23, Scott Dial <scott+python-dev at scottdial.com> wrote:
> On 6/18/2013 4:40 PM, Victor Stinner wrote:
>> No context argument
>> -------------------
>>
>> Simplify the signature of allocator functions, remove the context
>> argument:
>>
>> * ``void* malloc(size_t size)``
>> * ``void* realloc(void *ptr, size_t new_size)``
>> * ``void free(void *ptr)``
>>
>> It is likely for an allocator hook to be reused for
>> ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
>> ``PyMem_SetRawAllocator()``, but the hook must call a different function
>> depending on the allocator. The context is a convenient way to reuse the
>> same custom allocator or hook for different Python allocators.
>
> I think there is a lack of justification for the extra argument, and the
> extra argument is not free. The typical use-case for doing this
> continuation-passing style is when the set of contexts is either
> unknown, arbitrarily large, or infinite. In other words, when it would
> be either impossible or impractical to enumerate all of the contexts.
> However, in this case, we have only 3.

Note that the context is part of the BlockAllocator structure, NOT
predefined by Python.

> Your proposal already puts forward having 3 pairs of Get/Set functions,
> so there is no distinct advantage in having a single typedef instance
> that you pass in to all 3 of them. And, having all 3 pairs use the same
> typedef is a bit of an attractive nuisance, in that one could pass the
> wrong allocators to the wrong setter. With that, I could argue that
> there should be 3 typedefs to prevent coding errors.

I'm not sure we *should* be restricting this to the CPython internal
domains indefinitely. If we use a domain based model from the start,
then that will allow us in the future to let extension modules declare
additional domains rather than having to employ library specific logic
in either the CPython core or in embedding applications.

> Nevertheless, the ctx argument buys the implementer nothing if they have
> to begin their alloc function with "if(ctx == X)". In other words, there
> is nothing simpler about:
>
> """
> void *_alloc(void *ctx, size_t size) {
>   if(ctx == PYALLOC_PYMEM)
>     return _alloc_pymem(size);
>   else if(ctx == PYALLOC_PYMEM_RAW)
>     return _alloc_pymem_raw(size);
>   else if(ctx == PYALLOC_PYOBJECT)
>     return _alloc_pyobject(size);
>   else
>     abort();
> }
>
> PyMemBlockAllocator pymem_allocator =
>   {.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free};
> PyMemBlockAllocator pymem_raw_allocator =
>   {.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free};
> PyMemBlockAllocator pyobject_allocator =
>   {.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free};
> """

Why would anyone do that? The context is so embedding applications can
distinguish the CPython runtime from their *other* domains that use
the same allocator functions. If you wanted to use completely
different allocators for each domain, you would just do that and
ignore the context argument entirely.

Agreed more of that rationale needs to be moved from the issue tracker
into the PEP, though.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Wed Jun 19 05:37:27 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 Jun 2013 13:37:27 +1000
Subject: [Python-Dev] backported Enum
In-Reply-To: <51BCC498.1030100@stoneleaf.us>
References: <51BCC498.1030100@stoneleaf.us>
Message-ID: <CADiSq7dR--Z3GRk1LxV7mo0W8E6G23rF5f_CawK5p=3eEEnFTw@mail.gmail.com>

On 16 June 2013 05:46, Ethan Furman <ethan at stoneleaf.us> wrote:
> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4
> in the 2.x series.

We should resolve http://bugs.python.org/issue17961 (switching the
functional creation API to use strings instead of 1-based integers)
before we get too gung ho about encouraging people to use the
backport.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From scott+python-dev at scottdial.com  Wed Jun 19 06:33:54 2013
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Wed, 19 Jun 2013 00:33:54 -0400
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CADiSq7e76dUOKQfB2SgMy1rw8qSneLYx-FJPsyKDPVaxpwmGqQ@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<51C0EC0B.5040704@scottdial.com>
	<CADiSq7e76dUOKQfB2SgMy1rw8qSneLYx-FJPsyKDPVaxpwmGqQ@mail.gmail.com>
Message-ID: <51C134B2.8040408@scottdial.com>

On 6/18/2013 11:32 PM, Nick Coghlan wrote:
> Agreed more of that rationale needs to be moved from the issue tracker
> into the PEP, though.

Thanks for the clarification. I hadn't read the issue tracker at all. On
it's face value, I didn't see what purpose it served, but having read
Kristj?n's comments on the issue tracker, he would like to store state
for the allocators in that ctx pointer.[1] Having read that (previously,
I thought the only utility was distinguishing which domain it was -- a
small, glorified enumeration), but his use-case makes sense and
definitely is informative to have in the PEP, because the utility of
that wasn't obvious to me.

Thanks,
-Scott

[1] http://bugs.python.org/issue3329#msg190529
"""
One particular trick we have been using, which might be of interest, is
to be able to tag each allocation with a "context" id.  This is then set
according to a global sys.memcontext variable, which the program will
modify according to what it is doing.  This can then be used to track
memory usage by different parts of the program.
"""

-- 
Scott Dial
scott at scottdial.com

From victor.stinner at gmail.com  Wed Jun 19 07:42:08 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 07:42:08 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
	memory allocators
In-Reply-To: <51C0EC0B.5040704@scottdial.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<51C0EC0B.5040704@scottdial.com>
Message-ID: <CAMpsgwaWzwx_QK4J1fwOLa=p60ADUy_d5KADUZPDfgs1vq=xjA@mail.gmail.com>

Le mercredi 19 juin 2013, Scott Dial a ?crit :

> On 6/18/2013 4:40 PM, Victor Stinner wrote:
> > No context argument
>
> I think there is a lack of justification for the extra argument, and the
> extra argument is not free. The typical use-case for doing this
> continuation-passing style is when the set of contexts is either
> unknown, arbitrarily large, or infinite. In other words, when it would
> be either impossible or impractical to enumerate all of the contexts.
> However, in this case, we have only 3.


See the use case 3 in examples. Without the context argument, you have to
copy/paste 3 times each functions: 3 functions -> 9 functions. I don't like
having to copy/paste code, it sounds like a bad design.


> And in the latter case, there is no extra indirect branching in the
> hot-path of the allocators.


Are you concerned by performances? Did you see the Performances section,
there is no overhead according to the benchmark suite.


>
> Also, none of the external libraries cited introduce this CPS/ctx stuff.


Oops, the list is incomplete. Copy/paste from the issue:


Some customizable memory allocators I know have an extra parameter
"void *opaque" that is passed to all functions:

- in zlib: zalloc and zfree: http://www.zlib.net/manual.html#Usage
- same thing for bz2.
- lzma's ISzAlloc: http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html
- Oracle's OCI:
http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm

OTOH, expat, libxml, libmpdec don't have this extra parameter.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130619/a207c406/attachment.html>

From trent at snakebite.org  Wed Jun 19 15:10:02 2013
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 19 Jun 2013 21:10:02 +0800
Subject: [Python-Dev] pyparallel and new memory API discussions...
Message-ID: <20130619131001.GB10039@snakebite.org>


    The new memory API discussions (and PEP) warrant a quick pyparallel
    update: a couple of weeks after PyCon, I came up with a solution for
    the biggest show-stopper that has been plaguing pyparallel since its
    inception: being able to detect the modification of "main thread"
    Python objects from within a parallel context.

    For example, `data.append(4)` in the example below will generate an
    AssignmentError exception, because data is a main thread object, and
    `data.append(4)` gets executed from within a parallel context::

        data = [ 1, 2, 3 ]

        def work():
            data.append(4)

        async.submit_work(work)

    The solution turned out to be deceptively simple:

      1.  Prior to running parallel threads, lock all "main thread"
          memory pages as read-only (via VirtualProtect on Windows,
          mprotect on POSIX).

      2.  Detect attempts to write to main thread pages during parallel
          thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
          and raise an exception instead (detection is done in the ceval
          frame exec loop).

      3.  Prior to returning control back to the main thread (which will
          be paused whilst all the parallel threads are running), unlock
          all the "main thread" pages.

      4.  Pause all parallel threads while the main thread runs.

      5.  Go back to 1.

    I got a proof-of-concept working on Windows a while back (and also
    played around with large page support in the same commit).  The main
    changes were to obmalloc.c:

        https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe808c1c11779f#chg-Objects/obmalloc.c

    The key was the introduction of two new API calls, intended to be
    called by the pyparallel.c infrastructure:

        _PyMem_LockMainThreadPages()
        _PyMem_UnlockMainThreadPages()

    The implementation is pretty simple:

+int
+_PyMem_LockMainThreadPages(void)
+{
+    DWORD old = 0;
+
+    if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, &old)) {
+        PyErr_SetFromWindowsErr(0);
+        return -1;
+    }

    Note the `base_addr` and `nbytes_committed` argument.  Basically, I
    re-organized obmalloc.c a little bit such that we never actually
    call malloc() directly.  Instead, we exploit the ability to reserve
    huge virtual address ranges without actually committing the memory,
    giving us a fixed `base_addr` void pointer that we can pass to calls
    like VirtualProtect or mprotect.

    We then incrementally commit more pages as demand increases, and
    simply adjust our `nbytes_committed` counter as we go along.  The
    net effect is that we can call VirtualProtect/mprotect once, with a
    single base void pointer and size_t range, and immediately affect the
    protection of all memory pages that fall within that range.

    As an added bonus, we also get a very cheap and elegant way to test
    if a pointer (or any arbitrary memory address, actually) belongs to
    the main thread's memory range (at least in comparison to the
    existing _PyMem_InRange black magic).  (This is very useful for my
    pyparallel infrastructure, which makes extensive use of conditional
    logic based on address tests.)

        (Side-bar: a side-effect of the approach I've used in the proof-
         of-concept (by only having a single base addr pointer) is that
         we effectively limit the maximum memory we could eventually
         commit.

         I actually quite like this -- in fact, I'd like to tweak it
         such that we can actually expose min/max memory values to the
         Python interpreter at startup (analogous to the JVM).

         Having known upper bounds on maximum memory usage will vastly
         simplify some other areas of my pyparallel work (like the async
         socket stuff).

         For example, consider network programs these days that take a
         "max clients" configuration parameter.  That seems a bit
         backwards to me.

         It would be better if we simply said, "here, Python, you have
         1GB to work with".  That allows us to calculate how many
         clients we could simultaneously serve based on socket memory
         requirements, which allows for much more graceful behavior
         under load than leaving it open-ended.

         Maximum memory constraints would also be useful for the
         parallel.map(callable, iterable) stuff I've got in the works,
         as it'll allow us to optimally chunk work and assign to threads
         based on available memory.)

    So, Victor, I'm interested to hear how the new API you're proposing
    will affect this solution I've come up with for pyparallel; I'm
    going to be absolutely dependent upon the ability to lock main
    thread pages as read-only in one fell-swoop -- am I still going to
    be able to do that with your new API in place?

    Regards,

        Trent.

From ncoghlan at gmail.com  Wed Jun 19 15:33:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 Jun 2013 23:33:02 +1000
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <20130619131001.GB10039@snakebite.org>
References: <20130619131001.GB10039@snakebite.org>
Message-ID: <CADiSq7eLoGcoaMBO2mMZTRLCXOiWCGAbnELLJ0gmmhNqo5490g@mail.gmail.com>

On 19 June 2013 23:10, Trent Nelson <trent at snakebite.org> wrote:
>     So, Victor, I'm interested to hear how the new API you're proposing
>     will affect this solution I've come up with for pyparallel; I'm
>     going to be absolutely dependent upon the ability to lock main
>     thread pages as read-only in one fell-swoop -- am I still going to
>     be able to do that with your new API in place?

By default, nothing will change for the ordinary CPython runtime. It's
only if an embedding application starts messing with the allocators
that things might change, but at that point, pyparallel would break
anyway.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From cf.natali at gmail.com  Wed Jun 19 16:01:49 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Wed, 19 Jun 2013 16:01:49 +0200
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <20130619131001.GB10039@snakebite.org>
References: <20130619131001.GB10039@snakebite.org>
Message-ID: <CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>

2013/6/19 Trent Nelson <trent at snakebite.org>:
>
>     The new memory API discussions (and PEP) warrant a quick pyparallel
>     update: a couple of weeks after PyCon, I came up with a solution for
>     the biggest show-stopper that has been plaguing pyparallel since its
>     inception: being able to detect the modification of "main thread"
>     Python objects from within a parallel context.
>
>     For example, `data.append(4)` in the example below will generate an
>     AssignmentError exception, because data is a main thread object, and
>     `data.append(4)` gets executed from within a parallel context::
>
>         data = [ 1, 2, 3 ]
>
>         def work():
>             data.append(4)
>
>         async.submit_work(work)
>
>     The solution turned out to be deceptively simple:
>
>       1.  Prior to running parallel threads, lock all "main thread"
>           memory pages as read-only (via VirtualProtect on Windows,
>           mprotect on POSIX).
>
>       2.  Detect attempts to write to main thread pages during parallel
>           thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
>           and raise an exception instead (detection is done in the ceval
>           frame exec loop).

Quick stupid question: because of refcounts, the pages will be written
to even in case of read-only access. How do you deal with this?

cf

From solipsis at pitrou.net  Wed Jun 19 16:15:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 19 Jun 2013 16:15:58 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
Message-ID: <20130619161558.1f042553@pitrou.net>

Le Tue, 18 Jun 2013 22:40:49 +0200,
Victor Stinner <victor.stinner at gmail.com> a ?crit :
> 
> Other changes
> -------------
> 
[...]
> 
> * Configure external libraries like zlib or OpenSSL to allocate memory
>   using ``PyMem_RawMalloc()``

Why so, and is it done by default?

> Only one get/set function for block allocators
> ----------------------------------------------
> 
> Replace the 6 functions:
> 
> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
> 
> with 2 functions with an additional *domain* argument:
> 
> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator
> *allocator)``
> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator
> *allocator)``

I would much prefer this solution.

> Drawback: the caller has to check if the result is 0, or handle the
> error.

Or you can just call Py_FatalError() if the domain is invalid.

> If an hook is used to the track memory usage, the ``malloc()`` memory
> will not be seen. Remaining ``malloc()`` may allocate a lot of memory
> and so would be missed in reports.

A lot of memory? In main()?

Regards

Antoine.



From trent at snakebite.org  Wed Jun 19 16:33:10 2013
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 19 Jun 2013 22:33:10 +0800
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
References: <20130619131001.GB10039@snakebite.org>
	<CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
Message-ID: <20130619143309.GA10788@snakebite.org>

Hi Charles-Fran?ois!

    Good to hear from you again.  It was actually your e-mail a few
    months ago that acted as the initial catalyst for this memory
    protection idea, so, thanks for that :-)

    Answer below.

On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-Fran?ois Natali wrote:
> 2013/6/19 Trent Nelson <trent at snakebite.org>:
> >
> >     The new memory API discussions (and PEP) warrant a quick pyparallel
> >     update: a couple of weeks after PyCon, I came up with a solution for
> >     the biggest show-stopper that has been plaguing pyparallel since its
> >     inception: being able to detect the modification of "main thread"
> >     Python objects from within a parallel context.
> >
> >     For example, `data.append(4)` in the example below will generate an
> >     AssignmentError exception, because data is a main thread object, and
> >     `data.append(4)` gets executed from within a parallel context::
> >
> >         data = [ 1, 2, 3 ]
> >
> >         def work():
> >             data.append(4)
> >
> >         async.submit_work(work)
> >
> >     The solution turned out to be deceptively simple:
> >
> >       1.  Prior to running parallel threads, lock all "main thread"
> >           memory pages as read-only (via VirtualProtect on Windows,
> >           mprotect on POSIX).
> >
> >       2.  Detect attempts to write to main thread pages during parallel
> >           thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
> >           and raise an exception instead (detection is done in the ceval
> >           frame exec loop).
> 
> Quick stupid question: because of refcounts, the pages will be written
> to even in case of read-only access. How do you deal with this?

    Easy: I don't refcount in parallel contexts :-)

    There's no need, for two reasons:

     1. All memory allocated in a parallel context is localized to a
        private heap.  When the parallel context is finished, the entire
        heap can be blown away in one fell-swoop.  There's no need for
        reference counting or GC because none of the objects will exist
        after the parallel context completes.

     2. The main thread won't be running when parallel threads/contexts
        are executing, which means main thread objects being accessed in
        parallel contexts (read-only access is fine) won't be suddenly
        free()'d or GC-collected or whatever.

    You get credit for that second point; you asked a similar question a
    few months ago that made me realize I absolutely couldn't have the
    main thread running at the same time the parallel threads were
    running.

    Once I accepted that as a design constraint, everything else came
    together nicely... "Hmmm, if the main thread isn't running, it won't
    need write-access to any of its pages!  If we mark them read-only,
    we could catch the traps/SEHs from parallel threads, then raise an
    exception, ahh, simple!".

    I'm both chuffed at how simple it is (considering it was *the* major
    show-stopper), and miffed at how it managed to elude me for so long
    ;-)

    Regards,

        Trent.

From victor.stinner at gmail.com  Wed Jun 19 17:24:21 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 17:24:21 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <20130619161558.1f042553@pitrou.net>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
Message-ID: <CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>

2013/6/19 Antoine Pitrou <solipsis at pitrou.net>:
> Le Tue, 18 Jun 2013 22:40:49 +0200,
> Victor Stinner <victor.stinner at gmail.com> a ?crit :
>>
>> Other changes
>> -------------
>>
> [...]
>>
>> * Configure external libraries like zlib or OpenSSL to allocate memory
>>   using ``PyMem_RawMalloc()``
>
> Why so, and is it done by default?

(Oh, I realized that PyMem_Malloc() may be used instead of
PyMem_RawMalloc() if we are sure that the library will only be used
when the GIL is held.)

"is it done by default?"

First, it would be safer to only reuse PyMem_RawMalloc() allocator if
PyMem_SetRawMalloc() was called. Just to avoid regressions in Python
3.4.

Then, it depends on the library: if the allocator can be replaced for
one library object (ex: expat supports this), it can always be
replaced. Otherwise, we should only replace the library allocator if
Python is a standalone program (don't replace the library allocator if
Python is embedded). That's why I asked if it is possible to check if
Python is embedded or not.

"Why so,"

For the "track memory usage" use case, it is important to track memory
allocated in external libraries to have accurate reports, because
these allocations may be huge.

>> Only one get/set function for block allocators
>> ----------------------------------------------
>>
>> Replace the 6 functions:
>>
>> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
>> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
>> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
>> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
>> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
>> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
>>
>> with 2 functions with an additional *domain* argument:
>>
>> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator
>> *allocator)``
>> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator
>> *allocator)``
>
> I would much prefer this solution.

I don't have a strong preference between these two choices.

Oh, one argument in favor of one generic function is that code using
these functions would be simpler. Extract of the unit test of the
implementation (_testcapi.c):

+    if (api == 'o')
+        PyObject_SetAllocator(&hook.alloc);
+    else if (api == 'r')
+        PyMem_SetRawAllocator(&hook.alloc);
+    else
+        PyMem_SetAllocator(&hook.alloc);

With a generic function, this block can be replace with one unique
function call.

>> Drawback: the caller has to check if the result is 0, or handle the
>> error.
>
> Or you can just call Py_FatalError() if the domain is invalid.

I don't like Py_FatalError(), especially when Python is embedded. It's
safer to return -1 and expect the caller to check for the error case.

>> If an hook is used to the track memory usage, the ``malloc()`` memory
>> will not be seen. Remaining ``malloc()`` may allocate a lot of memory
>> and so would be missed in reports.
>
> A lot of memory? In main()?

Not in main(). The Python expat and zlib modules call directly
malloc() and may allocate large blocks. External libraries like
OpenSSL or bz2 may also allocate large blocks.

See issues #18203 and #18227.

Victor

From kristjan at ccpgames.com  Wed Jun 19 17:31:35 2013
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Wed, 19 Jun 2013 15:31:35 +0000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <51C134B2.8040408@scottdial.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<51C0EC0B.5040704@scottdial.com>
	<CADiSq7e76dUOKQfB2SgMy1rw8qSneLYx-FJPsyKDPVaxpwmGqQ@mail.gmail.com>
	<51C134B2.8040408@scottdial.com>
Message-ID: <EFE3877620384242A686D52278B7CCD35395179D@RKV-IT-EXCH104.ccp.ad.local>

Right, think of the "ctxt" as a "this" pointer from c++.
If you have an allocator object, that you got from some c++ api, and want to ask Python to use that, you need to be able to thunk the "this" pointer to get at the particular allocator instance.
It used to be a common mistake when writing C callback apis to forget to add an opaque "context" pointer along with the callback function.
This omission makes it difficult (but not impossible) to attach c++ methods to such callbacks.

K

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-
> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Scott Dial
> Sent: 19. j?n? 2013 04:34
> To: ncoghlan at gmail.com
> Cc: Python-Dev at python.org
> Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
> memory allocators
> 
> On 6/18/2013 11:32 PM, Nick Coghlan wrote:
> > Agreed more of that rationale needs to be moved from the issue tracker
> > into the PEP, though.
> 
> Thanks for the clarification. I hadn't read the issue tracker at all. On it's face
> value, I didn't see what purpose it served, but having read Kristj?n's
> comments on the issue tracker, he would like to store state for the allocators
> in that ctx pointer.[1] Having read that (previously, I thought the only utility
> was distinguishing which domain it was -- a small, glorified enumeration), but
> his use-case makes sense and definitely is informative to have in the PEP,
> because the utility of that wasn't obvious to me.
> 
> Thanks,
> -Scott
> 
> [1] http://bugs.python.org/issue3329#msg190529
> """
> One particular trick we have been using, which might be of interest, is to be
> able to tag each allocation with a "context" id.  This is then set according to a
> global sys.memcontext variable, which the program will modify according to
> what it is doing.  This can then be used to track memory usage by different
> parts of the program.
> """
> 
> --
> Scott Dial
> scott at scottdial.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com



From solipsis at pitrou.net  Wed Jun 19 17:43:54 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 19 Jun 2013 17:43:54 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
	<CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
Message-ID: <20130619174354.34a7d8f0@fsol>

On Wed, 19 Jun 2013 17:24:21 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> For the "track memory usage" use case, it is important to track memory
> allocated in external libraries to have accurate reports, because
> these allocations may be huge.
[...]
> Not in main(). The Python expat and zlib modules call directly
> malloc() and may allocate large blocks. External libraries like
> OpenSSL or bz2 may also allocate large blocks.

Fair enough.

> >> Drawback: the caller has to check if the result is 0, or handle the
> >> error.
> >
> > Or you can just call Py_FatalError() if the domain is invalid.
> 
> I don't like Py_FatalError(), especially when Python is embedded. It's
> safer to return -1 and expect the caller to check for the error case.

I don't think you need to check for errors. The domain is always one of
the existing constants, i.e. it should be hard-coded in the source, not
computed.

Regards

Antoine.

From victor.stinner at gmail.com  Wed Jun 19 17:45:55 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 17:45:55 +0200
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <20130619143309.GA10788@snakebite.org>
References: <20130619131001.GB10039@snakebite.org>
	<CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
	<20130619143309.GA10788@snakebite.org>
Message-ID: <CAMpsgwZ+gH4pJQGYAa8_antmUyfOFEtuox7ZxwBB16+-pOAEAw@mail.gmail.com>

>      1. All memory allocated in a parallel context is localized to a
>         private heap.

How do you allocate memory in this "private" heap? Did you add new
functions to allocate memory?

Victor

From victor.stinner at gmail.com  Wed Jun 19 17:49:02 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 17:49:02 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <20130619174354.34a7d8f0@fsol>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
	<CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
	<20130619174354.34a7d8f0@fsol>
Message-ID: <CAMpsgwZaFivV8uXfEYFLdNyzodRmKYHeuitdVGs_dqcpOwYkVQ@mail.gmail.com>

2013/6/19 Antoine Pitrou <solipsis at pitrou.net>:
> On Wed, 19 Jun 2013 17:24:21 +0200
>> >> Drawback: the caller has to check if the result is 0, or handle the
>> >> error.
>> >
>> > Or you can just call Py_FatalError() if the domain is invalid.
>>
>> I don't like Py_FatalError(), especially when Python is embedded. It's
>> safer to return -1 and expect the caller to check for the error case.
>
> I don't think you need to check for errors. The domain is always one of
> the existing constants, i.e. it should be hard-coded in the source, not
> computed.

Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and
that a new domain is added to Python 3.5. An application is written
for Python 3.5 and is run with Python 3.4: how would the application
notice that PyMem_GetBlockAllocator() does not know the new domain?

"I don't think you need to check for errors."

Do you mean that an unknown domain should be simply ignored?

Victor

From kristjan at ccpgames.com  Wed Jun 19 17:28:22 2013
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Wed, 19 Jun 2013 15:28:22 +0000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize
	Python	memory allocators
In-Reply-To: <CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
Message-ID: <EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>

Oh, it should be public, in my opinion.
We do exactly that when we embed python into UnrealEngine.  We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas.  That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs.

K

> -----Original Message-----
> From: Python-Dev [mailto:python-dev-
> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Victor Stinner
> Sent: 18. j?n? 2013 21:20
> To: Python Dev
> Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
> memory allocators
> 
>     typedef struct {
>         /* user context passed as the first argument
>            to the 2 functions */
>         void *ctx;
> 
>         /* allocate a memory mapping */
>         void* (*alloc) (void *ctx, size_t size);
> 
>         /* release a memory mapping */
>         void (*free) (void *ctx, void *ptr, size_t size);
>     } PyMemMappingAllocator;
> 
> The PyMemMappingAllocator structure is very specific to the pymalloc
> allocator. There is no "resize", "lock" nor "protect" method. There is no way
> to configure protection or flags of the mapping. The
> PyMem_SetMappingAllocator() function was initially called
> _PyObject_SetArenaAllocator(). I'm not sure that the structure and the
> 2 related functions should be public. Can an extension module call private
> (_Py*) functions or use a private structure?
> 
> Or the structure might be renamed to indicate that it is specific to arenas?
> 
> What do you think?
> 
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com



From victor.stinner at gmail.com  Wed Jun 19 17:58:36 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 17:58:36 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
Message-ID: <CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>

2013/6/19 Kristj?n Valur J?nsson <kristjan at ccpgames.com>:
> Oh, it should be public, in my opinion.

Ok. And do you think that the PyMemMappingAllocator structure is
complete, or that we should add something to be future-proof? At
least, PyMemMappingAllocator is enough for pymalloc usage :-)

Is PyMemMappingAllocator complete enough for your usage at CCP Games?

> We do exactly that when we embed python into UnrealEngine.  We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas.  That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs.

I hope that the PEP 445 is flexible enough to allow you to decide
which functions are hooked and replaced, and which functions will be
leaved unchanged. That's why I'm not in favor of the "Make
PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.

Victor

From trent at snakebite.org  Wed Jun 19 18:00:01 2013
From: trent at snakebite.org (Trent Nelson)
Date: Thu, 20 Jun 2013 00:00:01 +0800
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <CAMpsgwZ+gH4pJQGYAa8_antmUyfOFEtuox7ZxwBB16+-pOAEAw@mail.gmail.com>
References: <20130619131001.GB10039@snakebite.org>
	<CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
	<20130619143309.GA10788@snakebite.org>
	<CAMpsgwZ+gH4pJQGYAa8_antmUyfOFEtuox7ZxwBB16+-pOAEAw@mail.gmail.com>
Message-ID: <20130619160000.GA11044@snakebite.org>

On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> >      1. All memory allocated in a parallel context is localized to a
> >         private heap.
> 
> How do you allocate memory in this "private" heap? Did you add new
> functions to allocate memory?

    Yup:
        _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.

    All memory operations (PyObject_New/Malloc etc) get intercepted
    during parallel thread execution and redirected to _PyHeap_Malloc(),
    which is a very simple slab allocator.  (No need for convoluted
    buckets because we never free individual objects during parallel
    execution; instead, we just blow everything away at the end.)

        Trent.

From tjreedy at udel.edu  Wed Jun 19 18:13:06 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 19 Jun 2013 12:13:06 -0400
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
	memory allocators
In-Reply-To: <CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
	<CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
Message-ID: <kpslah$4bd$1@ger.gmane.org>

On 6/19/2013 11:24 AM, Victor Stinner wrote:
> 2013/6/19 Antoine Pitrou <solipsis at pitrou.net>:
>> Le Tue, 18 Jun 2013 22:40:49 +0200,
>> Victor Stinner <victor.stinner at gmail.com> a ?crit :

>>> Only one get/set function for block allocators
>>> ----------------------------------------------
>>>
>>> Replace the 6 functions:
>>>
>>> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
>>> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
>>> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
>>> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
>>> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
>>> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
>>>
>>> with 2 functions with an additional *domain* argument:
>>>
>>> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator
>>> *allocator)``
>>> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator
>>> *allocator)``
>>
>> I would much prefer this solution.

I do to. The two names can be remembered as one pair with only get/set 
difference.

-- 
Terry Jan Reedy



From victor.stinner at gmail.com  Wed Jun 19 18:20:15 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 19 Jun 2013 18:20:15 +0200
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <20130619160000.GA11044@snakebite.org>
References: <20130619131001.GB10039@snakebite.org>
	<CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
	<20130619143309.GA10788@snakebite.org>
	<CAMpsgwZ+gH4pJQGYAa8_antmUyfOFEtuox7ZxwBB16+-pOAEAw@mail.gmail.com>
	<20130619160000.GA11044@snakebite.org>
Message-ID: <CAMpsgwZVD2kwDhSf-ic85k=MC9UEbfAkJ_AzCMYR2q4P70cHaA@mail.gmail.com>

"""
    So, Victor, I'm interested to hear how the new API you're proposing
    will affect this solution I've come up with for pyparallel; I'm
    going to be absolutely dependent upon the ability to lock main
    thread pages as read-only in one fell-swoop -- am I still going to
    be able to do that with your new API in place?
"""

2013/6/19 Trent Nelson <trent at snakebite.org>:
> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
>> >      1. All memory allocated in a parallel context is localized to a
>> >         private heap.
>>
>> How do you allocate memory in this "private" heap? Did you add new
>> functions to allocate memory?
>
>     Yup:
>         _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
>
>     All memory operations (PyObject_New/Malloc etc) get intercepted
>     during parallel thread execution and redirected to _PyHeap_Malloc(),
>     which is a very simple slab allocator.  (No need for convoluted
>     buckets because we never free individual objects during parallel
>     execution; instead, we just blow everything away at the end.)

Ok, so I don't think that the PEP 445 would change anything for you.

The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.

If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-------------------------- 8< -----------------
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
{
    PyMemBlockAllocator *ctx;
    if (Py_PXCTX)
        return _PxMem_Malloc(size))
    else
        return alloc->malloc(alloc->ctx, size);
}

...

PyMemBlockAllocator pyparallel_pyobject;

static void *
setup_pyparallel_allocator(void)
{
    PyMemBlockAllocator alloc;
    PyObject_GetAllocator(&pyparallel_pyobject);
    alloc.ctx = &pyparallel_pyobject;
    alloc.malloc = _PxMem_AllocMalloc;
    ...
    PyObject_SetAllocator(&alloc);
}
-------------------------- 8< -----------------

But I don't know if you want pyparallel to be an "optional" feature
chosen at runtime...

Victor

From solipsis at pitrou.net  Wed Jun 19 18:22:29 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 19 Jun 2013 18:22:29 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwZaFivV8uXfEYFLdNyzodRmKYHeuitdVGs_dqcpOwYkVQ@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
	<CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
	<20130619174354.34a7d8f0@fsol>
	<CAMpsgwZaFivV8uXfEYFLdNyzodRmKYHeuitdVGs_dqcpOwYkVQ@mail.gmail.com>
Message-ID: <20130619182229.2fa7cd92@fsol>

On Wed, 19 Jun 2013 17:49:02 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:

> 2013/6/19 Antoine Pitrou <solipsis at pitrou.net>:
> > On Wed, 19 Jun 2013 17:24:21 +0200
> >> >> Drawback: the caller has to check if the result is 0, or handle the
> >> >> error.
> >> >
> >> > Or you can just call Py_FatalError() if the domain is invalid.
> >>
> >> I don't like Py_FatalError(), especially when Python is embedded. It's
> >> safer to return -1 and expect the caller to check for the error case.
> >
> > I don't think you need to check for errors. The domain is always one of
> > the existing constants, i.e. it should be hard-coded in the source, not
> > computed.
> 
> Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and
> that a new domain is added to Python 3.5. An application is written
> for Python 3.5 and is run with Python 3.4: how would the application
> notice that PyMem_GetBlockAllocator() does not know the new domain?

That's a good question. I don't know why guidelines Martin used when
designing the stable ABI, but I would expect important high-level
functions to end there, not memory allocation debugging.

Regards

Antoine.

From trent at snakebite.org  Wed Jun 19 18:42:06 2013
From: trent at snakebite.org (Trent Nelson)
Date: Thu, 20 Jun 2013 00:42:06 +0800
Subject: [Python-Dev] pyparallel and new memory API discussions...
In-Reply-To: <CAMpsgwZVD2kwDhSf-ic85k=MC9UEbfAkJ_AzCMYR2q4P70cHaA@mail.gmail.com>
References: <20130619131001.GB10039@snakebite.org>
	<CAH_1eM1JaMvkEgr8vuFKYtMX-=rZCirfL8a_NsH6CFZDPSLfWQ@mail.gmail.com>
	<20130619143309.GA10788@snakebite.org>
	<CAMpsgwZ+gH4pJQGYAa8_antmUyfOFEtuox7ZxwBB16+-pOAEAw@mail.gmail.com>
	<20130619160000.GA11044@snakebite.org>
	<CAMpsgwZVD2kwDhSf-ic85k=MC9UEbfAkJ_AzCMYR2q4P70cHaA@mail.gmail.com>
Message-ID: <20130619164205.GA11338@snakebite.org>

On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
> """
>     So, Victor, I'm interested to hear how the new API you're proposing
>     will affect this solution I've come up with for pyparallel; I'm
>     going to be absolutely dependent upon the ability to lock main
>     thread pages as read-only in one fell-swoop -- am I still going to
>     be able to do that with your new API in place?
> """
> 
> 2013/6/19 Trent Nelson <trent at snakebite.org>:
> > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> >> >      1. All memory allocated in a parallel context is localized to a
> >> >         private heap.
> >>
> >> How do you allocate memory in this "private" heap? Did you add new
> >> functions to allocate memory?
> >
> >     Yup:
> >         _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
> >
> >     All memory operations (PyObject_New/Malloc etc) get intercepted
> >     during parallel thread execution and redirected to _PyHeap_Malloc(),
> >     which is a very simple slab allocator.  (No need for convoluted
> >     buckets because we never free individual objects during parallel
> >     execution; instead, we just blow everything away at the end.)
> 
> Ok, so I don't think that the PEP 445 would change anything for you.
> 
> The following change might have an impact: If _PyHeap_Malloc is not
> thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
> GIL is not held would avoid bugs in your code.

    Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by
    design, but I'm not sure if we're referring to the same sort of
    thread-safe problem here.

    For one, _PyHeap_Malloc won't ever run if the GIL isn't being held.

    (Parallel threads are only allowed to run when the main thread has
     the GIL held and has relinquished control to parallel threads.)

    Also, I interpret PyMem_RawMalloc() as a direct shortcut to
    malloc() (or something else that returns void *s that are then
    free()'d down the track).  Is that right?

    I don't think that would impact pyparallel.

> If you want to choose dynamically the allocator at runtime, you can
> replace PyObject_Malloc allocator using:
> -------------------------- 8< -----------------
> static void *
> _PxMem_AllocMalloc(void *ctx, size_t size)
> {
>     PyMemBlockAllocator *ctx;
>     if (Py_PXCTX)
>         return _PxMem_Malloc(size))
>     else
>         return alloc->malloc(alloc->ctx, size);
> }
> 
> ...
> 
> PyMemBlockAllocator pyparallel_pyobject;
> 
> static void *
> setup_pyparallel_allocator(void)
> {
>     PyMemBlockAllocator alloc;
>     PyObject_GetAllocator(&pyparallel_pyobject);
>     alloc.ctx = &pyparallel_pyobject;
>     alloc.malloc = _PxMem_AllocMalloc;
>     ...
>     PyObject_SetAllocator(&alloc);
> }
> -------------------------- 8< -----------------
> 
> But I don't know if you want pyparallel to be an "optional" feature
> chosen at runtime...

    Hmmm, those code snippets are interesting.  Time for some more
    homework.

        Trent.

From victor.stinner at gmail.com  Thu Jun 20 00:21:13 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 00:21:13 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <20130619182229.2fa7cd92@fsol>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<20130619161558.1f042553@pitrou.net>
	<CAMpsgwbN=k=FrPx4bguhzLu6NRYx1c3vc+smeKeZ0ou8cOUqZg@mail.gmail.com>
	<20130619174354.34a7d8f0@fsol>
	<CAMpsgwZaFivV8uXfEYFLdNyzodRmKYHeuitdVGs_dqcpOwYkVQ@mail.gmail.com>
	<20130619182229.2fa7cd92@fsol>
Message-ID: <CAMpsgwZ1HzZKn3nGc8TThXFm12yWOfUffBNizY5Hfo0Vya_Y1A@mail.gmail.com>

PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree
that all other new fumctions ans structures should not.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/6d962f94/attachment.html>

From ncoghlan at gmail.com  Thu Jun 20 01:01:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Jun 2013 09:01:49 +1000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
	<CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
Message-ID: <CADiSq7dqkxoeNjHf+LmNkM20PKB=0D8kVCENFzOBNKQMcpvBzA@mail.gmail.com>

On 20 Jun 2013 02:03, "Victor Stinner" <victor.stinner at gmail.com> wrote:
>
> 2013/6/19 Kristj?n Valur J?nsson <kristjan at ccpgames.com>:
> > Oh, it should be public, in my opinion.
>
> Ok. And do you think that the PyMemMappingAllocator structure is
> complete, or that we should add something to be future-proof? At
> least, PyMemMappingAllocator is enough for pymalloc usage :-)
>
> Is PyMemMappingAllocator complete enough for your usage at CCP Games?

Can we go back to calling this the "Arena" allocator? Or at least "Mapped"?
When I see "Mapping" in the context of Python I think of the container API,
not a memory allocation API.

>
> > We do exactly that when we embed python into UnrealEngine.  We keep
pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for
its arenas.  That way, we can still keep track of python's memory usage
from with the larger application, even if the granularity of memory is now
on an "arena" level, rather than individual allocs.
>
> I hope that the PEP 445 is flexible enough to allow you to decide
> which functions are hooked and replaced, and which functions will be
> leaved unchanged. That's why I'm not in favor of the "Make
> PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.

It's also why I'm in favour of the "domain" API rather than separate
functions.

1. In the initial iteration, just have the three basic domains (raw,
interpreter, objects). Replacing allocators for third party libraries is
the responsibility of embedding applications.

2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs
so that extension modules can register new domains for wrapped libraries.
Replacing allocators is still the responsibility of embedding applications,
but there's a consistent API to do it.

(Alternatively, we could do both now)

And agreed PyMem_Raw* are the only new APIs that should be added to the
stable ABI.

Cheers,
Nick.

>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/73c9c52d/attachment.html>

From victor.stinner at gmail.com  Thu Jun 20 07:37:42 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 07:37:42 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
	memory allocators
In-Reply-To: <CADiSq7dqkxoeNjHf+LmNkM20PKB=0D8kVCENFzOBNKQMcpvBzA@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
	<CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
	<CADiSq7dqkxoeNjHf+LmNkM20PKB=0D8kVCENFzOBNKQMcpvBzA@mail.gmail.com>
Message-ID: <CAMpsgwZttWzKAzOufSYaULbo+HFn841-GXS01Lqwm3t+zcRy-A@mail.gmail.com>

Le jeudi 20 juin 2013, Nick Coghlan a ?crit :
>
> > Is PyMemMappingAllocator complete enough for your usage at CCP Games?
>
> Can we go back to calling this the "Arena" allocator? Or at least
> "Mapped"? When I see "Mapping" in the context of Python I think of the
> container API, not a memory allocation API.
>
This function is written to be able to use mmap() and VirtualAlloc(). There
is no Python function to use directly this allocator yet, but I chose
"memory mapping" name because it is very different than the heap and it may
be useful for other functions than pymalloc.

If I change the name, it would be called PyObject_SetArenaAllocator() with
a PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator
API is future-proof, so I'm fine to call it "arena" again.

> > I hope that the PEP 445 is flexible enough to allow you to decide
> > which functions are hooked and replaced, and which functions will be
> > leaved unchanged. That's why I'm not in favor of the "Make
> > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.
>
> It's also why I'm in favour of the "domain" API rather than separate
> functions.
>
> 1. In the initial iteration, just have the three basic domains (raw,
> interpreter, objects). Replacing allocators for third party libraries is
> the responsibility of embedding applications.
>
> 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs
> so that extension modules can register new domains for wrapped libraries.
> Replacing allocators is still the responsibility of embedding applications,
> but there's a consistent API to do it.
>
> (Alternatively, we could do both now)
>
How would you use an allocator of a new domain? PyMemBlockAllocator
structure is not convinient, and if Py_GetAllocator() only once, you may
loose a hook installed later.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/b3b40587/attachment-0001.html>

From victor.stinner at gmail.com  Thu Jun 20 12:16:56 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 12:16:56 +0200
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
	memory allocators
In-Reply-To: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
Message-ID: <CAMpsgwYhKROwEQxt42CTdGFhiUr+NB_ApXVK=gn9fcrO-VUF=g@mail.gmail.com>

> * Add new GIL-free (no need to hold the GIL) memory allocator functions:
>
>   - ``void* PyMem_RawMalloc(size_t size)``
>   - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
>   - ``void PyMem_RawFree(void *ptr)``
>   - the behaviour of requesting zero bytes is not defined: return *NULL*
>     or a distinct non-*NULL* pointer depending on the platform.
> (...)
> * Add new functions to get and set internal functions of
>   ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``:
>
>   - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
>   - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
>   - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
>     *NULL*: it would be treated as an error.
>   - default allocator: ``malloc()``, ``realloc()``, ``free()``;
>     ``PyMem_Malloc(0)`` calls ``malloc(1)``
>     and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)``

Oh, one more question: PyMem_RawMalloc(0) has an undefined behaviour,
whereas PyMem_Malloc(0) has a well defined behaviour (don't return
NULL). Adding "if (size == 1) size = 0;" in the default implementation
of PyMem_RawMalloc(0) should not have a visible overhead, but it gives
the same guarantee than PyMem_Malloc(0) (don't return NULL). Do you
agree to add the test?

I chose to implement "if (size > (size_t)PY_SSIZE_T_MAX) return NULL;"
in Py*_Malloc(), whereas "if (size == 0) size =1;" is implemented in
the inner function (_PyMem_Malloc). An application may use an
allocator which has already a well defined behaviour (a "malloc(0)"
that don't return NULL) and I expect malloc(1) to allocate "more"
memory than malloc(0) (malloc(0) may create a singleton) :-)

Victor

From kristjan at ccpgames.com  Thu Jun 20 12:44:08 2013
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Thu, 20 Jun 2013 10:44:08 +0000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
	<CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
Message-ID: <EFE3877620384242A686D52278B7CCD3539524BB@RKV-IT-EXCH104.ccp.ad.local>

Oh, please don't misunderstand.  I'm not making any demands or requirements, 
what I'm trying to do is to make recommendations based on experience that I 
have had with embedding.  This sound altogether too much like I'm trying to
push things one way or the other :)

The api as laid out certainly seems to work, and be adequate for the purpose.

I can add here as a point of information that since we
work on windows, there was no need to pass in the "size" argument to the
"munmap" callback.  VirtualFree(address, NULL) will release the entire chunk
of memory that was initially allocated at that place.  Therefor in our implementation
we can reuse the same allocator structo for those arenas.  But I understand
that munmap doesn't have this feature, so passing in the size is prudent.

K

> -----Original Message-----
> From: Victor Stinner [mailto:victor.stinner at gmail.com]
> Sent: 19. j?n? 2013 15:59
> To: Kristj?n Valur J?nsson
> Cc: Python Dev
> Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
> memory allocators
> 
> Is PyMemMappingAllocator complete enough for your usage at CCP Games?


From victor.stinner at gmail.com  Thu Jun 20 13:26:52 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 13:26:52 +0200
Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory
	allocators (second round)
Message-ID: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>

Hi,

I changed the PEP 445 according to the discussing on python-dev.

Read it online:
http://www.python.org/dev/peps/pep-0445/

Changes:

 * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or
PYALLOC_PYOBJECT
 * rename:

   - PyMemBlockAllocator structure => PyMemAllocator
   - PyMemMappingAllocator structure => PyObjectArenaAllocator
   - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator()
   - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator()

 * group get/set functions to only keep 2 functions:
   PyMem_GetAllocator() and PyMem_SetAllocator()
 * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined behaviour
 * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same allocator
 * Add more references for external libraries

As expected, most changes occurred in the Proposal section.

Full diff:
http://hg.python.org/peps/rev/a17ebebe52ca

I also updated the implementation attached to:
http://bugs.python.org/issue3329

Who is going give the final decision on this PEP? Guido? Another candidate?

Victor

From victor.stinner at gmail.com  Thu Jun 20 13:28:23 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 13:28:23 +0200
Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory
 allocators (second round)
In-Reply-To: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>
References: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>
Message-ID: <CAMpsgwaFh4e=c48XNG1=RAJY1fG5mjKnw-kSTuTpsdx8gwBP1Q@mail.gmail.com>

> I also updated the implementation attached to:
> http://bugs.python.org/issue3329

I excluded new enums, structures and functions from the stable ABI,
except PyMem_Raw*() functions.

Victor

From solipsis at pitrou.net  Thu Jun 20 14:16:48 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 20 Jun 2013 14:16:48 +0200
Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory
 allocators (second round)
References: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>
Message-ID: <20130620141648.0dbedfc1@pitrou.net>

Le Thu, 20 Jun 2013 13:26:52 +0200,
Victor Stinner <victor.stinner at gmail.com> a ?crit :

> Hi,
> 
> I changed the PEP 445 according to the discussing on python-dev.
> 
> Read it online:
> http://www.python.org/dev/peps/pep-0445/
> 
> Changes:
> 
>  * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or
> PYALLOC_PYOBJECT

PYMEM_DOMAIN_RAW?


>  * rename:
> 
>    - PyMemBlockAllocator structure => PyMemAllocator
>    - PyMemMappingAllocator structure => PyObjectArenaAllocator
>    - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator()
>    - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator()
> 
>  * group get/set functions to only keep 2 functions:
>    PyMem_GetAllocator() and PyMem_SetAllocator()
>  * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined
> behaviour
>  * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same
> allocator
>  * Add more references for external libraries
> 
> As expected, most changes occurred in the Proposal section.
> 
> Full diff:
> http://hg.python.org/peps/rev/a17ebebe52ca
> 
> I also updated the implementation attached to:
> http://bugs.python.org/issue3329
> 
> Who is going give the final decision on this PEP? Guido? Another
> candidate?
> 
> Victor




From ncoghlan at gmail.com  Thu Jun 20 14:19:58 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Jun 2013 22:19:58 +1000
Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python
 memory allocators
In-Reply-To: <CAMpsgwZttWzKAzOufSYaULbo+HFn841-GXS01Lqwm3t+zcRy-A@mail.gmail.com>
References: <CAMpsgwZkJ6RND-_GOkdLymTV4eN+WWrCco2S06cfUCJkyYZELw@mail.gmail.com>
	<CAMpsgwYTMAMFj+=0B7=qBRgUKvJ9_k8x9A8afBawBX9ZZ02NFA@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD35395178E@RKV-IT-EXCH104.ccp.ad.local>
	<CAMpsgwY3HbsWOvTrNMwqAW+8359bzURhtARRe9f0VYRgV2jnMg@mail.gmail.com>
	<CADiSq7dqkxoeNjHf+LmNkM20PKB=0D8kVCENFzOBNKQMcpvBzA@mail.gmail.com>
	<CAMpsgwZttWzKAzOufSYaULbo+HFn841-GXS01Lqwm3t+zcRy-A@mail.gmail.com>
Message-ID: <CADiSq7eXGMAoi5A8Wv5wthHRTeaRbuFkOp_+eLdNRTh9kcfdpA@mail.gmail.com>

On 20 June 2013 15:37, Victor Stinner <victor.stinner at gmail.com> wrote:
> Le jeudi 20 juin 2013, Nick Coghlan a ?crit :
>>
>> > Is PyMemMappingAllocator complete enough for your usage at CCP Games?
>>
>> Can we go back to calling this the "Arena" allocator? Or at least
>> "Mapped"? When I see "Mapping" in the context of Python I think of the
>> container API, not a memory allocation API.
>
> This function is written to be able to use mmap() and VirtualAlloc(). There
> is no Python function to use directly this allocator yet, but I chose
> "memory mapping" name because it is very different than the heap and it may
> be useful for other functions than pymalloc.
>
> If I change the name, it would be called PyObject_SetArenaAllocator() with a
> PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator
> API is future-proof, so I'm fine to call it "arena" again.

Yeah, I think making that API specifically about pymalloc is a good
idea. It also makes it clearer that if you're bypassing pymalloc
entirely (by replacing the
object allocators), then you shouldn't need to worry about those.

>> > I hope that the PEP 445 is flexible enough to allow you to decide
>> > which functions are hooked and replaced, and which functions will be
>> > leaved unchanged. That's why I'm not in favor of the "Make
>> > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.
>>
>> It's also why I'm in favour of the "domain" API rather than separate
>> functions.
>>
>> 1. In the initial iteration, just have the three basic domains (raw,
>> interpreter, objects). Replacing allocators for third party libraries is the
>> responsibility of embedding applications.
>>
>> 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs
>> so that extension modules can register new domains for wrapped libraries.
>> Replacing allocators is still the responsibility of embedding applications,
>> but there's a consistent API to do it.
>>
>> (Alternatively, we could do both now)
>
> How would you use an allocator of a new domain? PyMemBlockAllocator
> structure is not convinient, and if Py_GetAllocator() only once, you may
> loose a hook installed later.

Say that, for the current PEP, we assume we configure standard library
extension modules to use the PyMem or PyMem_Raw APIs (depending on GIL
usage), thus allowing those to be redirected automatically to an
externally configured allocator when the new PEP 445 APIs are used.

The notion I had an mind as a possible future change is that extension
modules could register a "set_allocator" callback with the interpreter
so they will be automatically notified if the allocator they're
interested in changes. However, I also realised that this would
actually be independent of the APIs in the current PEP. You could do
something like:

    typedef void (*PyMem_AllocatorSetter)(PyMemAllocator *allocator);

    void
    PyMem_AddExternalAllocator(
        PyMemAllocatorDomain domain,
        PyMemAllocatorSetter set_allocator
    );

Then, whenever the allocator for the specified domain was changed, the
"set_allocator" callback would be invoked to set the allocator in the
extension module as well. The setter would also be called immediately
on registration, using the currently defined allocator.

We don't have to do this right away (and we should give the basic API
a chance to establish itself first). I just like it as something that
the "memory domain" model may allow us to pursue in the future. (That
said, we may end up wanting something like this internally anyway,
even just for the standard library extension modules that do memory
allocations)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Thu Jun 20 15:05:34 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 20 Jun 2013 15:05:34 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
Message-ID: <51C2FE1E.8020801@python.org>

Hello,

I have re-implemented the entire stat module in C. [1] It's a necessary
step to fix portability issues. For most constants POSIX standards
dictate only the name of the constant and its meaning but not its value.
Only the values of permission bits (0644 == rw-r--r--) have fixed values.

For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some
platforms. Common file types seem to have the same value on all
important platforms. It's the only reason we were able to get away with
stat.py. But uncommon file types like door, event port and whiteout
don't have sensible default values. The C implementation is able to pick
up the platform's values easily.

What shall I do about stat.py? statmodule.c implements all features of
stat.py so there is no point in using it in CPython. It's one Python
file less to load on every startup. However it might still be useful to
alternative implementations of Python such as Jython or PyPy.

1) keep the file stat.py but let it be shadowed by the builtin stat
module. Antoine loathes my hack...
2) rename stat.py to _stat.py
3) remove stat.py

Opinions?

Christian

[1] http://bugs.python.org/issue11016

From florent.xicluna at gmail.com  Thu Jun 20 15:21:13 2013
From: florent.xicluna at gmail.com (Florent)
Date: Thu, 20 Jun 2013 15:21:13 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C2FE1E.8020801@python.org>
References: <51C2FE1E.8020801@python.org>
Message-ID: <CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>

we already have "_pyio.py", we could have "_pystat.py".

my 2c

--
Florent Xicluna

2013/6/20 Christian Heimes <christian at python.org>:
> Hello,
>
> I have re-implemented the entire stat module in C. [1] It's a necessary
> step to fix portability issues. For most constants POSIX standards
> dictate only the name of the constant and its meaning but not its value.
> Only the values of permission bits (0644 == rw-r--r--) have fixed values.
>
> For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some
> platforms. Common file types seem to have the same value on all
> important platforms. It's the only reason we were able to get away with
> stat.py. But uncommon file types like door, event port and whiteout
> don't have sensible default values. The C implementation is able to pick
> up the platform's values easily.
>
> What shall I do about stat.py? statmodule.c implements all features of
> stat.py so there is no point in using it in CPython. It's one Python
> file less to load on every startup. However it might still be useful to
> alternative implementations of Python such as Jython or PyPy.
>
> 1) keep the file stat.py but let it be shadowed by the builtin stat
> module. Antoine loathes my hack...
> 2) rename stat.py to _stat.py
> 3) remove stat.py
>
> Opinions?
>
> Christian
>
> [1] http://bugs.python.org/issue11016
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/florent.xicluna%40gmail.com

From christian at python.org  Thu Jun 20 15:29:42 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 20 Jun 2013 15:29:42 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
Message-ID: <51C303C6.8050609@python.org>

Am 20.06.2013 15:21, schrieb Florent:
> we already have "_pyio.py", we could have "_pystat.py".

Works for me.


From brett at python.org  Thu Jun 20 15:36:07 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jun 2013 09:36:07 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <3bbSw10tm2z7Ljf@mail.python.org>
References: <3bbSw10tm2z7Ljf@mail.python.org>
Message-ID: <CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>

On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran <
python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/dfead0696a71
> changeset:   84224:dfead0696a71
> branch:      3.3
> parent:      84221:0113247f894b
> user:        Senthil Kumaran <senthil at uthcode.com>
> date:        Wed Jun 19 22:19:46 2013 -0500
> summary:
>   Add -b and -X options to python man page.
> Patch contributed by Corey Brune.
>
> files:
>   Misc/python.man |  22 ++++++++++++++++++----
>   1 files changed, 18 insertions(+), 4 deletions(-)
>
>
> diff --git a/Misc/python.man b/Misc/python.man
> --- a/Misc/python.man
> +++ b/Misc/python.man
> @@ -11,6 +11,9 @@
>  .B \-B
>  ]
>  [
> +.B \-b
> +]
> +[
>  .B \-d
>  ]
>  [
> @@ -23,14 +26,14 @@
>  .B \-i
>  ]
>  [
> -.B \-m
> +.B \-m
>  .I module-name
>  ]
> -[
> -.B \-q
> -]
>  .br
>         [
> +.B \-q
> +]
> +[
>  .B \-O
>  ]
>  [
> @@ -60,6 +63,10 @@
>  .B \-x
>  ]
>  [
> +[
> +.B \-X
> +.I option
> +]
>  .B \-?
>  ]
>  .br
> @@ -105,6 +112,10 @@
>  .I .py[co]
>  files on import. See also PYTHONDONTWRITEBYTECODE.
>  .TP
> +.B \-b
> +Issue warnings about str(bytes_instance), str(bytearray_instance)
> +and comparing bytes/bytearray with str. (-bb: issue errors)
> +.TP
>  .BI "\-c " command
>  Specify the command to execute (see next section).
>  This terminates the option list (following options are passed as
> @@ -243,6 +254,9 @@
>  field matches the line number, where zero matches all line numbers and
>  is thus equivalent to an omitted line number.
>  .TP
> +.BI "\-X " option
> +Set implementation specific option.
>

Should probably be "Set the implementation-specific option."


> +.TP
>  .B \-x
>  Skip the first line of the source.  This is intended for a DOS
>  specific hack only.  Warning: the line numbers in error messages will
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/f12fa034/attachment.html>

From ncoghlan at gmail.com  Thu Jun 20 15:34:44 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Jun 2013 23:34:44 +1000
Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory
 allocators (second round)
In-Reply-To: <20130620141648.0dbedfc1@pitrou.net>
References: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>
	<20130620141648.0dbedfc1@pitrou.net>
Message-ID: <CADiSq7dQ6WOddRrXsR1MN7-+cBvk9XTyqMP4-JoxbYUxYuFe-A@mail.gmail.com>

On 20 June 2013 22:16, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Thu, 20 Jun 2013 13:26:52 +0200,
> Victor Stinner <victor.stinner at gmail.com> a ?crit :
>
>> Hi,
>>
>> I changed the PEP 445 according to the discussing on python-dev.
>>
>> Read it online:
>> http://www.python.org/dev/peps/pep-0445/
>>
>> Changes:
>>
>>  * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or
>> PYALLOC_PYOBJECT
>
> PYMEM_DOMAIN_RAW?

The challenge there is coming up with a good name for PYALLOC_PYMEM.

I think Victor's names for the domains work well:
- "PYALLOC_" prefix to say "this is an allocation domain"
- "PYMEM_RAW" suffix for PyMem_RawMalloc/Realloc/Free
- "PYMEM" suffix for PyMem_Malloc/Realloc/Free
- "PYOBJECT" suffix for PyObject_Malloc/Realloc/Free

Although, there's a copy-and-paste error in the domain definitions (2
*Reallocs in each entry and no *Frees).

If Guido doesn't want to do it, I'm happy to. Antoine would be a good
choice too, since he's the one who suggested taking the time to thrash
out the API details in a PEP (which I agree has substantially improved
the end result).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Jun 20 15:40:13 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Jun 2013 23:40:13 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C303C6.8050609@python.org>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
Message-ID: <CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>

On 20 June 2013 23:29, Christian Heimes <christian at python.org> wrote:
> Am 20.06.2013 15:21, schrieb Florent:
>> we already have "_pyio.py", we could have "_pystat.py".
>
> Works for me.

I suggest following the guidelines in PEP 399 for cross implementation
compatibility of the standard library:
http://www.python.org/dev/peps/pep-0399/#details

1. Keep stat.py
2. Make the C version "_stat"
3. Add the following to the end of stat.py:

    try:
        from _stat import *
    except ImportError:
        pass

Run the tests with and without the C module in the test suite (again,
as per PEP 399).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eric at trueblade.com  Thu Jun 20 16:14:25 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 20 Jun 2013 10:14:25 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
Message-ID: <51C30E41.9090301@trueblade.com>

On 6/20/2013 9:40 AM, Nick Coghlan wrote:
> On 20 June 2013 23:29, Christian Heimes <christian at python.org> wrote:
>> Am 20.06.2013 15:21, schrieb Florent:
>>> we already have "_pyio.py", we could have "_pystat.py".
>>
>> Works for me.
> 
> I suggest following the guidelines in PEP 399 for cross implementation
> compatibility of the standard library:
> http://www.python.org/dev/peps/pep-0399/#details
> 
> 1. Keep stat.py
> 2. Make the C version "_stat"
> 3. Add the following to the end of stat.py:
> 
>     try:
>         from _stat import *
>     except ImportError:
>         pass
> 
> Run the tests with and without the C module in the test suite (again,
> as per PEP 399).

Agreed with the above.

But isn't the real problem with this module in Python the fact that the
constants might be wrong? I'm not sure what, if anything, we can do
about that.

-- 
Eric.

From brett at python.org  Thu Jun 20 17:01:32 2013
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jun 2013 11:01:32 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C30E41.9090301@trueblade.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
Message-ID: <CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>

On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric at trueblade.com> wrote:

> On 6/20/2013 9:40 AM, Nick Coghlan wrote:
> > On 20 June 2013 23:29, Christian Heimes <christian at python.org> wrote:
> >> Am 20.06.2013 15:21, schrieb Florent:
> >>> we already have "_pyio.py", we could have "_pystat.py".
> >>
> >> Works for me.
> >
> > I suggest following the guidelines in PEP 399 for cross implementation
> > compatibility of the standard library:
> > http://www.python.org/dev/peps/pep-0399/#details
> >
> > 1. Keep stat.py
> > 2. Make the C version "_stat"
> > 3. Add the following to the end of stat.py:
> >
> >     try:
> >         from _stat import *
> >     except ImportError:
> >         pass
> >
> > Run the tests with and without the C module in the test suite (again,
> > as per PEP 399).
>
> Agreed with the above.
>
> But isn't the real problem with this module in Python the fact that the
> constants might be wrong? I'm not sure what, if anything, we can do
> about that.
>

There isn't anything we can do beyond at least trying to provide reasonable
defaults when something better isn't available (which is what the stats
module has done all this time). It might make testing difficult for the
Python code when the C code has the right values, but I don't think it's
necessarily worth tossing out the Python code as backup.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/5fa8a284/attachment.html>

From thomas at python.org  Thu Jun 20 17:04:56 2013
From: thomas at python.org (Thomas Wouters)
Date: Thu, 20 Jun 2013 08:04:56 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
Message-ID: <CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>

On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon <brett at python.org> wrote:

>
>
>
> On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric at trueblade.com>wrote:
>
>> On 6/20/2013 9:40 AM, Nick Coghlan wrote:
>> > On 20 June 2013 23:29, Christian Heimes <christian at python.org> wrote:
>> >> Am 20.06.2013 15:21, schrieb Florent:
>> >>> we already have "_pyio.py", we could have "_pystat.py".
>> >>
>> >> Works for me.
>> >
>> > I suggest following the guidelines in PEP 399 for cross implementation
>> > compatibility of the standard library:
>> > http://www.python.org/dev/peps/pep-0399/#details
>> >
>> > 1. Keep stat.py
>> > 2. Make the C version "_stat"
>> > 3. Add the following to the end of stat.py:
>> >
>> >     try:
>> >         from _stat import *
>> >     except ImportError:
>> >         pass
>> >
>> > Run the tests with and without the C module in the test suite (again,
>> > as per PEP 399).
>>
>> Agreed with the above.
>>
>> But isn't the real problem with this module in Python the fact that the
>> constants might be wrong? I'm not sure what, if anything, we can do
>> about that.
>>
>
> There isn't anything we can do beyond at least trying to provide
> reasonable defaults when something better isn't available (which is what
> the stats module has done all this time). It might make testing difficult
> for the Python code when the C code has the right values, but I don't think
> it's necessarily worth tossing out the Python code as backup.
>

If the .py file is going to be wrong or incomplete, why would we want to
keep it -- or use it as fallback -- at all? If we're dead set on having a
.py file instead of requiring it to be part of the interpreter (whichever
that is, however it was built), it should be generated as part of the build
process. Personally, I don't see the value in it; other implementations
will need to do *something* special to use it anyway.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/0e7cf0cb/attachment.html>

From dholth at gmail.com  Thu Jun 20 17:09:03 2013
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 20 Jun 2013 11:09:03 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
Message-ID: <CAG8k2+7p0nxxA-mWcdPwFQA7g1gN0qLfP7e4T_X7Twrfcbyfqg@mail.gmail.com>

cffi makes this kind of constant-grabbing very easy. However this is a
tiny module so no problem with having a C version.

On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters <thomas at python.org> wrote:
>
>
>
> On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon <brett at python.org> wrote:
>>
>>
>>
>>
>> On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith <eric at trueblade.com>
>> wrote:
>>>
>>> On 6/20/2013 9:40 AM, Nick Coghlan wrote:
>>> > On 20 June 2013 23:29, Christian Heimes <christian at python.org> wrote:
>>> >> Am 20.06.2013 15:21, schrieb Florent:
>>> >>> we already have "_pyio.py", we could have "_pystat.py".
>>> >>
>>> >> Works for me.
>>> >
>>> > I suggest following the guidelines in PEP 399 for cross implementation
>>> > compatibility of the standard library:
>>> > http://www.python.org/dev/peps/pep-0399/#details
>>> >
>>> > 1. Keep stat.py
>>> > 2. Make the C version "_stat"
>>> > 3. Add the following to the end of stat.py:
>>> >
>>> >     try:
>>> >         from _stat import *
>>> >     except ImportError:
>>> >         pass
>>> >
>>> > Run the tests with and without the C module in the test suite (again,
>>> > as per PEP 399).
>>>
>>> Agreed with the above.
>>>
>>> But isn't the real problem with this module in Python the fact that the
>>> constants might be wrong? I'm not sure what, if anything, we can do
>>> about that.
>>
>>
>> There isn't anything we can do beyond at least trying to provide
>> reasonable defaults when something better isn't available (which is what the
>> stats module has done all this time). It might make testing difficult for
>> the Python code when the C code has the right values, but I don't think it's
>> necessarily worth tossing out the Python code as backup.
>
>
> If the .py file is going to be wrong or incomplete, why would we want to
> keep it -- or use it as fallback -- at all? If we're dead set on having a
> .py file instead of requiring it to be part of the interpreter (whichever
> that is, however it was built), it should be generated as part of the build
> process. Personally, I don't see the value in it; other implementations will
> need to do *something* special to use it anyway.
>
> --
> Thomas Wouters <thomas at python.org>
>
> Hi! I'm an email virus! Think twice before sending your email to help me
> spread!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>

From cf.natali at gmail.com  Thu Jun 20 17:11:05 2013
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Thu, 20 Jun 2013 17:11:05 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
Message-ID: <CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>

2013/6/20 Thomas Wouters <thomas at python.org>:
> If the .py file is going to be wrong or incomplete, why would we want to
> keep it -- or use it as fallback -- at all? If we're dead set on having a
> .py file instead of requiring it to be part of the interpreter (whichever
> that is, however it was built), it should be generated as part of the build
> process. Personally, I don't see the value in it; other implementations will
> need to do *something* special to use it anyway.

That's exactly my rationale for pushing for removal.

cf

From benjamin at python.org  Thu Jun 20 17:35:22 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 20 Jun 2013 08:35:22 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
Message-ID: <CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>

2013/6/20 Charles-Fran?ois Natali <cf.natali at gmail.com>:
> 2013/6/20 Thomas Wouters <thomas at python.org>:
>> If the .py file is going to be wrong or incomplete, why would we want to
>> keep it -- or use it as fallback -- at all? If we're dead set on having a
>> .py file instead of requiring it to be part of the interpreter (whichever
>> that is, however it was built), it should be generated as part of the build
>> process. Personally, I don't see the value in it; other implementations will
>> need to do *something* special to use it anyway.
>
> That's exactly my rationale for pushing for removal.

+1 to nixing it.



--
Regards,
Benjamin

From christian at python.org  Thu Jun 20 20:16:23 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 20 Jun 2013 20:16:23 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
	<CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
Message-ID: <51C346F7.3080302@python.org>

Am 20.06.2013 17:35, schrieb Benjamin Peterson:
> 2013/6/20 Charles-Fran?ois Natali <cf.natali at gmail.com>:
>> 2013/6/20 Thomas Wouters <thomas at python.org>:
>>> If the .py file is going to be wrong or incomplete, why would we want to
>>> keep it -- or use it as fallback -- at all? If we're dead set on having a
>>> .py file instead of requiring it to be part of the interpreter (whichever
>>> that is, however it was built), it should be generated as part of the build
>>> process. Personally, I don't see the value in it; other implementations will
>>> need to do *something* special to use it anyway.
>>
>> That's exactly my rationale for pushing for removal.
> 
> +1 to nixing it.

I'm +0 for removal. The stat module contains merely constants, wrappers
for macros and one simple functions.

Somebody just say the word.

Christian


From storchaka at gmail.com  Thu Jun 20 21:16:08 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 20 Jun 2013 22:16:08 +0300
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C2FE1E.8020801@python.org>
References: <51C2FE1E.8020801@python.org>
Message-ID: <kpvkdi$qeb$1@ger.gmane.org>

20.06.13 16:05, Christian Heimes ???????(??):
> I have re-implemented the entire stat module in C. [1] It's a necessary
> step to fix portability issues. For most constants POSIX standards
> dictate only the name of the constant and its meaning but not its value.
> Only the values of permission bits (0644 == rw-r--r--) have fixed values.
>
> For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some
> platforms. Common file types seem to have the same value on all
> important platforms. It's the only reason we were able to get away with
> stat.py. But uncommon file types like door, event port and whiteout
> don't have sensible default values. The C implementation is able to pick
> up the platform's values easily.
>
> What shall I do about stat.py? statmodule.c implements all features of
> stat.py so there is no point in using it in CPython. It's one Python
> file less to load on every startup. However it might still be useful to
> alternative implementations of Python such as Jython or PyPy.
>
> 1) keep the file stat.py but let it be shadowed by the builtin stat
> module. Antoine loathes my hack...
> 2) rename stat.py to _stat.py
> 3) remove stat.py
>
> Opinions?

Now with enumerations in the stdlib the stat module constants are 
candidates for flag enumerations. How easy will be implement it on C?



From lukasz at langa.pl  Thu Jun 20 22:05:19 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Thu, 20 Jun 2013 22:05:19 +0200
Subject: [Python-Dev] How about we extend C3 to support ABCs?
Message-ID: <54088A4F-99E9-4B95-B3D3-A3A9D079B413@langa.pl>

Review and comments on the ticket needed, please.

http://bugs.python.org/issue18244#msg191535

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/e96e3d9d/attachment.html>

From amauryfa at gmail.com  Thu Jun 20 22:18:54 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 20 Jun 2013 22:18:54 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <kpvkdi$qeb$1@ger.gmane.org>
References: <51C2FE1E.8020801@python.org>
	<kpvkdi$qeb$1@ger.gmane.org>
Message-ID: <CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>

2013/6/20 Serhiy Storchaka <storchaka at gmail.com>

> Now with enumerations in the stdlib the stat module constants are
> candidates for flag enumerations. How easy will be implement it on C?


Aha. Should an internal C module fetch the value of the constants, and a
public stat.py nicely wrap them in enums?


-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/f70df6b9/attachment-0001.html>

From guido at python.org  Thu Jun 20 22:27:16 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 20 Jun 2013 13:27:16 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org> <kpvkdi$qeb$1@ger.gmane.org>
	<CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>
Message-ID: <CAP7+vJKJqOYo5jtZihRtfxiEzyWWzyAqGUoTtYU+V2q4m5_pzA@mail.gmail.com>

But aren't most of these masks? enum,IntEnum doesn't handle those very
gracefully (the | operator returns a plain int).

On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2013/6/20 Serhiy Storchaka <storchaka at gmail.com>
>>
>> Now with enumerations in the stdlib the stat module constants are
>> candidates for flag enumerations. How easy will be implement it on C?
>
>
> Aha. Should an internal C module fetch the value of the constants, and a
> public stat.py nicely wrap them in enums?
>
>
> --
> Amaury Forgeot d'Arc
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From ethan at stoneleaf.us  Thu Jun 20 22:24:48 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 20 Jun 2013 13:24:48 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org> <kpvkdi$qeb$1@ger.gmane.org>
	<CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>
Message-ID: <51C36510.5060305@stoneleaf.us>

On 06/20/2013 01:18 PM, Amaury Forgeot d'Arc wrote:
> 2013/6/20 Serhiy Storchaka wrote:
>>
>> Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be
>> implement it on C?
>
> Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums?

That's pretty much the approach I'm thinking about for sockets.

--
~Ethan~

From victor.stinner at gmail.com  Thu Jun 20 22:52:22 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 22:52:22 +0200
Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory
	allocators (second round)
In-Reply-To: <CADiSq7dQ6WOddRrXsR1MN7-+cBvk9XTyqMP4-JoxbYUxYuFe-A@mail.gmail.com>
References: <CAMpsgwZAtRBdY3tH0w87TH8Ev6YSmqi+fW1sGoGB4UdFT-KeXg@mail.gmail.com>
	<20130620141648.0dbedfc1@pitrou.net>
	<CADiSq7dQ6WOddRrXsR1MN7-+cBvk9XTyqMP4-JoxbYUxYuFe-A@mail.gmail.com>
Message-ID: <CAMpsgwaBS_tC8KsUPdGKUDc_=fyiqjvgK1D+LnT7+RJK-J5ZXQ@mail.gmail.com>

2013/6/20 Antoine Pitrou <solipsis at pitrou.net <javascript:;>>:
> Victor Stinner <victor.stinner at gmail.com <javascript:;>> a ?crit :
>> Changes:
>>
>>  * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or
>> PYALLOC_PYOBJECT
>
> PYMEM_DOMAIN_RAW?

I prefer your suggestion because it shares the PYMEM/PyMem prefix with
PyMem_SetAllocator().

So the whole list would be:

- PYMEM_DOMAIN_RAW
- PYMEM_DOMAIN_MEM
- PYMEM_DOMAIN_OBJ

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/fac70658/attachment.html>

From victor.stinner at gmail.com  Thu Jun 20 22:53:04 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Jun 2013 22:53:04 +0200
Subject: [Python-Dev]  stat module in C -- what to do with stat.py?
In-Reply-To: <51C30E41.9090301@trueblade.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
Message-ID: <CAMpsgwaCBS0CRwG-q_F8Pkm7KXo8u9939=3e7-m13LyzW4_qTw@mail.gmail.com>

2013/6/20 Eric V. Smith <eric at trueblade.com <javascript:;>>:
> But isn't the real problem with this module in Python the fact that the
> constants might be wrong? I'm not sure what, if anything, we can do
> about that.

Python is providing a stat module implemented in Python since 10 years, or
maybe 20 years, and I don't remember that someone complained that constants
are wrong.

At the same time, Python distributes IN.py which contains things like that:

LONG_MAX = 9223372036854775807
LONG_MAX = 2147483647
LONG_MIN = (-LONG_MAX - 1)

In my opinion, the situation of plat-*/*.py modules is worse than stat.py
:-)

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/ab246690/attachment.html>

From eric at trueblade.com  Thu Jun 20 23:55:22 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 20 Jun 2013 17:55:22 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAMpsgwaCBS0CRwG-q_F8Pkm7KXo8u9939=3e7-m13LyzW4_qTw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAMpsgwaCBS0CRwG-q_F8Pkm7KXo8u9939=3e7-m13LyzW4_qTw@mail.gmail.com>
Message-ID: <51C37A4A.3050102@trueblade.com>

On 6/20/2013 4:53 PM, Victor Stinner wrote:
> 2013/6/20 Eric V. Smith <eric at trueblade.com <javascript:;>>:
>> But isn't the real problem with this module in Python the fact that the
>> constants might be wrong? I'm not sure what, if anything, we can do
>> about that.
> 
> Python is providing a stat module implemented in Python since 10 years,
> or maybe 20 years, and I don't remember that someone complained that
> constants are wrong.

This is serious, not argumentative: If there's really no concern that
the values be correct, then why implement it in C? Let's just leave the
python version with the hard-coded, usually-correct values.

My personal opinion is that having the correct values in C is the way to
go, and delete stat.py.

Eric.


From victor.stinner at gmail.com  Fri Jun 21 00:18:16 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 21 Jun 2013 00:18:16 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C37A4A.3050102@trueblade.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAMpsgwaCBS0CRwG-q_F8Pkm7KXo8u9939=3e7-m13LyzW4_qTw@mail.gmail.com>
	<51C37A4A.3050102@trueblade.com>
Message-ID: <CAMpsgwaaEeqqe5umSrFpCxX6YTe0z05-sgkJmeW8SwVpcXs13w@mail.gmail.com>

2013/6/20 Eric V. Smith <eric at trueblade.com>:
> This is serious, not argumentative: If there's really no concern that
> the values be correct, then why implement it in C?

I read again the issue. The problem is to add new flags. Current flags
(type: block device/symlink/..., file mode) are well defined and
portable, whereas new flags are usually specific to an OS and no
standardize yet. Examples:

- Solaris: "door"
- Solaris: "event port"
- OSX/Darwin/FreeBSD: "whiteout"

The other problem is that values of the new flags are not portable.
Martin v. L?wis wrote:
"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
0xe000 as S_IFWHT (whiteout), but Solaris defines it as
S_IFPORT (event port)."

A C module reusing existing S_ISxxx() macros is the obvious way to
solve this issue.

Victor

From ethan at stoneleaf.us  Fri Jun 21 00:19:55 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 20 Jun 2013 15:19:55 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAP7+vJKJqOYo5jtZihRtfxiEzyWWzyAqGUoTtYU+V2q4m5_pzA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org> <kpvkdi$qeb$1@ger.gmane.org>
	<CAGmFidYr_sib8kFkOV8ZuSjhHB4ZB8emwbi8OkkLDKb5As8AEw@mail.gmail.com>
	<CAP7+vJKJqOYo5jtZihRtfxiEzyWWzyAqGUoTtYU+V2q4m5_pzA@mail.gmail.com>
Message-ID: <51C3800B.3060404@stoneleaf.us>

On 06/20/2013 01:27 PM, Guido van Rossum wrote:
> On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote:
>> 2013/6/20 Serhiy Storchaka wrote:
>>>
>>> Now with enumerations in the stdlib the stat module constants are
>>> candidates for flag enumerations. How easy will be implement it on C?
>>
>> Aha. Should an internal C module fetch the value of the constants, and a
>> public stat.py nicely wrap them in enums?
>
> But aren't most of these masks? enum.IntEnum doesn't handle those very
> gracefully (the | operator returns a plain int).

As is, IntEnum would not be a great choice.

We could either add a another Enum (BitMaskEnum?) that was also int-based, or flesh out IntEnum with enough smarts to 
handle |, &, and ^ and return an IntEnum when possible and fall back to a plain int when not possible.

--
~Ethan~

From victor.stinner at gmail.com  Fri Jun 21 00:58:44 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 21 Jun 2013 00:58:44 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <kpvkdi$qeb$1@ger.gmane.org>
References: <51C2FE1E.8020801@python.org> <kpvkdi$qeb$1@ger.gmane.org>
Message-ID: <CAMpsgwYngExmJnSj2fs4BeUU1ibA7SztpHQ5+VOU5b+cEh7e6Q@mail.gmail.com>

2013/6/20 Serhiy Storchaka <storchaka at gmail.com>:
> Now with enumerations in the stdlib the stat module constants are candidates
> for flag enumerations. How easy will be implement it on C?

Numerical values are less important than S_ISxxx() macros. Example:
   #define      S_ISDOOR(mode)  (((mode)&0xF000) == 0xd000)
0xd000 is (stat.S_IFSOCK + stat.S_IFIFO).

And how do you represent the file mode with enums? I don't think that
enum should be used in the stat module.


I would prefer a stat object with methods than having to calls
low-level functions. Something like:
   os.stat("document.txt").st_mode.is_reg()
versus
   stat.S_ISREG(os.stat("document.txt").st_mode)

The idea was discussed in http://bugs.python.org/issue11406 to solve a
real API design issue. How should os.scandir() return the "is a
directory" information with a portable API.

I'm not saying that stat.S_ISREG should go away. The two approaches
are complementary.

Victor

From ncoghlan at gmail.com  Fri Jun 21 03:06:24 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jun 2013 11:06:24 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
Message-ID: <CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>

On 21 June 2013 01:04, Thomas Wouters <thomas at python.org> wrote:
> If the .py file is going to be wrong or incomplete, why would we want to
> keep it -- or use it as fallback -- at all? If we're dead set on having a
> .py file instead of requiring it to be part of the interpreter (whichever
> that is, however it was built), it should be generated as part of the build
> process. Personally, I don't see the value in it; other implementations will
> need to do *something* special to use it anyway.

Because practicality beats purity. This "wrong" Python code has been
good enough for all Python version up until 3.4, it makes sense to
keep it as a fallback instead of throwing it away.

As Daniel notes, it also means PyPy can just have a small cffi module
that adds (or overwrites) the platform specific constants, instead of
having to reimplement the whole module.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Fri Jun 21 09:16:01 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 21 Jun 2013 17:16:01 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
Message-ID: <CAPTjJmoVA-sidq4B1jgxxqtiuU_sTaJ5fKKhN=gz-ZAvF-4Oaw@mail.gmail.com>

On Fri, Jun 21, 2013 at 11:06 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 21 June 2013 01:04, Thomas Wouters <thomas at python.org> wrote:
>> If the .py file is going to be wrong or incomplete, why would we want to
>> keep it -- or use it as fallback -- at all? If we're dead set on having a
>> .py file instead of requiring it to be part of the interpreter (whichever
>> that is, however it was built), it should be generated as part of the build
>> process. Personally, I don't see the value in it; other implementations will
>> need to do *something* special to use it anyway.
>
> Because practicality beats purity. This "wrong" Python code has been
> good enough for all Python version up until 3.4, it makes sense to
> keep it as a fallback instead of throwing it away.

How would you know if the Python you're running on has an incorrect bitflag?

If the "wrong" code is simply incomplete (it has the standard flags
but none of the contended ones), that would at least be safe - you'll
never get a false flag, but you might be unable to recognize the
platform-specific ones. And then the platform-specific modules would
always be adding, never overwriting, bitflags.

ChrisA

From victor.stinner at gmail.com  Fri Jun 21 09:25:35 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 21 Jun 2013 09:25:35 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
Message-ID: <CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>

2013/6/21 Nick Coghlan <ncoghlan at gmail.com>:
> Because practicality beats purity. This "wrong" Python code has been
> good enough for all Python version up until 3.4, it makes sense to
> keep it as a fallback instead of throwing it away.

How do you plan to handle the following case in Python?

"Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
0xe000 as S_IFWHT (whiteout), but Solaris defines it as
S_IFPORT (event port)."

We may keep the Python module if it is kept unchanged, but the Python
and C modules should have the same public API (the C module should not
have more features).

Or should play with "if sys.platform == ..."?

Victor

From hrvoje.niksic at avl.com  Fri Jun 21 10:42:47 2013
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Fri, 21 Jun 2013 08:42:47 -0000
Subject: [Python-Dev] PEP 409 and the stdlib
In-Reply-To: <knfbme$eou$1@ger.gmane.org>
References: <519A2149.3040903@stoneleaf.us>
	<CADiSq7e2z+FYdt0pyi0V+T9qHtvRoftO7zMFTENyq+Gjv++Tgw@mail.gmail.com>
	<519A2F37.6020406@stoneleaf.us>
	<20130520145004.1B1FA250BCF@webabinitio.net>
	<519A3DFC.5090705@stoneleaf.us> <519B1F89.4030003@avl.com>
	<knfbme$eou$1@ger.gmane.org>
Message-ID: <519B3DE3.8020508@avl.com>

On 05/21/2013 10:36 AM, Serhiy Storchaka wrote:
> 21.05.13 10:17, Hrvoje Niksic ???????(??):
>> On 05/20/2013 05:15 PM, Ethan Furman wrote:
>>> 1)  Do nothing and be happy I use 'raise ... from None' in my own
>>> libraries
>>>
>>> 2)  Change the wording of 'During handling of the above exception,
>>> another exception occurred' (no ideas as to what at
>>> the moment)
>>
>> The word "occurred" misleads one to think that, during handling of the
>> real exception, an unrelated and unintended exception occurred.  This is
>> not the case when the "raise" keyword is used.  In that case, the
>> exception was intentionally *converted* from one type to another.  For
>> the "raise" case a wording like the following might work better:
>>
>>      The above exception was converted to the following exception:
>>      ...
>>
>> That makes it clear that the conversion was explicit and (hopefully)
>> intentional, and that the latter exception supersedes the former.
>
> How do you distinguish intentional and unintentional exceptions?

By the use of the "raise" keyword.  Given the code:

try:
     x = bla['foo']
except KeyError:
     raise BusinessError(...)

...explicit raise is a giveaway that the exception replacement was quite 
intentional.

Hrvoje


From ncoghlan at gmail.com  Fri Jun 21 13:39:10 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jun 2013 21:39:10 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
Message-ID: <CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>

On 21 June 2013 17:25, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2013/6/21 Nick Coghlan <ncoghlan at gmail.com>:
>> Because practicality beats purity. This "wrong" Python code has been
>> good enough for all Python version up until 3.4, it makes sense to
>> keep it as a fallback instead of throwing it away.
>
> How do you plan to handle the following case in Python?
>
> "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
> 0xe000 as S_IFWHT (whiteout), but Solaris defines it as
> S_IFPORT (event port)."
>
> We may keep the Python module if it is kept unchanged, but the Python
> and C modules should have the same public API (the C module should not
> have more features).

I think it's OK to expose additional platform specific features in the
C version, and have them fail cleanly with the pure Python version
(rather than silently giving the wrong answer). What's not OK is for
the standard library to regress for other implementations just because
we added a C implementation for the benefit of CPython. That's exactly
the kind of thing PEP 399 says we *won't* do.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Fri Jun 21 13:45:50 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 21 Jun 2013 13:45:50 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
Message-ID: <20130621134550.1021345e@pitrou.net>

Le Fri, 21 Jun 2013 21:39:10 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :

> On 21 June 2013 17:25, Victor Stinner <victor.stinner at gmail.com>
> wrote:
> > 2013/6/21 Nick Coghlan <ncoghlan at gmail.com>:
> >> Because practicality beats purity. This "wrong" Python code has
> >> been good enough for all Python version up until 3.4, it makes
> >> sense to keep it as a fallback instead of throwing it away.
> >
> > How do you plan to handle the following case in Python?
> >
> > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD
> > defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as
> > S_IFPORT (event port)."
> >
> > We may keep the Python module if it is kept unchanged, but the
> > Python and C modules should have the same public API (the C module
> > should not have more features).
> 
> I think it's OK to expose additional platform specific features in the
> C version, and have them fail cleanly with the pure Python version
> (rather than silently giving the wrong answer).

PEP 399 says we don't do it:

"Acting as a drop-in replacement also dictates that no public API be
provided in accelerated code that does not exist in the pure Python
code. Without this requirement people could accidentally come to rely
on a detail in the accelerated code which is not made available to
other VMs that use the pure Python implementation."

> What's not OK is for
> the standard library to regress for other implementations just because
> we added a C implementation for the benefit of CPython. That's exactly
> the kind of thing PEP 399 says we *won't* do.

For me, PEP 399 should not be considered a requirement but a guideline.
stat.py is algorithmically trivial and I don't think it saves much work
for authors of third-party Python implementations.

Regards

Antoine.



From christian at python.org  Fri Jun 21 14:21:13 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 21 Jun 2013 14:21:13 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <20130621134550.1021345e@pitrou.net>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
Message-ID: <51C44539.70508@python.org>

Am 21.06.2013 13:45, schrieb Antoine Pitrou:
> For me, PEP 399 should not be considered a requirement but a guideline.
> stat.py is algorithmically trivial and I don't think it saves much work
> for authors of third-party Python implementations.

The module's content is rather boring. It's just a bunch of constants,
some wrapper functions for macros and one trivial function that turns
st_mode into "-rwxr-xr-x" string.

In my opinion stat module falls into the same line as the errno module.
The latter is also implemented in C in order to pull a bunch of
constants from header files.

I have yet another argument in favor of a C implementation. POSIX
standards won't gain any new S_IF* numeric constants for new files
types. Some file types such as mq, semaphore, shmem and typed memory
from POSIX real time extensions aren't testable with S_IF*, too. (But I
don't know any platform that implements them as files, though).
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html#tag_13_61_06

Christian


From eric at trueblade.com  Fri Jun 21 14:33:43 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 21 Jun 2013 08:33:43 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
Message-ID: <51C44827.7030908@trueblade.com>

On 6/21/2013 7:39 AM, Nick Coghlan wrote:
> On 21 June 2013 17:25, Victor Stinner <victor.stinner at gmail.com> wrote:
>> 2013/6/21 Nick Coghlan <ncoghlan at gmail.com>:
>>> Because practicality beats purity. This "wrong" Python code has been
>>> good enough for all Python version up until 3.4, it makes sense to
>>> keep it as a fallback instead of throwing it away.
>>
>> How do you plan to handle the following case in Python?
>>
>> "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
>> 0xe000 as S_IFWHT (whiteout), but Solaris defines it as
>> S_IFPORT (event port)."
>>
>> We may keep the Python module if it is kept unchanged, but the Python
>> and C modules should have the same public API (the C module should not
>> have more features).
> 
> I think it's OK to expose additional platform specific features in the
> C version, and have them fail cleanly with the pure Python version
> (rather than silently giving the wrong answer). What's not OK is for
> the standard library to regress for other implementations just because
> we added a C implementation for the benefit of CPython. That's exactly
> the kind of thing PEP 399 says we *won't* do.

I was just writing up something similar. But as always, Nick said it
better than me.

-- 
Eric.

From ani at aristanetworks.com  Fri Jun 21 07:18:27 2013
From: ani at aristanetworks.com (Ani Sinha)
Date: Thu, 20 Jun 2013 22:18:27 -0700
Subject: [Python-Dev] [python 2.7] a question about exporting a new method
	to socket object
Message-ID: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>

Hi python devs :

As a part of exploring an idea, I am trying to add a new method to the
python socket object. Here's what I have been trying to do (the patch may
be whitespace damaged so please bear with me) :


Index: Python-2.7/Modules/socketmodule.c
===================================================================
--- Python-2.7.orig/Modules/socketmodule.c
+++ Python-2.7/Modules/socketmodule.c
@@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a
 recv(buflen[, flags]) -- receive data\n\
 recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\
 recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\
+arecvfrom(buflen[, flags]) -- same as recvfrom \n\
 recvfrom_into(buffer[, nbytes, [, flags])\n\
   -- receive data and sender\'s address (into a buffer)\n\
 sendall(data[, flags]) -- send all data\n\
@@ -468,6 +469,13 @@ static PyTypeObject sock_type;
 #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <=
0.0)
 #endif

@@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc,
 \n\
 Like recv(buffersize, flags) but also return the sender's address info.");

+static PyObject *
+sock_arecvfrom(PySocketSockObject *s, PyObject *args)
+{
+    return sock_recvfrom(s,args);
+}
+
+PyDoc_STRVAR(arecvfrom_doc,
+"arecvfrom(buffersize[, flags]) -> (data, address info)\n\
+\n\
+ experimental stuff");

 /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */

@@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = {
                       recv_into_doc},
     {"recvfrom",          (PyCFunction)sock_recvfrom, METH_VARARGS,
                       recvfrom_doc},
+    {"arecvfrom",          (PyCFunction)sock_arecvfrom, METH_VARARGS,
+                      arecvfrom_doc},
     {"recvfrom_into",  (PyCFunction)sock_recvfrom_into, METH_VARARGS |
METH_KEYWORDS,
                       recvfrom_into_doc},
     {"send",              (PyCFunction)sock_send, METH_VARARGS,


When I compile this and run a simple script like the following that uses
arecvfrom() :

sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW )
sock.bind( ( intf, ETH_P_ALL ) )

(pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT )

I get this exception :

AttributeError: '_socketobject' object has no attribute 'arecvfrom'

So what am I doing wrong? How do I export this new socket method?

any help/pointer will be greatly appreciated.

cheers,
ani

-- 
Ani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130620/f2ed2bc3/attachment.html>

From guido at python.org  Fri Jun 21 17:07:04 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 21 Jun 2013 08:07:04 -0700
Subject: [Python-Dev] [python 2.7] a question about exporting a new
 method to socket object
In-Reply-To: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>
References: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>
Message-ID: <CAP7+vJKN_uKba60iBwjNHC2v+OmmKyDvQcfdUyt=iNReJmtbLQ@mail.gmail.com>

Technically this list isn't the right place, but you probably are
bitten by the duplication of functionality in Lib/socket.py. There's a
list of methods at the top of that file, _socketmethods.

But you really shouldn't be asking here -- this list is for changes to
Python, not using or hacking it. (Try StackOverflow for that.)

--Guido

On Thu, Jun 20, 2013 at 10:18 PM, Ani Sinha <ani at aristanetworks.com> wrote:
> Hi python devs :
>
> As a part of exploring an idea, I am trying to add a new method to the
> python socket object. Here's what I have been trying to do (the patch may be
> whitespace damaged so please bear with me) :
>
>
> Index: Python-2.7/Modules/socketmodule.c
> ===================================================================
> --- Python-2.7.orig/Modules/socketmodule.c
> +++ Python-2.7/Modules/socketmodule.c
> @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a
>  recv(buflen[, flags]) -- receive data\n\
>  recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\
>  recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\
> +arecvfrom(buflen[, flags]) -- same as recvfrom \n\
>  recvfrom_into(buffer[, nbytes, [, flags])\n\
>    -- receive data and sender\'s address (into a buffer)\n\
>  sendall(data[, flags]) -- send all data\n\
> @@ -468,6 +469,13 @@ static PyTypeObject sock_type;
>  #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <=
> 0.0)
>  #endif
>
> @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc,
>  \n\
>  Like recv(buffersize, flags) but also return the sender's address info.");
>
> +static PyObject *
> +sock_arecvfrom(PySocketSockObject *s, PyObject *args)
> +{
> +    return sock_recvfrom(s,args);
> +}
> +
> +PyDoc_STRVAR(arecvfrom_doc,
> +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\
> +\n\
> + experimental stuff");
>
>  /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */
>
> @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = {
>                        recv_into_doc},
>      {"recvfrom",          (PyCFunction)sock_recvfrom, METH_VARARGS,
>                        recvfrom_doc},
> +    {"arecvfrom",          (PyCFunction)sock_arecvfrom, METH_VARARGS,
> +                      arecvfrom_doc},
>      {"recvfrom_into",  (PyCFunction)sock_recvfrom_into, METH_VARARGS |
> METH_KEYWORDS,
>                        recvfrom_into_doc},
>      {"send",              (PyCFunction)sock_send, METH_VARARGS,
>
>
> When I compile this and run a simple script like the following that uses
> arecvfrom() :
>
> sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW )
> sock.bind( ( intf, ETH_P_ALL ) )
>
> (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT )
>
> I get this exception :
>
> AttributeError: '_socketobject' object has no attribute 'arecvfrom'
>
> So what am I doing wrong? How do I export this new socket method?
>
> any help/pointer will be greatly appreciated.
>
> cheers,
> ani
>
> --
> Ani
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From python.00 at klix.ch  Fri Jun 21 17:06:25 2013
From: python.00 at klix.ch (Gerald Klix)
Date: Fri, 21 Jun 2013 17:06:25 +0200
Subject: [Python-Dev] [python 2.7] a question about exporting a new
	method to socket object
In-Reply-To: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>
References: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>
Message-ID: <174A8F55-3226-49C7-A104-F2D62ABEB4B1@klix.ch>

Hi Ani,

just a simple question:
Are you sure, that you imported the modified socket module and not the original module from the distribution?

HTH,

Gerald


Am 21.06.2013 um 07:18 schrieb Ani Sinha <ani at aristanetworks.com>:

> Hi python devs :
> 
> As a part of exploring an idea, I am trying to add a new method to the
> python socket object. Here's what I have been trying to do (the patch may
> be whitespace damaged so please bear with me) :
> 
> 
> Index: Python-2.7/Modules/socketmodule.c
> ===================================================================
> --- Python-2.7.orig/Modules/socketmodule.c
> +++ Python-2.7/Modules/socketmodule.c
> @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a
> recv(buflen[, flags]) -- receive data\n\
> recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\
> recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\
> +arecvfrom(buflen[, flags]) -- same as recvfrom \n\
> recvfrom_into(buffer[, nbytes, [, flags])\n\
>   -- receive data and sender\'s address (into a buffer)\n\
> sendall(data[, flags]) -- send all data\n\
> @@ -468,6 +469,13 @@ static PyTypeObject sock_type;
> #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <=
> 0.0)
> #endif
> 
> @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc,
> \n\
> Like recv(buffersize, flags) but also return the sender's address info.");
> 
> +static PyObject *
> +sock_arecvfrom(PySocketSockObject *s, PyObject *args)
> +{
> +    return sock_recvfrom(s,args);
> +}
> +
> +PyDoc_STRVAR(arecvfrom_doc,
> +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\
> +\n\
> + experimental stuff");
> 
> /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */
> 
> @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = {
>                       recv_into_doc},
>     {"recvfrom",          (PyCFunction)sock_recvfrom, METH_VARARGS,
>                       recvfrom_doc},
> +    {"arecvfrom",          (PyCFunction)sock_arecvfrom, METH_VARARGS,
> +                      arecvfrom_doc},
>     {"recvfrom_into",  (PyCFunction)sock_recvfrom_into, METH_VARARGS |
> METH_KEYWORDS,
>                       recvfrom_into_doc},
>     {"send",              (PyCFunction)sock_send, METH_VARARGS,
> 
> 
> When I compile this and run a simple script like the following that uses
> arecvfrom() :
> 
> sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW )
> sock.bind( ( intf, ETH_P_ALL ) )
> 
> (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT )
> 
> I get this exception :
> 
> AttributeError: '_socketobject' object has no attribute 'arecvfrom'
> 
> So what am I doing wrong? How do I export this new socket method?
> 
> any help/pointer will be greatly appreciated.
> 
> cheers,
> ani
> 
> -- 
> Ani
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python.00%40klix.ch


From ncoghlan at gmail.com  Fri Jun 21 17:29:40 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Jun 2013 01:29:40 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <20130621134550.1021345e@pitrou.net>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
Message-ID: <CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>

On 21 June 2013 21:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Fri, 21 Jun 2013 21:39:10 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :
>> What's not OK is for
>> the standard library to regress for other implementations just because
>> we added a C implementation for the benefit of CPython. That's exactly
>> the kind of thing PEP 399 says we *won't* do.
>
> For me, PEP 399 should not be considered a requirement but a guideline.
> stat.py is algorithmically trivial and I don't think it saves much work
> for authors of third-party Python implementations.

So why not just replace the *broken* parts of stat.py and keep the
rest of it? Why make pointless work for the other implementations?

Basically, I want to hear from the Jython, PyPy and IronPython devs
that they're OK with us deleting Lib/stat.py. Hearing other CPython
devs say they're fine with it doesn't mean anything, since we're not
the ones that will have to do additional work as a result.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Fri Jun 21 17:38:26 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 21 Jun 2013 17:38:26 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
	<CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
Message-ID: <20130621173826.3610dd5b@pitrou.net>

Le Sat, 22 Jun 2013 01:29:40 +1000,
Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> On 21 June 2013 21:45, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > Le Fri, 21 Jun 2013 21:39:10 +1000,
> > Nick Coghlan <ncoghlan at gmail.com> a ?crit :
> >> What's not OK is for
> >> the standard library to regress for other implementations just
> >> because we added a C implementation for the benefit of CPython.
> >> That's exactly the kind of thing PEP 399 says we *won't* do.
> >
> > For me, PEP 399 should not be considered a requirement but a
> > guideline. stat.py is algorithmically trivial and I don't think it
> > saves much work for authors of third-party Python implementations.
> 
> So why not just replace the *broken* parts of stat.py and keep the
> rest of it? Why make pointless work for the other implementations?

I guess the answer is: because it's more work for us :-)

> Basically, I want to hear from the Jython, PyPy and IronPython devs
> that they're OK with us deleting Lib/stat.py. Hearing other CPython
> devs say they're fine with it doesn't mean anything, since we're not
> the ones that will have to do additional work as a result.

Yes, I agree with that.

Regards

Antoine.



From benjamin at python.org  Fri Jun 21 17:47:17 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 21 Jun 2013 08:47:17 -0700
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
	<CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
Message-ID: <CAPZV6o_FFQGbS0SLgvBhD9gyWjNPMqRSpVKu3tPeQPuZyS=7qQ@mail.gmail.com>

2013/6/21 Nick Coghlan <ncoghlan at gmail.com>:
> Basically, I want to hear from the Jython, PyPy and IronPython devs
> that they're OK with us deleting Lib/stat.py. Hearing other CPython
> devs say they're fine with it doesn't mean anything, since we're not
> the ones that will have to do additional work as a result.

<hat kind="pypy">
Kill it. I would argue having incorrect constants makes the
implementation incompatible with CPython anyway. This not that much
work (as long as there are tests that the constants exist at least) to
emulate.
</hat>


--
Regards,
Benjamin

From status at bugs.python.org  Fri Jun 21 18:07:31 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 21 Jun 2013 18:07:31 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130621160731.5407A56A50@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-06-14 - 2013-06-21)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4057 (+15)
  closed 25993 (+47)
  total  30050 (+62)

Open issues with patches: 1804 


Issues opened (42)
==================

#18113: Memory leak in curses.panel
http://bugs.python.org/issue18113  reopened by serhiy.storchaka

#18216: gettext doesn't check MO versions
http://bugs.python.org/issue18216  opened by jwilk

#18217: Deprecate and remove gettext.install
http://bugs.python.org/issue18217  opened by alex

#18219: csv.DictWriter is slow when writing files with large number of
http://bugs.python.org/issue18219  opened by mtraskin

#18220: In itertools.islice() make prototype like in help()
http://bugs.python.org/issue18220  opened by py.user

#18221: abspath strips trailing spaces on win32
http://bugs.python.org/issue18221  opened by Jeremy.Gray

#18222: os.path.abspath should accept multiple path parts and join the
http://bugs.python.org/issue18222  opened by ??ukasz.Balcerzak

#18224: pyvenv pydoc.py script causing AttributeErrors on Windows
http://bugs.python.org/issue18224  opened by peter at psantoro.net

#18226: IDLE Unit test for FormatParagrah.py
http://bugs.python.org/issue18226  opened by Todd.Rovito

#18227: Use Python memory allocators in external libraries like zlib o
http://bugs.python.org/issue18227  opened by haypo

#18228: AIX locale parsing failure
http://bugs.python.org/issue18228  opened by David.Edelsohn

#18229: attribute headers of http.server.BaseHTTPRequestHandler someti
http://bugs.python.org/issue18229  opened by joru

#18230: test_builtin fails/hangs when run after test_getopt
http://bugs.python.org/issue18230  opened by brett.cannon

#18231: What's new in Python should explain what's new in UCD
http://bugs.python.org/issue18231  opened by belopolsky

#18232: running a suite with no tests is not an error
http://bugs.python.org/issue18232  opened by rbcollins

#18233: SSLSocket.getpeercertchain()
http://bugs.python.org/issue18233  opened by christian.heimes

#18234: Unicodedata module should provide access to codepoint aliases
http://bugs.python.org/issue18234  opened by belopolsky

#18235: _sysconfigdata.py wrong on AIX installations
http://bugs.python.org/issue18235  opened by David.Edelsohn

#18236: int() and float() do not accept strings with trailing separato
http://bugs.python.org/issue18236  opened by belopolsky

#18237: unittest.assertRaisesRegex(p) example is wrong in docs
http://bugs.python.org/issue18237  opened by jtratner

#18240: hmac unnecessarily restricts input to "bytes"
http://bugs.python.org/issue18240  opened by jborgstrom

#18241: Add unique option to heapq functions
http://bugs.python.org/issue18241  opened by pitrou

#18242: IDLE should not be replacing warnings.formatwarning
http://bugs.python.org/issue18242  opened by brett.cannon

#18243: mktime_tz documentation out-of-date
http://bugs.python.org/issue18243  opened by jwilk

#18244: singledispatch: When virtual-inheriting ABCs at distinct point
http://bugs.python.org/issue18244  opened by ecatmur

#18246: tkinter.Text() add a newline to the content - bug?
http://bugs.python.org/issue18246  opened by Friedrich.Spee.von.Langenfeld

#18249: Incorrect and incomplete help docs for close() method
http://bugs.python.org/issue18249  opened by DaveA

#18254: Accessing attr dict at definition time of a class from a metac
http://bugs.python.org/issue18254  opened by archardlias

#18255: CPython setup.py problems
http://bugs.python.org/issue18255  opened by ronaldoussoren

#18257: Two copies of python-config
http://bugs.python.org/issue18257  opened by ronaldoussoren

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258  opened by zach.ware

#18260: configparser: TypeError occurs when handling errors in files w
http://bugs.python.org/issue18260  opened by Arfrever

#18262: ZipInfo.external_attr are not documented
http://bugs.python.org/issue18262  opened by techtonik

#18264: enum.IntEnum is not compatible with JSON serialisation
http://bugs.python.org/issue18264  opened by ncoghlan

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266  opened by zach.ware

#18268: ElementTree.fromstring non-deterministically gives unicode tex
http://bugs.python.org/issue18268  opened by Brendan.OConnor

#18269: Add new parameter format for converter function w/ position nu
http://bugs.python.org/issue18269  opened by techtonik

#18270: IDLE on OS X fails with Attribute Error if no initial shell an
http://bugs.python.org/issue18270  opened by dbackhaus

#18273: Simplify calling and discovery of json test package
http://bugs.python.org/issue18273  opened by zach.ware

#18275: Make isinstance() work with super type instances
http://bugs.python.org/issue18275  opened by brett.cannon

#18276: posixpath.c:_fd_converter() should use PyObject_AsFileDescript
http://bugs.python.org/issue18276  opened by christian.heimes

#18277: Queue is empty right after put from the same process/thread
http://bugs.python.org/issue18277  opened by shwouchk



Most recent 15 issues with no replies (15)
==========================================

#18277: Queue is empty right after put from the same process/thread
http://bugs.python.org/issue18277

#18273: Simplify calling and discovery of json test package
http://bugs.python.org/issue18273

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266

#18260: configparser: TypeError occurs when handling errors in files w
http://bugs.python.org/issue18260

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258

#18254: Accessing attr dict at definition time of a class from a metac
http://bugs.python.org/issue18254

#18246: tkinter.Text() add a newline to the content - bug?
http://bugs.python.org/issue18246

#18243: mktime_tz documentation out-of-date
http://bugs.python.org/issue18243

#18242: IDLE should not be replacing warnings.formatwarning
http://bugs.python.org/issue18242

#18237: unittest.assertRaisesRegex(p) example is wrong in docs
http://bugs.python.org/issue18237

#18236: int() and float() do not accept strings with trailing separato
http://bugs.python.org/issue18236

#18232: running a suite with no tests is not an error
http://bugs.python.org/issue18232

#18230: test_builtin fails/hangs when run after test_getopt
http://bugs.python.org/issue18230

#18229: attribute headers of http.server.BaseHTTPRequestHandler someti
http://bugs.python.org/issue18229

#18226: IDLE Unit test for FormatParagrah.py
http://bugs.python.org/issue18226



Most recent 15 issues waiting for review (15)
=============================================

#18273: Simplify calling and discovery of json test package
http://bugs.python.org/issue18273

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258

#18255: CPython setup.py problems
http://bugs.python.org/issue18255

#18244: singledispatch: When virtual-inheriting ABCs at distinct point
http://bugs.python.org/issue18244

#18240: hmac unnecessarily restricts input to "bytes"
http://bugs.python.org/issue18240

#18237: unittest.assertRaisesRegex(p) example is wrong in docs
http://bugs.python.org/issue18237

#18233: SSLSocket.getpeercertchain()
http://bugs.python.org/issue18233

#18222: os.path.abspath should accept multiple path parts and join the
http://bugs.python.org/issue18222

#18220: In itertools.islice() make prototype like in help()
http://bugs.python.org/issue18220

#18219: csv.DictWriter is slow when writing files with large number of
http://bugs.python.org/issue18219

#18214: Stop purging modules which are garbage collected before shutdo
http://bugs.python.org/issue18214

#18206: license url in site.py should always use X.Y.Z form of version
http://bugs.python.org/issue18206

#18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_
http://bugs.python.org/issue18203

#18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly
http://bugs.python.org/issue18191



Top 10 most discussed issues (10)
=================================

#18264: enum.IntEnum is not compatible with JSON serialisation
http://bugs.python.org/issue18264  16 msgs

#18081: test_logging failure in WarningsTest on buildbots
http://bugs.python.org/issue18081  15 msgs

#18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_
http://bugs.python.org/issue18203   9 msgs

#18103: Create a GUI test framework for Idle
http://bugs.python.org/issue18103   8 msgs

#16499: CLI option for isolated mode
http://bugs.python.org/issue16499   7 msgs

#18113: Memory leak in curses.panel
http://bugs.python.org/issue18113   7 msgs

#18153: python imaplib - error 'unexpected repsonse'
http://bugs.python.org/issue18153   7 msgs

#18162: Add index attribute to IndexError
http://bugs.python.org/issue18162   7 msgs

#18269: Add new parameter format for converter function w/ position nu
http://bugs.python.org/issue18269   7 msgs

#13483: Use VirtualAlloc to allocate memory arenas
http://bugs.python.org/issue13483   6 msgs



Issues closed (43)
==================

#14015: surrogateescape largely missing from documentation
http://bugs.python.org/issue14015  closed by python-dev

#15198: multiprocessing Pipe send of non-picklable objects doesn't rai
http://bugs.python.org/issue15198  closed by sbt

#16507: Patch selectmodule.c to support WSAPoll on Windows
http://bugs.python.org/issue16507  closed by sbt

#17018: Inconsistent behaviour of methods waiting for child process
http://bugs.python.org/issue17018  closed by sbt

#17177: Deprecate imp
http://bugs.python.org/issue17177  closed by brett.cannon

#17689: Fix test discovery for test_tarfile.py
http://bugs.python.org/issue17689  closed by zach.ware

#17907: Deprecate imp.new_module() in favour of types.ModuleType
http://bugs.python.org/issue17907  closed by brett.cannon

#17961: Use enum names as values in enum.Enum() functional API
http://bugs.python.org/issue17961  closed by ncoghlan

#18014: Problem compiling on Windows, VC++Express 2010
http://bugs.python.org/issue18014  closed by terry.reedy

#18057: Register NamespaceLoader with importlib.abc.Loader
http://bugs.python.org/issue18057  closed by brett.cannon

#18058: Define is_package for NamespaceLoader
http://bugs.python.org/issue18058  closed by brett.cannon

#18076: Implement importlib.util.decode_source()
http://bugs.python.org/issue18076  closed by brett.cannon

#18115: Use importlib.util.module_to_load in all loaders in importlib
http://bugs.python.org/issue18115  closed by brett.cannon

#18147: SSL: diagnostic functions to list loaded CA certs
http://bugs.python.org/issue18147  closed by christian.heimes

#18149: filecmp.cmp() incorrect results when previously compared file 
http://bugs.python.org/issue18149  closed by ned.deily

#18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n
http://bugs.python.org/issue18167  closed by serhiy.storchaka

#18192: Move imp.get_magic() to importlib
http://bugs.python.org/issue18192  closed by brett.cannon

#18193: Move imp.reload() to importlib
http://bugs.python.org/issue18193  closed by brett.cannon

#18194: Move imp.source_from_cache/cache_from_source to importlib
http://bugs.python.org/issue18194  closed by brett.cannon

#18197: insufficient error checking causes crash on windows
http://bugs.python.org/issue18197  closed by ncoghlan

#18202: Minor fixes for test_coding
http://bugs.python.org/issue18202  closed by serhiy.storchaka

#18207: OpenSSL may ignore seconds in notAfter
http://bugs.python.org/issue18207  closed by christian.heimes

#18218: In itertools.count() clarify the starting point
http://bugs.python.org/issue18218  closed by python-dev

#18223: Refactor test_tarfile
http://bugs.python.org/issue18223  closed by serhiy.storchaka

#18225: ctypes Structure data size is incorrect
http://bugs.python.org/issue18225  closed by alex

#18238: test_signal.py wait_helper hangs on AIX
http://bugs.python.org/issue18238  closed by haypo

#18239: In itertools docstring update arguments in count() example
http://bugs.python.org/issue18239  closed by python-dev

#18245: In itertools.groupby() make data plural
http://bugs.python.org/issue18245  closed by r.david.murray

#18247: Add Lib/test/data/ to .gitignore
http://bugs.python.org/issue18247  closed by python-dev

#18248: fficonfig.py.in wrong for AIX
http://bugs.python.org/issue18248  closed by python-dev

#18250: In itertools.repeat() object shadows object()
http://bugs.python.org/issue18250  closed by akuchling

#18251: test_subprocess_jy fails when Argument has embedded quote
http://bugs.python.org/issue18251  closed by r.david.murray

#18252: timeit makes code run faster?
http://bugs.python.org/issue18252  closed by r.david.murray

#18253: make standard library PEP8 compliant
http://bugs.python.org/issue18253  closed by r.david.murray

#18256: thread_pthread.h fixes for AIX
http://bugs.python.org/issue18256  closed by pitrou

#18259: Declare sethostname in socketmodule.c for AIX
http://bugs.python.org/issue18259  closed by christian.heimes

#18261: Confusing description in Minimal DOM implementation
http://bugs.python.org/issue18261  closed by ned.deily

#18263: python.man: no documentation for -b, -X
http://bugs.python.org/issue18263  closed by orsenthil

#18265: typedef lookupfunc defined by not used
http://bugs.python.org/issue18265  closed by David.Edelsohn

#18267: xmlrpc.client documentation multicall example missleading for 
http://bugs.python.org/issue18267  closed by python-dev

#18271: get_payload method returns bytes which cannot be decoded using
http://bugs.python.org/issue18271  closed by r.david.murray

#18272: In itertools recipes there is a typo in __builtins__
http://bugs.python.org/issue18272  closed by python-dev

#18274: python: not found
http://bugs.python.org/issue18274  closed by brett.cannon

From tjreedy at udel.edu  Fri Jun 21 18:28:57 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 21 Jun 2013 12:28:57 -0400
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <20130621134550.1021345e@pitrou.net>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
Message-ID: <kq1v0a$kp$1@ger.gmane.org>

On 6/21/2013 7:45 AM, Antoine Pitrou wrote:
> Le Fri, 21 Jun 2013 21:39:10 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a ?crit :

>> I think it's OK to expose additional platform specific features in the
>> C version, and have them fail cleanly with the pure Python version
>> (rather than silently giving the wrong answer).
>
> PEP 399 says we don't do it:
>
> "Acting as a drop-in replacement also dictates that no public API be
> provided in accelerated code that does not exist in the pure Python
> code. Without this requirement people could accidentally come to rely
> on a detail in the accelerated code which is not made available to
> other VMs that use the pure Python implementation."

Any C accelerator extensions should by documented as CPython extensions 
not necessarily available elsewhere. Or the doc should have, in this 
case, a separate _stat that says "In addition to the above, CPython's 
_stat, imported by stat, also provides ...


-- 
Terry Jan Reedy



From christian at python.org  Fri Jun 21 18:29:27 2013
From: christian at python.org (Christian Heimes)
Date: Fri, 21 Jun 2013 18:29:27 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPZV6o_FFQGbS0SLgvBhD9gyWjNPMqRSpVKu3tPeQPuZyS=7qQ@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
	<CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
	<CAPZV6o_FFQGbS0SLgvBhD9gyWjNPMqRSpVKu3tPeQPuZyS=7qQ@mail.gmail.com>
Message-ID: <51C47F67.2090808@python.org>

Am 21.06.2013 17:47, schrieb Benjamin Peterson:
> <hat kind="pypy">
> Kill it. I would argue having incorrect constants makes the
> implementation incompatible with CPython anyway. This not that much
> work (as long as there are tests that the constants exist at least) to
> emulate.
> </hat>

My patch adds extensive tests for all features of the stat module. The
test check the existence of attributes and the return value of all
functions. A missing or ill-defined attribute is going to raise an error.

But see for yourself: http://hg.python.org/cpython/rev/f8ff61f44aca

Christian


From solipsis at pitrou.net  Fri Jun 21 19:28:02 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 21 Jun 2013 19:28:02 +0200
Subject: [Python-Dev] PEP 445 delegate
Message-ID: <20130621192802.1f960f40@fsol>


Hello,

I've been appointed PEP 445 delegate by Nick and Guido. I would like to
know if there are still pending changes to the PEP. If not, I expect to
give it a review in the coming days or weeks, and then make a final
pronouncement (which will probably be positive anyway).

Regards

Antoine.



From ani at aristanetworks.com  Fri Jun 21 19:28:20 2013
From: ani at aristanetworks.com (Ani Sinha)
Date: Fri, 21 Jun 2013 10:28:20 -0700
Subject: [Python-Dev] [python 2.7] a question about exporting a new
 method to socket object
In-Reply-To: <CAP7+vJKN_uKba60iBwjNHC2v+OmmKyDvQcfdUyt=iNReJmtbLQ@mail.gmail.com>
References: <CAOxq_8MAqwGyiPhkVRzsLhryDebXmfgg24dxmBr+yPZYcpZxzA@mail.gmail.com>
	<CAP7+vJKN_uKba60iBwjNHC2v+OmmKyDvQcfdUyt=iNReJmtbLQ@mail.gmail.com>
Message-ID: <CAOxq_8Ng8W-AbRXmQdkCW0QCyJe+mn3GayHB2Tv-X2VVZS+XiA@mail.gmail.com>

Hi Guido :

Thanks a lot. That helped.

I will try StackOverflow next time.

cheers,
ani



On Fri, Jun 21, 2013 at 8:07 AM, Guido van Rossum <guido at python.org> wrote:

> Technically this list isn't the right place, but you probably are
> bitten by the duplication of functionality in Lib/socket.py. There's a
> list of methods at the top of that file, _socketmethods.
>
> But you really shouldn't be asking here -- this list is for changes to
> Python, not using or hacking it. (Try StackOverflow for that.)
>
> --Guido
>
> On Thu, Jun 20, 2013 at 10:18 PM, Ani Sinha <ani at aristanetworks.com>
> wrote:
> > Hi python devs :
> >
> > As a part of exploring an idea, I am trying to add a new method to the
> > python socket object. Here's what I have been trying to do (the patch
> may be
> > whitespace damaged so please bear with me) :
> >
> >
> > Index: Python-2.7/Modules/socketmodule.c
> > ===================================================================
> > --- Python-2.7.orig/Modules/socketmodule.c
> > +++ Python-2.7/Modules/socketmodule.c
> > @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a
> >  recv(buflen[, flags]) -- receive data\n\
> >  recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\
> >  recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\
> > +arecvfrom(buflen[, flags]) -- same as recvfrom \n\
> >  recvfrom_into(buffer[, nbytes, [, flags])\n\
> >    -- receive data and sender\'s address (into a buffer)\n\
> >  sendall(data[, flags]) -- send all data\n\
> > @@ -468,6 +469,13 @@ static PyTypeObject sock_type;
> >  #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout
> <=
> > 0.0)
> >  #endif
> >
> > @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc,
> >  \n\
> >  Like recv(buffersize, flags) but also return the sender's address
> info.");
> >
> > +static PyObject *
> > +sock_arecvfrom(PySocketSockObject *s, PyObject *args)
> > +{
> > +    return sock_recvfrom(s,args);
> > +}
> > +
> > +PyDoc_STRVAR(arecvfrom_doc,
> > +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\
> > +\n\
> > + experimental stuff");
> >
> >  /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */
> >
> > @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = {
> >                        recv_into_doc},
> >      {"recvfrom",          (PyCFunction)sock_recvfrom, METH_VARARGS,
> >                        recvfrom_doc},
> > +    {"arecvfrom",          (PyCFunction)sock_arecvfrom, METH_VARARGS,
> > +                      arecvfrom_doc},
> >      {"recvfrom_into",  (PyCFunction)sock_recvfrom_into, METH_VARARGS |
> > METH_KEYWORDS,
> >                        recvfrom_into_doc},
> >      {"send",              (PyCFunction)sock_send, METH_VARARGS,
> >
> >
> > When I compile this and run a simple script like the following that uses
> > arecvfrom() :
> >
> > sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW )
> > sock.bind( ( intf, ETH_P_ALL ) )
> >
> > (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT )
> >
> > I get this exception :
> >
> > AttributeError: '_socketobject' object has no attribute 'arecvfrom'
> >
> > So what am I doing wrong? How do I export this new socket method?
> >
> > any help/pointer will be greatly appreciated.
> >
> > cheers,
> > ani
> >
> > --
> > Ani
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
Ani
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130621/1c5b0c85/attachment.html>

From steve at pearwood.info  Fri Jun 21 20:33:45 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 22 Jun 2013 04:33:45 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
	<CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
Message-ID: <51C49C89.2060900@pearwood.info>

On 22/06/13 01:29, Nick Coghlan wrote:
> Basically, I want to hear from the Jython, PyPy and IronPython devs
> that they're OK with us deleting Lib/stat.py. Hearing other CPython
> devs say they're fine with it doesn't mean anything, since we're not
> the ones that will have to do additional work as a result.

It's not just established Python implementations. A pure-python standard library makes a good foundation for any future implementations.


-- 
Steven

From steve at pearwood.info  Fri Jun 21 21:20:24 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 22 Jun 2013 05:20:24 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
	<CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
Message-ID: <51C4A778.9070803@pearwood.info>

On 21/06/13 01:35, Benjamin Peterson wrote:
> 2013/6/20 Charles-Fran?ois Natali <cf.natali at gmail.com>:
>> 2013/6/20 Thomas Wouters <thomas at python.org>:
>>> If the .py file is going to be wrong or incomplete, why would we want to
>>> keep it -- or use it as fallback -- at all? If we're dead set on having a
>>> .py file instead of requiring it to be part of the interpreter (whichever
>>> that is, however it was built), it should be generated as part of the build
>>> process. Personally, I don't see the value in it; other implementations will
>>> need to do *something* special to use it anyway.

That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change.


>> That's exactly my rationale for pushing for removal.
>
> +1 to nixing it.

-1

Reading the Python source code is a very good way for beginner programmers to learn about things like this. Being able to read stat.py in Python makes a good, complementary source of information for those new to stat. Getting rid of stat.py means there's no source code to read at all, unless the user has built Python from source and kept the C source code. And even if they have, it's easy to forget how intimidating C can be to neophytes.

I'm with Nick on this one: PEP 399 already answers the question of what to do with stat.py. We keep it as a fallback, which guarantees that any Python implementation that uses the standard library is no worse off than what we have now. When available, we override the default constants with platform specific ones generated from whatever language is used by the implementation. The accuracy of those constants then becomes a matter of quality of implementation. A low quality implementation may take the lazy way out and just use the default constants, while a high quality implementation will not.

At the moment, the stat.py module contains ten small functions. Getting rid of the stat.py means that they have to be re-implemented in C/Java/RPython/etc. for every implementation. Even if the re-implementations are individually trivial, it's still a cost for no real gain. Keeping the pure Python implementation also lowers the bar for adding new functions in the future, and for documentation changes.



-- 
Steven

From ericsnowcurrently at gmail.com  Fri Jun 21 22:04:20 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 21 Jun 2013 14:04:20 -0600
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CADiSq7c+u6yiSCiS71Fpfg85My-86E2jDaNw8CUULtZ+GOaFjw@mail.gmail.com>
	<CAMpsgwb0SOxO-zdkkrG_YOZMAsyaKV36CddqxhytjqjsNsCoUw@mail.gmail.com>
	<CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
	<20130621134550.1021345e@pitrou.net>
	<CADiSq7d2_-D-MkqcKdjYDDBe3vcOxvXJ66_WAqvuZeirDa06uA@mail.gmail.com>
Message-ID: <CALFfu7AoJhgTPsTbDHm3r94ZoOYRQ3uGzqsEmwE1z7RqJX8NKw@mail.gmail.com>

On Fri, Jun 21, 2013 at 9:29 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> So why not just replace the *broken* parts of stat.py and keep the
> rest of it?

In some ways this sounds like yet another use case for what amounts to
properties on modules.  The sketchy contants that are causing concern
could raise NotImplementedError in stat.py, but the C version would
implement them.  Then PEP 399 remains satisfied and appropriately so.

-eric

From victor.stinner at gmail.com  Fri Jun 21 22:43:58 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 21 Jun 2013 22:43:58 +0200
Subject: [Python-Dev] PEP 445 delegate
In-Reply-To: <20130621192802.1f960f40@fsol>
References: <20130621192802.1f960f40@fsol>
Message-ID: <CAMpsgwZ1=i2LcphY=szg0bNZOXYznkgiedOHbn_tjPx=DQ1XYA@mail.gmail.com>

Hi,

2013/6/21 Antoine Pitrou <solipsis at pitrou.net>:
> I've been appointed PEP 445 delegate by Nick and Guido. I would like to
> know if there are still pending changes to the PEP.

Cool.

Hum, there is maybe something. In my pytracemalloc project, I added
another API to track usage of free lists:

PyAPI_FUNC(int) _PyFreeList_SetAllocators(
    void (*alloc) (PyObject *),
    void (*free) (PyObject *)
);
PyAPI_FUNC(void) _PyFreeList_Alloc(void *);
PyAPI_FUNC(void) _PyFreeList_Free(void *);

Then I'm using Py_TYPE(op)->tp_basicsize to get the size of the object.

I didn't add this API to the PEP because I'm not really convinced that
is useful. I wrote it becase Python 2 has unlimited free lists,
especially for the int type. In Python 3, the situation is different:
free lists have a limited size, and the "int" type of Python 3 is the
old "long" type, and int in Python 3 does not use a free list. (By the
way, it may provide better performances if we had a free list for
"small" ints.)

I expect better results with the new PyMem_RawMalloc() function and
the "Don't call malloc() directly anymore" section of the PEP than
with tracking free lists.


I'm testing Python 3.4 with all patches related to the PEP 445 (#3329,
#16742, #18203) and it works fine. You can also try my pytracemalloc
project with its "userdata" branch. I just found a bug in
pytracemalloc for allocation of zero bytes :-)

Victor

From gjcarneiro at gmail.com  Fri Jun 21 23:02:44 2013
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Fri, 21 Jun 2013 22:02:44 +0100
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C4A778.9070803@pearwood.info>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
	<CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
	<51C4A778.9070803@pearwood.info>
Message-ID: <CAO-CpELgyUi3-BbCoruqKVC3Gi6KY15OiDPt9NzODB8Lpg1OiQ@mail.gmail.com>

On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> On 21/06/13 01:35, Benjamin Peterson wrote:
>
>> 2013/6/20 Charles-Fran?ois Natali <cf.natali at gmail.com>:
>>
>>> 2013/6/20 Thomas Wouters <thomas at python.org>:
>>>
>>>> If the .py file is going to be wrong or incomplete, why would we want to
>>>> keep it -- or use it as fallback -- at all? If we're dead set on having
>>>> a
>>>> .py file instead of requiring it to be part of the interpreter
>>>> (whichever
>>>> that is, however it was built), it should be generated as part of the
>>>> build
>>>> process. Personally, I don't see the value in it; other implementations
>>>> will
>>>> need to do *something* special to use it anyway.
>>>>
>>>
> That's not correct. Other implementations can do exactly what CPython 3.3
> does, namely just use stat.py as given. Not all implementations necessarily
> care about multiple platforms where stat constants are likely to change.
>
>
>
>  That's exactly my rationale for pushing for removal.
>>>
>>
>> +1 to nixing it.
>>
>
> -1
>
> Reading the Python source code is a very good way for beginner programmers
> to learn about things like this.


On the other hand, it is counter-productive to learn about code that is
conceptually _wrong_.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130621/1202a356/attachment.html>

From cs at zip.com.au  Sat Jun 22 01:58:59 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 22 Jun 2013 09:58:59 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
References: <CADiSq7c+YfvbT3N+hA_GX-K8ua9Fak7JS=yU3P=UMqmb3cRqmQ@mail.gmail.com>
Message-ID: <20130621235859.GA89388@cskk.homeip.net>

On 21Jun2013 21:39, Nick Coghlan <ncoghlan at gmail.com> wrote:
| On 21 June 2013 17:25, Victor Stinner <victor.stinner at gmail.com> wrote:
| > How do you plan to handle the following case in Python?
| >
| > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines
| > 0xe000 as S_IFWHT (whiteout), but Solaris defines it as
| > S_IFPORT (event port)."
| >
| > We may keep the Python module if it is kept unchanged, but the Python
| > and C modules should have the same public API (the C module should not
| > have more features).
| 
| I think it's OK to expose additional platform specific features in the
| C version, and have them fail cleanly with the pure Python version
| (rather than silently giving the wrong answer). What's not OK is for
| the standard library to regress for other implementations just because
| we added a C implementation for the benefit of CPython. That's exactly
| the kind of thing PEP 399 says we *won't* do.

+1

I'm all for the C module exposing any and all of the S_* macros for
the local platform, and good with the python module (if used because
the C module isn't present, or conceivably is compiled out because
it is known broken on this platform) exposing only the portable
stuff.

At least you can detect "I don't know what to do" rather than
ploughing on mistakenly.
-- 
Cameron Simpson <cs at zip.com.au>

Ignorance is preferable to error; and he is less remote from the truth
who believes nothing, than he who believes what is wrong.
        - Thomas Jefferson

From stephen at xemacs.org  Sat Jun 22 02:01:53 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 22 Jun 2013 09:01:53 +0900
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAO-CpELgyUi3-BbCoruqKVC3Gi6KY15OiDPt9NzODB8Lpg1OiQ@mail.gmail.com>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
	<CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
	<51C4A778.9070803@pearwood.info>
	<CAO-CpELgyUi3-BbCoruqKVC3Gi6KY15OiDPt9NzODB8Lpg1OiQ@mail.gmail.com>
Message-ID: <87wqpn109a.fsf@uwakimon.sk.tsukuba.ac.jp>

Gustavo Carneiro writes:

 > On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano <steve at pearwood.info> wrote:

 >> -1 Reading the Python source code is a very good way for beginner
 >> programmers to learn about things like this.

 > On the other hand, it is counter-productive to learn about code
 > that is conceptually _wrong_.

It's hardly _conceptually_ wrong when it's simply a pure-Python
version that reflects *exactly* the implementation used by many OSes
written in C.  It's purely a quality of implementation issue in that
it fails to observe the DRY principle and is non-portable.  But if you
look at <sys/stat.h> on Mac OS X or Linux, you'll see that the visible
definitions are protected by a thicket of #ifdefs, and the actual
definitions may be drawn from other files #include'd there, and not
using the definitions visible in <sys/stat.h> at all.  Ie, on those
systems the OS implementation possesses exactly the same structure
that CPython would have with a stat.py + _stat module implementation.

If the presence of the _stat module is documented and the rationale
briefly explained in stat.py, I suppose you have the best of both
worlds for the novice programmer (at the expense of a test for the
presence of _stat at import time).

From cs at zip.com.au  Sat Jun 22 03:42:02 2013
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 22 Jun 2013 11:42:02 +1000
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
References: <CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
Message-ID: <20130622014202.GA43943@cskk.homeip.net>

On 20Jun2013 08:35, Benjamin Peterson <benjamin at python.org> wrote:
| 2013/6/20 Charles-Fran?ois Natali <cf.natali at gmail.com>:
| > 2013/6/20 Thomas Wouters <thomas at python.org>:
| >> If the .py file is going to be wrong or incomplete, why would we want to
| >> keep it -- or use it as fallback -- at all? If we're dead set on having a
| >> .py file instead of requiring it to be part of the interpreter (whichever
| >> that is, however it was built), it should be generated as part of the build
| >> process. Personally, I don't see the value in it; other implementations will
| >> need to do *something* special to use it anyway.
| >
| > That's exactly my rationale for pushing for removal.
| 
| +1 to nixing it.

-1 to nixing it.

I think there should be a pure python reference implementation.
If it exposes only the portable constants/macros or, better, has a
machine generated _section_ for the local platform macros, all to
the good; it would not lie.

- A huge amount of code only needs to care about the portable stuff
  (is this a dir, is this a regular file, is it neither).
  Missing local constants won't break such code.

- A reference implementation exposes algorithms and purpose in a
  common language (Python, of course; everyone caring about such
  an impl can already read Python:-).

- A reference implementation provides a base for other implementations
  to use outright, or to build on.

- A reference implementation provides something to test against for
  comparison of the common stuff.

- The implementation cost is low; the ref implementation already exists!

Getting rid of it seems mad. Pruning/fixing/adapting the nonportable
bits might be good. Christian Heimes' test patch would make that
much easier to identify.

So, -1 on removal of stat.py.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

I will not do it as a hack       I will not do it for my friends
I will not do it on a Mac        I will not write for Uncle Sam
I will not do it on weekends     I won't do ADA, Sam-I-Am
        - Gregory Bond <gnb at bby.com.au>

From arigo at tunes.org  Sat Jun 22 10:55:03 2013
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jun 2013 10:55:03 +0200
Subject: [Python-Dev] stat module in C -- what to do with stat.py?
In-Reply-To: <51C4A778.9070803@pearwood.info>
References: <51C2FE1E.8020801@python.org>
	<CAPFzasMZcY91w0TGSDTBy7Cxs0NvD5GuTK0hSpGsS2z2cGU8pg@mail.gmail.com>
	<51C303C6.8050609@python.org>
	<CADiSq7e5PmbFtbHy=n2N27fHMb-SAszkL6ZhwVw3NbHduQbJog@mail.gmail.com>
	<51C30E41.9090301@trueblade.com>
	<CAP1=2W6DeVyUrGQNocSqJ40ub5k=3gPzcortQix0Bwt47nurMA@mail.gmail.com>
	<CAPdQG2qooUbTA2Hq=0YNnD3WNpHMvVwveo8us3Y_Cc8Z5t6AvA@mail.gmail.com>
	<CAH_1eM28bid2jMx_Eh-2JEqC0eCEEd94O5CGmtCevVYJZfTWpA@mail.gmail.com>
	<CAPZV6o9rUCvMjaFf2_WDjsOg-hR4Endcrx01z6ezJFsxZSMLtg@mail.gmail.com>
	<51C4A778.9070803@pearwood.info>
Message-ID: <CAMSv6X0iEBJfUAPXW+b9qpb4R6-UKi6setG-G74h-8gCc=BKjg@mail.gmail.com>

Hi,

On Fri, Jun 21, 2013 at 9:20 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>>>> process. Personally, I don't see the value in it; other implementations
>>>> will need to do *something* special to use it anyway.
>
> That's not correct. Other implementations can do exactly what CPython 3.3
> does, namely just use stat.py as given. Not all implementations necessarily
> care about multiple platforms where stat constants are likely to change.

Thanks Steven.  That's PyPy's position: for now we really care only
about <insert some list of common platform there> where stat.py has
been obviously sufficient, as shown by CPython's long experience with
stat.py.


A bient?t,

Armin.

From benjamin at python.org  Sat Jun 22 20:17:12 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 22 Jun 2013 11:17:12 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
Message-ID: <CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>

Many people have raised concerns about this change, so I've now backed it out.

2013/6/18 Gregory P. Smith <greg at krypto.org>:
> Raymond -
>
> Why did you do this in the 2.7 branch?
>
> It hasn't been done in 3.3 or default and it isn't not the sort of change we
> make in a stable release branch without justification.  What issue was this
> for?  What problem were you solving?
>
> -gps
>
>
>
> On Mon, Jun 17, 2013 at 11:08 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>> On 06/17/2013 09:05 PM, Eli Bendersky wrote:
>>>
>>>
>>>
>>>
>>> On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka <storchaka at gmail.com
>>> <mailto:storchaka at gmail.com>> wrote:
>>>
>>>     14.06.13 11:46, Antoine Pitrou ???????(??):
>>>
>>>         On Fri, 14 Jun 2013 07:06:49 +0200 (CEST)
>>>         raymond.hettinger <python-checkins at python.org
>>> <mailto:python-checkins at python.org>> wrote:
>>>
>>>             http://hg.python.org/cpython/__rev/5accb0ac8bfb
>>> <http://hg.python.org/cpython/rev/5accb0ac8bfb>
>>>
>>>             changeset:   84116:5accb0ac8bfb
>>>                 Fix comment blocks.  Adjust blocksize to a power-of-two
>>> for better divmod computations.
>>>
>>>
>>>         Is there any rationale for changing the heuristic from "fits in a
>>> whole
>>>         number of cachelines" to "allows fast division by the blocksize"?
>>>
>>>         I personally would prefer if such changes were justified a bit
>>> more
>>>         than by a one-liner changeset message without any corresponding
>>> open
>>>         issue.
>>>
>>>
>>>     I share the doubts of Antoine and I was going to write the same
>>> comment. I thought there were good reasons for
>>>     previous code. What has changed?
>>>
>>>
>>> I agree it would be interesting to hear about the reasons for the change.
>>> Raymond?
>>
>>
>> Asking as a learner:  are such non-bugfix changes appropriate for the 2.7
>> line?
>>
>> --
>> ~Ethan~
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>



-- 
Regards,
Benjamin

From scott+python-dev at scottdial.com  Sat Jun 22 22:43:44 2013
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Sat, 22 Jun 2013 16:43:44 -0400
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
Message-ID: <51C60C80.1010207@scottdial.com>

On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
> Many people have raised concerns about this change, so I've now backed it out.

I think that change also goes with this change:

http://hg.python.org/cpython/rev/f1dc30a1be72

changeset 84248:f1dc30a1be72 2.7
Arrange structure to match the common access patterns.

     1.1 --- a/Modules/_collectionsmodule.c
     1.2 +++ b/Modules/_collectionsmodule.c
     1.3 @@ -48,8 +48,8 @@
     1.4
     1.5  typedef struct BLOCK {
     1.6      struct BLOCK *leftlink;
     1.7 +    PyObject *data[BLOCKLEN];
     1.8      struct BLOCK *rightlink;
     1.9 -    PyObject *data[BLOCKLEN];
    1.10  } block;
    1.11
    1.12  #define MAXFREEBLOCKS 10

, which seems like a strange micro-optimization, at best.

Based on that structure, it would seem that neither BLOCKLEN being 62
(previous value) nor 64 (the new value) make much sense. It would seem
best that sizeof(block) == 64, so BLOCKLEN should be (64 -
2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
this structure provides any meaningful performance improvement.

-- 
Scott Dial
scott at scottdial.com

From guido at python.org  Sat Jun 22 23:55:09 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 22 Jun 2013 14:55:09 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <51C60C80.1010207@scottdial.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
Message-ID: <CAP7+vJ+pMtmrgR4x20dBB78GDnM3L103pne5kEPWmz7wgnkyMw@mail.gmail.com>

On Sat, Jun 22, 2013 at 1:43 PM, Scott Dial
<scott+python-dev at scottdial.com> wrote:
> On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
>> Many people have raised concerns about this change, so I've now backed it out.
>
> I think that change also goes with this change:
>
> http://hg.python.org/cpython/rev/f1dc30a1be72
>
> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
>
>      1.1 --- a/Modules/_collectionsmodule.c
>      1.2 +++ b/Modules/_collectionsmodule.c
>      1.3 @@ -48,8 +48,8 @@
>      1.4
>      1.5  typedef struct BLOCK {
>      1.6      struct BLOCK *leftlink;
>      1.7 +    PyObject *data[BLOCKLEN];
>      1.8      struct BLOCK *rightlink;
>      1.9 -    PyObject *data[BLOCKLEN];
>     1.10  } block;
>     1.11
>     1.12  #define MAXFREEBLOCKS 10
>
> , which seems like a strange micro-optimization, at best.
>
> Based on that structure, it would seem that neither BLOCKLEN being 62
> (previous value) nor 64 (the new value) make much sense. It would seem
> best that sizeof(block) == 64, so BLOCKLEN should be (64 -
> 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
> this structure provides any meaningful performance improvement.

Actually the data buffer is an array of pointers too, so with the
original BLOCKLEN value of 62, sizeof(block) would be 64 times
sizeof(PyObject *). In the Py3 version of the source there's even a
comment explaining this right before the #define BLOCKLEN. Raymond,
can you explain?

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Sun Jun 23 14:40:30 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 23 Jun 2013 14:40:30 +0200
Subject: [Python-Dev] PEP 445 delegate
In-Reply-To: <20130621192802.1f960f40@fsol>
References: <20130621192802.1f960f40@fsol>
Message-ID: <CAMpsgwZPebOQJvFukt7RoqNM7cBns4bpFNCdBnnADU5rs8SjbQ@mail.gmail.com>

Hi,

2013/6/21 Antoine Pitrou <solipsis at pitrou.net>:
> I've been appointed PEP 445 delegate by Nick and Guido. I would like to
> know if there are still pending changes to the PEP. If not, I expect to
> give it a review in the coming days or weeks, and then make a final
> pronouncement (which will probably be positive anyway).

As you asked on IRC, I added a section to the PEP 445 explaining in
detail the redesign of debug checks on memory allocators:
http://www.python.org/dev/peps/pep-0445/#redesign-debug-checks-on-memory-allocators-as-hooks

The section show also how "hooks" work in the simple "call traces"
example. Hooks are just inserted between the public function (ex:
PyMem_Malloc) and the inner allocator function (ex:
_PyMem_RawMalloc()).

Minor nit: in practice, there is only one "debug hook" (reused on 3
allocators), so we may drop the trailing "s" from the name
"PyMem_SetupDebugHooks".

Diff on pep-0445.txt:
http://hg.python.org/peps/rev/c6199202522a

Victor

From fijall at gmail.com  Sun Jun 23 17:40:13 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 23 Jun 2013 17:40:13 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
Message-ID: <CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>

On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
>
>
>
> On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
> <python-checkins at python.org> wrote:
>>
>> http://hg.python.org/cpython/rev/dfead0696a71
>> changeset:   84224:dfead0696a71
>> branch:      3.3
>> parent:      84221:0113247f894b
>> user:        Senthil Kumaran <senthil at uthcode.com>
>> date:        Wed Jun 19 22:19:46 2013 -0500
>> summary:
>>   Add -b and -X options to python man page.
>> Patch contributed by Corey Brune.
>>
>> files:
>>   Misc/python.man |  22 ++++++++++++++++++----
>>   1 files changed, 18 insertions(+), 4 deletions(-)
>>
>>
>> diff --git a/Misc/python.man b/Misc/python.man
>> --- a/Misc/python.man
>> +++ b/Misc/python.man
>> @@ -11,6 +11,9 @@
>>  .B \-B
>>  ]
>>  [
>> +.B \-b
>> +]
>> +[
>>  .B \-d
>>  ]
>>  [
>> @@ -23,14 +26,14 @@
>>  .B \-i
>>  ]
>>  [
>> -.B \-m
>> +.B \-m
>>  .I module-name
>>  ]
>> -[
>> -.B \-q
>> -]
>>  .br
>>         [
>> +.B \-q
>> +]
>> +[
>>  .B \-O
>>  ]
>>  [
>> @@ -60,6 +63,10 @@
>>  .B \-x
>>  ]
>>  [
>> +[
>> +.B \-X
>> +.I option
>> +]
>>  .B \-?
>>  ]
>>  .br
>> @@ -105,6 +112,10 @@
>>  .I .py[co]
>>  files on import. See also PYTHONDONTWRITEBYTECODE.
>>  .TP
>> +.B \-b
>> +Issue warnings about str(bytes_instance), str(bytearray_instance)
>> +and comparing bytes/bytearray with str. (-bb: issue errors)
>> +.TP
>>  .BI "\-c " command
>>  Specify the command to execute (see next section).
>>  This terminates the option list (following options are passed as
>> @@ -243,6 +254,9 @@
>>  field matches the line number, where zero matches all line numbers and
>>  is thus equivalent to an omitted line number.
>>  .TP
>> +.BI "\-X " option
>> +Set implementation specific option.
>
>
> Should probably be "Set the implementation-specific option."

Is there anyone respecting this notation? (I know pypy does not, it
uses --jit and stuff)

>
>>
>> +.TP
>>  .B \-x
>>  Skip the first line of the source.  This is intended for a DOS
>>  specific hack only.  Warning: the line numbers in error messages will
>>
>> --
>> Repository URL: http://hg.python.org/cpython
>>
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> http://mail.python.org/mailman/listinfo/python-checkins
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From rdmurray at bitdance.com  Sun Jun 23 18:14:38 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 23 Jun 2013 12:14:38 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
	options to python man page.
In-Reply-To: <CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
Message-ID: <20130623161440.2587A250338@webabinitio.net>

On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
> > <python-checkins at python.org> wrote:
> >>  .TP
> >> +.BI "\-X " option
> >> +Set implementation specific option.
> >
> >
> > Should probably be "Set the implementation-specific option."
> 
> Is there anyone respecting this notation? (I know pypy does not, it
> uses --jit and stuff)

CPython does.  We introduced it for ourselves, it is up to other
implementations whether or not to use it, or use something else.

--David

From benjamin at python.org  Sun Jun 23 20:38:26 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 23 Jun 2013 11:38:26 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <51C60C80.1010207@scottdial.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
Message-ID: <CAPZV6o-diP84vUXCis7RPhvrvJtb1oGRGkG7twNOggZ+GuyjAg@mail.gmail.com>

I've backed this one out, too.

2013/6/22 Scott Dial <scott+python-dev at scottdial.com>:
> On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
>> Many people have raised concerns about this change, so I've now backed it out.
>
> I think that change also goes with this change:
>
> http://hg.python.org/cpython/rev/f1dc30a1be72
>
> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
>
>      1.1 --- a/Modules/_collectionsmodule.c
>      1.2 +++ b/Modules/_collectionsmodule.c
>      1.3 @@ -48,8 +48,8 @@
>      1.4
>      1.5  typedef struct BLOCK {
>      1.6      struct BLOCK *leftlink;
>      1.7 +    PyObject *data[BLOCKLEN];
>      1.8      struct BLOCK *rightlink;
>      1.9 -    PyObject *data[BLOCKLEN];
>     1.10  } block;
>     1.11
>     1.12  #define MAXFREEBLOCKS 10
>
> , which seems like a strange micro-optimization, at best.
>
> Based on that structure, it would seem that neither BLOCKLEN being 62
> (previous value) nor 64 (the new value) make much sense. It would seem
> best that sizeof(block) == 64, so BLOCKLEN should be (64 -
> 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
> this structure provides any meaningful performance improvement.
>
> --
> Scott Dial
> scott at scottdial.com



-- 
Regards,
Benjamin

From storchaka at gmail.com  Sun Jun 23 20:59:37 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 23 Jun 2013 21:59:37 +0300
Subject: [Python-Dev] A type of out-of-range exception for %c
Message-ID: <kq7giq$g4p$1@ger.gmane.org>

Currently %c formatting raises OverflowError if an argument is out of range.

 >>> '%c' % 1114112
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
OverflowError: %c arg not in range(0x110000)
 >>> '{:c}'.format(1114112)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
OverflowError: %c arg not in range(0x110000)

The same for PyUnicode_Format(), PyUnicode_FromFromat() and 
PyUnicode_FromFromatV() in C API.

But chr() raises ValueError.

 >>> chr(1114112)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ValueError: chr() arg not in range(0x110000)

And ValueError looks more suitable, because it is not really an overflow 
(limitation of the C language, or a C type). Can we change an exception 
type for %c formatting from OverflowError to ValueError?


From solipsis at pitrou.net  Sun Jun 23 21:07:03 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 23 Jun 2013 21:07:03 +0200
Subject: [Python-Dev] A type of out-of-range exception for %c
References: <kq7giq$g4p$1@ger.gmane.org>
Message-ID: <20130623210703.4e25fa32@fsol>

On Sun, 23 Jun 2013 21:59:37 +0300
Serhiy Storchaka <storchaka at gmail.com> wrote:
> Currently %c formatting raises OverflowError if an argument is out of range.
> 
>  >>> '%c' % 1114112
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> OverflowError: %c arg not in range(0x110000)
>  >>> '{:c}'.format(1114112)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> OverflowError: %c arg not in range(0x110000)
> 
> The same for PyUnicode_Format(), PyUnicode_FromFromat() and 
> PyUnicode_FromFromatV() in C API.
> 
> But chr() raises ValueError.
> 
>  >>> chr(1114112)
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> ValueError: chr() arg not in range(0x110000)
> 
> And ValueError looks more suitable, because it is not really an overflow 
> (limitation of the C language, or a C type). Can we change an exception 
> type for %c formatting from OverflowError to ValueError?

Well, it overflows the limited range of the "unicode character" type
(which is virtual in both Python and C, but still conceptually exists).

The most annoying thing here is that OverflowError doesn't subclass
ValueError.

Regards

Antoine.



From lukasz at langa.pl  Sun Jun 23 23:27:27 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 23 Jun 2013 23:27:27 +0200
Subject: [Python-Dev] A type of out-of-range exception for %c
In-Reply-To: <20130623210703.4e25fa32@fsol>
References: <kq7giq$g4p$1@ger.gmane.org> <20130623210703.4e25fa32@fsol>
Message-ID: <C9E15045-39C4-4545-A87A-CDF0B0E7A27B@langa.pl>

On 23 cze 2013, at 21:07, Antoine Pitrou <solipsis at pitrou.net> wrote:

> The most annoying thing here is that OverflowError doesn't subclass
> ValueError.


My intuition would rather make OverflowError subclass RuntimeError. Am I wrong?

That means I support Serhiy's point that %c would be less surprising raising a ValueError. Backwards compatibility is an unfortunate blocker here, I suppose.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130623/e74f04b5/attachment.html>

From raymond.hettinger at gmail.com  Mon Jun 24 02:16:32 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 23 Jun 2013 17:16:32 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
	blocksize to a power-of-two for better divmod
In-Reply-To: <51C60C80.1010207@scottdial.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
Message-ID: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>


On Jun 18, 2013, at 12:00 AM, Gregory P. Smith <greg at krypto.org> wrote:

> Why did you do this in the 2.7 branch?
> 
> It hasn't been done in 3.3 or default

I worked on the 2.7 branch first because that was the one I had loaded
and the one where I did timings and code disassembly.  I intended to
post it to 3.3 and 3.4 as well over the weekend.   

Ideally, it makes maintenance simpler for me if I keep the branches
the same as possible.  I viewed the one-line struct transposition,
comment correction, and one-line blocklen change as being somewhat 
innocuous non-algorithmic changes.  The struct change fixed an unintended
cache miss for left links and the blocksize change causes the
deque_index code to compile more efficiently (using a right-shift
and bitwise-and and rather than a measurably more expensive
division and modulo calculation).

I truly wasn't expecting the Spanish Inquisition :-)


On Jun 22, 2013, at 1:43 PM, Scott Dial <scott+python-dev at scottdial.com> wrote:

> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
> 
>     1.1 --- a/Modules/_collectionsmodule.c
>     1.2 +++ b/Modules/_collectionsmodule.c
>     1.3 @@ -48,8 +48,8 @@
>     1.4
>     1.5  typedef struct BLOCK {
>     1.6      struct BLOCK *leftlink;
>     1.7 +    PyObject *data[BLOCKLEN];
>     1.8      struct BLOCK *rightlink;
>     1.9 -    PyObject *data[BLOCKLEN];
>    1.10  } block;
>    1.11
>    1.12  #define MAXFREEBLOCKS 10
> 
> , which seems like a strange micro-optimization, at best.

Yes, this is a micro-optimization.  In working on implementing
deque slicing for 3.4, I restudied the block access patterns.   
On an appendleft(), popleft() or extendleft() operation, the left link is
accessed immediately before or after the leftmost entry in the data block. 
The old struct arrangement can cause an unnecessary cache miss
when jumping leftward.  This was something I overlooked when I
originally wrote the code almost a decade ago.

On Jun 23, 2013, at 11:38 AM, Benjamin Peterson <benjamin at python.org> wrote:

> I've backed this one out, too.

Really, you reverted my one-line change within 24 hours of it being posted?
I can't be on-line every day and sometimes it takes a little while to catch up
with python email.

On Jun 22, 2013, at 2:55 PM, Guido van Rossum <guido at python.org> wrote:

> Actually the data buffer is an array of pointers too, so with the
> original BLOCKLEN value of 62, sizeof(block) would be 64 times
> sizeof(PyObject *). In the Py3 version of the source there's even a
> comment explaining this right before the #define BLOCKLEN. Raymond,
> can you explain?

I also tried to fix that comment so it would stop emphasizing the blocklen
being a multiple of the cache line.   Also long as there is a reasonably long
data block, it matters less whether the data block size is an exact multiple
of the cache line length (62 or 64 words of data versus a typical 8 byte
cache line).

The data block size does matter elsewhere though.
The benefit of the having the data block being a power-of-two
is that the deque_index computation can use bits shifts
rather division and modulo calculations.  The benefit
of switching data block size from 62 to 64 was measurable
(a few percent) and observable in the disassembly of the code.

I experimented with one other ordering as well
(putting the data block first and the links afterwards).
That saved the scaled-index byte in the generated code
but produced no measureable speed-up.

In short, I believe the following should go in:

* The comment fix. (Incorrectly suggesting that a 64-bit 
   Py_ssize_t would overflow).  The revised comment
   is simpler, more general, and correct.

* Putting the leftlink before the data block in the structure.

The follow is up for debate:

Changing the BLOCKLEN from 62 to 64 is debatable.
It measureably improved deque_index without an
observable negative effect on the other operations.

Lastly, there was a change I just put in to Py 3.4 replacing
the memcpy() with a simple loop and replacing the
"deque->" references with local variables.  Besides
giving a small speed-up, it made the code more clear
and less at the mercy of various implementations
of memcpy().

Ideally, I would like 2.7 and 3.3 to replace their use of
memcpy() as well, but the flavor of this thread suggests
that is right out.


Raymond



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130623/9d70fa47/attachment-0001.html>

From scott+python-dev at scottdial.com  Mon Jun 24 03:52:52 2013
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Sun, 23 Jun 2013 21:52:52 -0400
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
Message-ID: <51C7A674.4010706@scottdial.com>

On 6/23/2013 8:16 PM, Raymond Hettinger wrote:
> Yes, this is a micro-optimization.  In working on implementing
> deque slicing for 3.4, I restudied the block access patterns.   
> On an appendleft(), popleft() or extendleft() operation, the left link is
> accessed immediately before or after the leftmost entry in the data block. 
> The old struct arrangement can cause an unnecessary cache miss
> when jumping leftward.  This was something I overlooked when I
> originally wrote the code almost a decade ago.

A decade ago, cache lines were 64 bytes, pointers were 4 bytes, and
allocations were 16 byte aligned, so there could never be a cache miss.
Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and
we still allocating on 16 byte alignment, so you have a 25% chance of a
cache miss now.

-- 
Scott Dial
scott at scottdial.com

From raymond.hettinger at gmail.com  Mon Jun 24 05:37:37 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 23 Jun 2013 20:37:37 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
	blocksize to a power-of-two for better divmod
In-Reply-To: <51C7A674.4010706@scottdial.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
Message-ID: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>


On Jun 23, 2013, at 6:52 PM, Scott Dial <scott+python-dev at scottdial.com> wrote:

> Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and
> we still allocating on 16 byte alignment, so you have a 25% chance of a
> cache miss now.

Honestly, I'm not sure what you're arguing for or against.

The struct should to be reordered so that the leftmost data element
and left link are positioned side-by-side, and the same goes for the
rightmost element and right link.

Whether the blocksize should change is debatable.
It causes an unfortunate odd-size at the end (not good for
the final cache line), but it does improve the speed of the 
deque_index() code.  The former seems to make no difference
in timings while the latter gives a measureable speed-up.

Unfortunately, I'm losing interest in this part of the deque work.
I've already invested substantial time reading papers
(such as http://www.akkadia.org/drepper/cpumemory.pdf
and http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html),
analyzing the code, reading disassembly, and making timings.
But it isn't worth all the second guessing (and what feels like sniping).
I've worked on this code for almost a decade.  As far as I can tell, none
of the participants in this thread has ever previously shown any interest
in the deque object.  It is discouraging to have a simple parameter 
change and struct reordering reverted.  This thread has offered zero 
support or encouragement for my work.  The python-dev 
social environment appears to be degrading over time.
I only have a few hours of development time each week
and now I'm wasting that time responding to these emails
(it may go with the territory, but it is a waste none-the-less).

If it is what you guys what, then leave the code as is
(with an incorrect comment, a blocklen that is unfavorable
to indexing, and a struct order that doesn't exploit
cache locality by following the natural access patterns
in the code).

I understand that the code for Py2.7 is sensitive and understand
if you all prefer to leave it untouched. 



Raymond


P.S. This has all arisen in context of my working on patch
for implementing slicing for deques.  In that context, the
code for deque_item() and deque_rotate() will become
more important than they were before.   Along the way,
I am re-examining all my existing code.




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130623/aeb06fef/attachment.html>

From alexander.belopolsky at gmail.com  Mon Jun 24 06:06:56 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 24 Jun 2013 00:06:56 -0400
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
	<85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
Message-ID: <CAP7h-xbDS0rg5BBigu16uO_7y157e_r-gfuEu9_xUGEnA5Co1Q@mail.gmail.com>

On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

> As far as I can tell, none
> of the participants in this thread has ever previously shown any interest
> in the deque object.  It is discouraging to have a simple parameter
> change and struct reordering reverted.  This thread has offered zero
> support or encouragement for my work.  The python-dev
> social environment appears to be degrading over time.
>

FWIW, I was surprised to see Raymond's patch reverted in what looked like a
knee-jerk reaction.  In my view Raymond through his contributions has
earned the presumption of validity for his commits.  A risk of introducing
regressions may ultimately outweigh the benefit of optimization and keeping
2.x and 3.x code in sync, but there is no need in rushing the decision.
 This patch could be reverted at any time before the 2.7.6 release which as
far as I know has not even been scheduled.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/a80f7dfc/attachment.html>

From greg at krypto.org  Mon Jun 24 06:11:58 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 23 Jun 2013 21:11:58 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
	<85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
Message-ID: <CAGE7PNLyv8H4E4TQtEiU5P1M2Q1Yz-F4mbTLwqHObXTTvrhU8g@mail.gmail.com>

Your work is great! I even agree with the changes on a coding best
practices level.

It's just that it belongs on the default (3.4) branch as it is an
enhancement, not a bug fix.

We don't do new things on release branches.

I agree that can be extremely frustrating at times, knowing that code won't
see the light of day for most users for years, but it is what has always
kept the Python project stable.

-gps
 On Jun 23, 2013 8:38 PM, "Raymond Hettinger" <raymond.hettinger at gmail.com>
wrote:

>
> On Jun 23, 2013, at 6:52 PM, Scott Dial <scott+python-dev at scottdial.com>
> wrote:
>
> Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and
> we still allocating on 16 byte alignment, so you have a 25% chance of a
> cache miss now.
>
>
> Honestly, I'm not sure what you're arguing for or against.
>
> The struct should to be reordered so that the leftmost data element
> and left link are positioned side-by-side, and the same goes for the
> rightmost element and right link.
>
> Whether the blocksize should change is debatable.
> It causes an unfortunate odd-size at the end (not good for
> the final cache line), but it does improve the speed of the
> deque_index() code.  The former seems to make no difference
> in timings while the latter gives a measureable speed-up.
>
> Unfortunately, I'm losing interest in this part of the deque work.
> I've already invested substantial time reading papers
> (such as http://www.akkadia.org/drepper/cpumemory.pdf
> and
> http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html
> ),
> analyzing the code, reading disassembly, and making timings.
> But it isn't worth all the second guessing (and what feels like sniping).
> I've worked on this code for almost a decade.  As far as I can tell, none
> of the participants in this thread has ever previously shown any interest
> in the deque object.  It is discouraging to have a simple parameter
> change and struct reordering reverted.  This thread has offered zero
> support or encouragement for my work.  The python-dev
> social environment appears to be degrading over time.
> I only have a few hours of development time each week
> and now I'm wasting that time responding to these emails
> (it may go with the territory, but it is a waste none-the-less).
>
> If it is what you guys what, then leave the code as is
> (with an incorrect comment, a blocklen that is unfavorable
> to indexing, and a struct order that doesn't exploit
> cache locality by following the natural access patterns
> in the code).
>
> I understand that the code for Py2.7 is sensitive and understand
> if you all prefer to leave it untouched.
>
>
>
> Raymond
>
>
> P.S. This has all arisen in context of my working on patch
> for implementing slicing for deques.  In that context, the
> code for deque_item() and deque_rotate() will become
> more important than they were before.   Along the way,
> I am re-examining all my existing code.
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130623/b7dcc8a7/attachment.html>

From benjamin at python.org  Mon Jun 24 08:03:14 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 23 Jun 2013 23:03:14 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAP7h-xbDS0rg5BBigu16uO_7y157e_r-gfuEu9_xUGEnA5Co1Q@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
	<85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
	<CAP7h-xbDS0rg5BBigu16uO_7y157e_r-gfuEu9_xUGEnA5Co1Q@mail.gmail.com>
Message-ID: <CAPZV6o9d_enmjroRCQfsyVJZ0yH+yyFhpmbRi_7t1B5LkWMFOA@mail.gmail.com>

2013/6/23 Alexander Belopolsky <alexander.belopolsky at gmail.com>:
>
> On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>> As far as I can tell, none
>> of the participants in this thread has ever previously shown any interest
>> in the deque object.  It is discouraging to have a simple parameter
>> change and struct reordering reverted.  This thread has offered zero
>> support or encouragement for my work.  The python-dev
>> social environment appears to be degrading over time.
>
>
> FWIW, I was surprised to see Raymond's patch reverted in what looked like a
> knee-jerk reaction.  In my view Raymond through his contributions has earned
> the presumption of validity for his commits.  A risk of introducing
> regressions may ultimately outweigh the benefit of optimization and keeping
> 2.x and 3.x code in sync, but there is no need in rushing the decision.
> This patch could be reverted at any time before the 2.7.6 release which as
> far as I know has not even been scheduled.

The first concern about 5accb0ac8bfb was raised more than a week ago
and more piled up over the course of the week. I knew Raymond wasn't
offline, since he continued to commit. Since we don't have mandatory
reviews before push (which more and more I think we should),
post-facto reviews on python-checkins need to be acknowledged. A
backout is not the end of the world; it takes seconds to reland once a
conclusion has been reached. This incident should not be construed to
diminish Raymond's long history of contribution or his technical
ability.


My second reversion (86d512e0ec66) was knee-jerk and shouldn't have happened.



--
Regards,
Benjamin

From ncoghlan at gmail.com  Mon Jun 24 08:23:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Jun 2013 16:23:18 +1000
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
	<85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
Message-ID: <CADiSq7d_sxHO9zKe1UdW1vSBBB+N3Cpazu96NXhM5Z21UKNeYQ@mail.gmail.com>

On 24 June 2013 13:37, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> But it isn't worth all the second guessing (and what feels like sniping).
> I've worked on this code for almost a decade.  As far as I can tell, none
> of the participants in this thread has ever previously shown any interest
> in the deque object.  It is discouraging to have a simple parameter
> change and struct reordering reverted.  This thread has offered zero
> support or encouragement for my work.  The python-dev
> social environment appears to be degrading over time.
> I only have a few hours of development time each week
> and now I'm wasting that time responding to these emails
> (it may go with the territory, but it is a waste none-the-less).

The problem wasn't the change in and of itself, it was:
- the fact it happened in a stable branch
- the fact there was no explanation, even a week after one was
requested (not even a quick "I do have the data to back up this
assertion, please leave the commit alone until I have time to provide
it").

The mailing lists, commit history, source code comments and files like
dict_notes.txt are an important part of understanding *why* various
parts of CPython are the way they are.

So, no, "responding to emails" is *not* a waste of anybody's time. It
helps to ensure important knowledge is transferred to more than the
person that was responsible for the commit. This is *especially*
important for areas where one person has substantially more knowledge
and experience than others (alleviating that kind of problem is one of
the reasons Brett, myself and others have been embarked on a
multi-year campaign to make the import system less arcane and
mysterious, and why I want to make the startup code more
comprehensible).

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Mon Jun 24 10:44:53 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 24 Jun 2013 11:44:53 +0300
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
Message-ID: <kq90u0$tek$1@ger.gmane.org>

24.06.13 03:16, Raymond Hettinger ???????(??):
> I truly wasn't expecting the Spanish Inquisition :-)

I only asked about the reasons. Previous reasons seemed reasonable to me 
and I wanted to know why a new code is better than the old. It will help 
me to use the best style in other places.

Thank you for your clarification.


From victor.stinner at gmail.com  Mon Jun 24 12:39:31 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 24 Jun 2013 12:39:31 +0200
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
Message-ID: <CAMpsgwbrH0a2QQA8ABTsQou3DNWmHf2iWHiNjwoBdLzOVu1Pxg@mail.gmail.com>

Hi Raymond,

Thank you for your long explanation, it is exactly what Antoine asked
for :-) I like micro-optimization even if I know that some other
developers only see such work as noise, not providing an real speed up
:-) So it was interesting to read your email!

I'm sorry that you was injured by the revert of your commits (even I
didn't do the revert myself, I agreed with Benjamin, and he's the
maintainer of Python 2.7). I hope that you understood why they were
rejected after reading this mail thread: the last Python 2.7 release
(which one exactly?) contains regressions, and changes on the 2.7
stable branch are sensitive. By experience, there is always someone in
the world relying to very specific implementation details. Cython,
PyPy CPyExt, ctypes, etc. are sensitive to such change. People should
not rely on such implementation details, but they do :-)

For this specific case, I "hope" that nobody relies on the exact BLOCK
structure (since it is a private structure). But it is just safer to
not change it that in a minor Python version. If your compare the risk
of introducing a regression versus the speedup, I guess that the risk
is higher.

As I wrote, I'm interested in micro-optimization, so please continue
your work! It is just safer to only modify the active development
branch (Python 3.4).

Victor


2013/6/24 Raymond Hettinger <raymond.hettinger at gmail.com>:
>
> On Jun 18, 2013, at 12:00 AM, Gregory P. Smith <greg at krypto.org> wrote:
>
> Why did you do this in the 2.7 branch?
>
> It hasn't been done in 3.3 or default
>
>
> I worked on the 2.7 branch first because that was the one I had loaded
> and the one where I did timings and code disassembly.  I intended to
> post it to 3.3 and 3.4 as well over the weekend.
>
> Ideally, it makes maintenance simpler for me if I keep the branches
> the same as possible.  I viewed the one-line struct transposition,
> comment correction, and one-line blocklen change as being somewhat
> innocuous non-algorithmic changes.  The struct change fixed an unintended
> cache miss for left links and the blocksize change causes the
> deque_index code to compile more efficiently (using a right-shift
> and bitwise-and and rather than a measurably more expensive
> division and modulo calculation).
>
> I truly wasn't expecting the Spanish Inquisition :-)
>
>
> On Jun 22, 2013, at 1:43 PM, Scott Dial <scott+python-dev at scottdial.com>
> wrote:
>
> changeset 84248:f1dc30a1be72 2.7
> Arrange structure to match the common access patterns.
>
>     1.1 --- a/Modules/_collectionsmodule.c
>     1.2 +++ b/Modules/_collectionsmodule.c
>     1.3 @@ -48,8 +48,8 @@
>     1.4
>     1.5  typedef struct BLOCK {
>     1.6      struct BLOCK *leftlink;
>     1.7 +    PyObject *data[BLOCKLEN];
>     1.8      struct BLOCK *rightlink;
>     1.9 -    PyObject *data[BLOCKLEN];
>    1.10  } block;
>    1.11
>    1.12  #define MAXFREEBLOCKS 10
>
> , which seems like a strange micro-optimization, at best.
>
>
> Yes, this is a micro-optimization.  In working on implementing
> deque slicing for 3.4, I restudied the block access patterns.
> On an appendleft(), popleft() or extendleft() operation, the left link is
> accessed immediately before or after the leftmost entry in the data block.
> The old struct arrangement can cause an unnecessary cache miss
> when jumping leftward.  This was something I overlooked when I
> originally wrote the code almost a decade ago.
>
> On Jun 23, 2013, at 11:38 AM, Benjamin Peterson <benjamin at python.org> wrote:
>
> I've backed this one out, too.
>
>
> Really, you reverted my one-line change within 24 hours of it being posted?
> I can't be on-line every day and sometimes it takes a little while to catch
> up
> with python email.
>
> On Jun 22, 2013, at 2:55 PM, Guido van Rossum <guido at python.org> wrote:
>
> Actually the data buffer is an array of pointers too, so with the
> original BLOCKLEN value of 62, sizeof(block) would be 64 times
> sizeof(PyObject *). In the Py3 version of the source there's even a
> comment explaining this right before the #define BLOCKLEN. Raymond,
> can you explain?
>
>
> I also tried to fix that comment so it would stop emphasizing the blocklen
> being a multiple of the cache line.   Also long as there is a reasonably
> long
> data block, it matters less whether the data block size is an exact multiple
> of the cache line length (62 or 64 words of data versus a typical 8 byte
> cache line).
>
> The data block size does matter elsewhere though.
> The benefit of the having the data block being a power-of-two
> is that the deque_index computation can use bits shifts
> rather division and modulo calculations.  The benefit
> of switching data block size from 62 to 64 was measurable
> (a few percent) and observable in the disassembly of the code.
>
> I experimented with one other ordering as well
> (putting the data block first and the links afterwards).
> That saved the scaled-index byte in the generated code
> but produced no measureable speed-up.
>
> In short, I believe the following should go in:
>
> * The comment fix. (Incorrectly suggesting that a 64-bit
>    Py_ssize_t would overflow).  The revised comment
>    is simpler, more general, and correct.
>
> * Putting the leftlink before the data block in the structure.
>
> The follow is up for debate:
>
> Changing the BLOCKLEN from 62 to 64 is debatable.
> It measureably improved deque_index without an
> observable negative effect on the other operations.
>
> Lastly, there was a change I just put in to Py 3.4 replacing
> the memcpy() with a simple loop and replacing the
> "deque->" references with local variables.  Besides
> giving a small speed-up, it made the code more clear
> and less at the mercy of various implementations
> of memcpy().
>
> Ideally, I would like 2.7 and 3.3 to replace their use of
> memcpy() as well, but the flavor of this thread suggests
> that is right out.
>
>
> Raymond
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From victor.stinner at gmail.com  Mon Jun 24 12:50:01 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 24 Jun 2013 12:50:01 +0200
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
Message-ID: <CAMpsgwaJawPStvREEfA0tfJUMEiv1oj2XBK4k0m-_fVXsY=wuA@mail.gmail.com>

2013/6/24 Raymond Hettinger <raymond.hettinger at gmail.com>:
> Lastly, there was a change I just put in to Py 3.4 replacing
> the memcpy() with a simple loop and replacing the
> "deque->" references with local variables.  Besides
> giving a small speed-up, it made the code more clear
> and less at the mercy of various implementations
> of memcpy().
>
> Ideally, I would like 2.7 and 3.3 to replace their use of
> memcpy() as well, but the flavor of this thread suggests
> that is right out.

The specific memcpy() function is usually highly optimized with
assembler code for each architecture. The GNU libc now does better: it
can choose the fastest version depending on the CPU version (MMX, SSE,
etc.) at runtime. If I understood correctly, the glibc contains
different version of memcpy, and the dynamic linker (ld.so) chooses
the version depending on the CPU.

GCC is also able to inline memcpy() when the size is known at compile
time. I also saw two code paths when the size is only known at
runtime: inline version for small size, and function call for larger
copy. Python has a Py_MEMCPY which implements exactly that, but only
for Visual Studio. I suppose that Visual Studio does not implement
this optimization. By the way, Py_MEMCPY() is only used in few places.

So it's surprising to read that a dummy loop is faster than
memcpy()... even if I already see this in my own micro-benchmarks :-)
Do you have an idea on how we can decide between the dummy loop and
memcpy()? Using a benchmark? Or can it be decided just by reading the
C code?

What is the policy for using Py_MEMCPY() vs memcpy()?

Victor

From victor.stinner at gmail.com  Mon Jun 24 13:07:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 24 Jun 2013 13:07:14 +0200
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
Message-ID: <CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>

2013/6/24 Raymond Hettinger <raymond.hettinger at gmail.com>:
> Changing the BLOCKLEN from 62 to 64 is debatable.
> It measureably improved deque_index without an
> observable negative effect on the other operations.

Out of curiosity, do you know (remember) how was the number 62 chosen?
Is it a compromise between memory usage and performances? 62 is
surprising because it is not a power of two :-)

Is it to just have 64 (2+62) pointers in the structure? (64 is a power
of 2) Does it help the memory allocator (reduce memory fragmentation)
to have a structure of 256 bytes (32-bit pointer) or 512 bytes (64-bit
pointer), which are power of 2, instead of 264 or 528 bytes?

It would be interesting to test pymalloc memory allocator on deque: I
plan to test Python globally with PyMem_Malloc using pymalloc.
pymalloc has a threshold of 512 bytes (and  528 bytes is greater than
512!). I suppose that the memory allocator is (much) more expensive
than integer divisions.

Would it be possible to change dynamically BLOCKLEN depending on the
total size of the deque? Python dict does something like that, but for
other reasons (reduce the number of hash collisions). According to
your comment, "Larger numbers  reduce the number of calls to the
memory allocator, give faster  indexing and rotation, and reduce the
link::data overhead ratio."

With dynamic BLOCKLEN, it would also be possible to reduce the memory
usage for small deque (less than 62 items).

Victor

From amk at amk.ca  Mon Jun 24 20:30:28 2013
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 24 Jun 2013 14:30:28 -0400
Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #18277:
 Document quirks of multiprocessing queue.
In-Reply-To: <3bfBmR47hXz7Lls@mail.python.org>
References: <3bfBmR47hXz7Lls@mail.python.org>
Message-ID: <20130624183028.GA12278@datl9makuchling.home>

On Mon, Jun 24, 2013 at 03:53:27PM +0200, richard.oudkerk wrote:
> http://hg.python.org/cpython/rev/8dcc4e017d42
> +   but should not cause any pratical difficulties -- you can always
                               ^^ practical

> +       infinitessimal delay before the queue's :meth:`~Queue.empty`
          ^^ infinitesimal (or just "short")

--amk

From solipsis at pitrou.net  Mon Jun 24 21:15:01 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 24 Jun 2013 21:15:01 +0200
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<51C7A674.4010706@scottdial.com>
	<85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com>
Message-ID: <20130624211501.373a88b2@fsol>

On Sun, 23 Jun 2013 20:37:37 -0700
Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> But it isn't worth all the second guessing (and what feels like sniping).
> I've worked on this code for almost a decade.  As far as I can tell, none
> of the participants in this thread has ever previously shown any interest
> in the deque object.

How do you measure "showing any interest"? I certainly use deques
regularly (thank you for implementing them) and I'm sure others do too.
I'm also interested in how the type is implemented and what the
tradeoffs are there. Otherwise I wouldn't have noticed your commit.

(if you remember, I was also interested enough to propose grafting
deque-like functionality on OrderedDict:
http://bugs.python.org/issue17100)

Regards

Antoine.



From shibturn at gmail.com  Mon Jun 24 21:35:23 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Mon, 24 Jun 2013 20:35:23 +0100
Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #18277:
 Document quirks of multiprocessing queue.
In-Reply-To: <20130624183028.GA12278@datl9makuchling.home>
References: <3bfBmR47hXz7Lls@mail.python.org>
	<20130624183028.GA12278@datl9makuchling.home>
Message-ID: <kqa71o$lqi$1@ger.gmane.org>

On 24/06/2013 7:30pm, A.M. Kuchling wrote:
> On Mon, Jun 24, 2013 at 03:53:27PM +0200, richard.oudkerk wrote:
>> http://hg.python.org/cpython/rev/8dcc4e017d42
>> +   but should not cause any pratical difficulties -- you can always
>                                 ^^ practical
>
>> +       infinitessimal delay before the queue's :meth:`~Queue.empty`
>            ^^ infinitesimal (or just "short")
>
> --amk
>

Thanks.  The first has already been pointed out.

-- 
Richard


From fijall at gmail.com  Mon Jun 24 22:14:46 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 24 Jun 2013 22:14:46 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <20130623161440.2587A250338@webabinitio.net>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
Message-ID: <CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>

On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
>> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
>> > <python-checkins at python.org> wrote:
>> >>  .TP
>> >> +.BI "\-X " option
>> >> +Set implementation specific option.
>> >
>> >
>> > Should probably be "Set the implementation-specific option."
>>
>> Is there anyone respecting this notation? (I know pypy does not, it
>> uses --jit and stuff)
>
> CPython does.  We introduced it for ourselves, it is up to other
> implementations whether or not to use it, or use something else.
>
> --David

you mean "CPython does not have any implementation-specific options"?
I would claim -O behavior should be implementation-specific since it's
nonsense in the optimizations sense, but other than that, it does not
seem that there is any -X options?

From alex.gaynor at gmail.com  Mon Jun 24 22:17:30 2013
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 24 Jun 2013 13:17:30 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
Message-ID: <CAFRnB2WU_6p=2TzBbo2v7QvuOzSyk1i-0FU2J4YXx+xmxRfj3Q@mail.gmail.com>

3.3 adds some -X options around faulthandler if I recall correctly.

Alex


On Mon, Jun 24, 2013 at 1:14 PM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
> >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
> >> > <python-checkins at python.org> wrote:
> >> >>  .TP
> >> >> +.BI "\-X " option
> >> >> +Set implementation specific option.
> >> >
> >> >
> >> > Should probably be "Set the implementation-specific option."
> >>
> >> Is there anyone respecting this notation? (I know pypy does not, it
> >> uses --jit and stuff)
> >
> > CPython does.  We introduced it for ourselves, it is up to other
> > implementations whether or not to use it, or use something else.
> >
> > --David
>
> you mean "CPython does not have any implementation-specific options"?
> I would claim -O behavior should be implementation-specific since it's
> nonsense in the optimizations sense, but other than that, it does not
> seem that there is any -X options?
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/4965f04d/attachment.html>

From rdmurray at bitdance.com  Mon Jun 24 22:43:15 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 24 Jun 2013 16:43:15 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
	options to python man page.
In-Reply-To: <CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
Message-ID: <20130624204316.028CB2500E3@webabinitio.net>

On Mon, 24 Jun 2013 22:14:46 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
> >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
> >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
> >> > <python-checkins at python.org> wrote:
> >> >>  .TP
> >> >> +.BI "\-X " option
> >> >> +Set implementation specific option.
> >> >
> >> >
> >> > Should probably be "Set the implementation-specific option."
> >>
> >> Is there anyone respecting this notation? (I know pypy does not, it
> >> uses --jit and stuff)
> >
> > CPython does.  We introduced it for ourselves, it is up to other
> > implementations whether or not to use it, or use something else.
> >
> > --David
> 
> you mean "CPython does not have any implementation-specific options"?
> I would claim -O behavior should be implementation-specific since it's
> nonsense in the optimizations sense, but other than that, it does not
> seem that there is any -X options?

There is one.  -X faulthandler.  I'm sure others would agree about
-O, but that long predates -X.

So, the idea is that -X *can* be used by other implementations for their
own purposes, but there is certainly no requirement that they do so.
Our promise is that anything CPython uses it for is something we don't
expect other implementations to support.

--David

From benjamin at python.org  Mon Jun 24 22:43:45 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 24 Jun 2013 13:43:45 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
Message-ID: <CAPZV6o_ZR1pyydwLhatWQPp1caW4=scJff3cPKCRwviBvqE=qw@mail.gmail.com>

2013/6/24 Maciej Fijalkowski <fijall at gmail.com>:
> On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com> wrote:
>> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
>>> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
>>> > <python-checkins at python.org> wrote:
>>> >>  .TP
>>> >> +.BI "\-X " option
>>> >> +Set implementation specific option.
>>> >
>>> >
>>> > Should probably be "Set the implementation-specific option."
>>>
>>> Is there anyone respecting this notation? (I know pypy does not, it
>>> uses --jit and stuff)
>>
>> CPython does.  We introduced it for ourselves, it is up to other
>> implementations whether or not to use it, or use something else.
>>
>> --David
>
> you mean "CPython does not have any implementation-specific options"?
> I would claim -O behavior should be implementation-specific since it's
> nonsense in the optimizations sense, but other than that, it does not
> seem that there is any -X options?

I wouldn't object to making that -Xno-docstrings or such, but the ship
sailed long ago on -O.


--
Regards,
Benjamin

From brett at python.org  Mon Jun 24 23:00:26 2013
From: brett at python.org (Brett Cannon)
Date: Mon, 24 Jun 2013 17:00:26 -0400
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <CAPZV6o_ZR1pyydwLhatWQPp1caW4=scJff3cPKCRwviBvqE=qw@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
	<CAPZV6o_ZR1pyydwLhatWQPp1caW4=scJff3cPKCRwviBvqE=qw@mail.gmail.com>
Message-ID: <CAP1=2W6s-uepwPBRtkxbM+nHEh46gHajUFxt_dH5CnJkwqJW5w@mail.gmail.com>

On Mon, Jun 24, 2013 at 4:43 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/6/24 Maciej Fijalkowski <fijall at gmail.com>:
> > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> >> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <
> fijall at gmail.com> wrote:
> >>> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org>
> wrote:
> >>> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
> >>> > <python-checkins at python.org> wrote:
> >>> >>  .TP
> >>> >> +.BI "\-X " option
> >>> >> +Set implementation specific option.
> >>> >
> >>> >
> >>> > Should probably be "Set the implementation-specific option."
> >>>
> >>> Is there anyone respecting this notation? (I know pypy does not, it
> >>> uses --jit and stuff)
> >>
> >> CPython does.  We introduced it for ourselves, it is up to other
> >> implementations whether or not to use it, or use something else.
> >>
> >> --David
> >
> > you mean "CPython does not have any implementation-specific options"?
> > I would claim -O behavior should be implementation-specific since it's
> > nonsense in the optimizations sense, but other than that, it does not
> > seem that there is any -X options?
>
> I wouldn't object to making that -Xno-docstrings or such, but the ship
> sailed long ago on -O.
>

Python 4 change! =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/9d95f3c0/attachment-0001.html>

From jeremy.kloth at gmail.com  Mon Jun 24 23:48:24 2013
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 24 Jun 2013 15:48:24 -0600
Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: Fix a
 compiler warning in tupleiter_setstate() on Windows x64
In-Reply-To: <3bfNzS6mryz7Lpx@mail.python.org>
References: <3bfNzS6mryz7Lpx@mail.python.org>
Message-ID: <CAGvrs3JYgw2dVSFuUThaAk7t+=0WT4OK=Eud7sLDgZ4gxEox8w@mail.gmail.com>

On Mon, Jun 24, 2013 at 3:33 PM, victor.stinner
<python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/6b4d279508a3
> changeset:   84325:6b4d279508a3
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Mon Jun 24 23:31:48 2013 +0200
> summary:
>   Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64
>
> files:
>   Objects/tupleobject.c |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
> --- a/Objects/tupleobject.c
> +++ b/Objects/tupleobject.c
> @@ -997,7 +997,7 @@
>  static PyObject *
>  tupleiter_setstate(tupleiterobject *it, PyObject *state)
>  {
> -    long index = PyLong_AsLong(state);
> +    Py_ssize_t index = PyLong_AsLong(state);

Actually, this will still lose data when state > MAX_INT (on Windows).
 It should be changed to PyLong_AsSssize_t(state) to ensure consistent
behavior on all platforms.

-- 
Jeremy Kloth

From victor.stinner at gmail.com  Tue Jun 25 00:00:41 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 25 Jun 2013 00:00:41 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: Fix a
 compiler warning in tupleiter_setstate() on Windows x64
In-Reply-To: <CAGvrs3JYgw2dVSFuUThaAk7t+=0WT4OK=Eud7sLDgZ4gxEox8w@mail.gmail.com>
References: <3bfNzS6mryz7Lpx@mail.python.org>
	<CAGvrs3JYgw2dVSFuUThaAk7t+=0WT4OK=Eud7sLDgZ4gxEox8w@mail.gmail.com>
Message-ID: <CAMpsgwae9+0CzvL2zrdOG_pfPM4NFRZzT6QK8+EKddLuZTsNVw@mail.gmail.com>

Ah yes correct, it should be better with the following commit:
http://hg.python.org/cpython/rev/3a393fc86b29

Victor

2013/6/24 Jeremy Kloth <jeremy.kloth at gmail.com>:
> On Mon, Jun 24, 2013 at 3:33 PM, victor.stinner
> <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/6b4d279508a3
>> changeset:   84325:6b4d279508a3
>> user:        Victor Stinner <victor.stinner at gmail.com>
>> date:        Mon Jun 24 23:31:48 2013 +0200
>> summary:
>>   Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64
>> ...
>>  static PyObject *
>>  tupleiter_setstate(tupleiterobject *it, PyObject *state)
>>  {
>> -    long index = PyLong_AsLong(state);
>> +    Py_ssize_t index = PyLong_AsLong(state);
>
> Actually, this will still lose data when state > MAX_INT (on Windows).
>  It should be changed to PyLong_AsSssize_t(state) to ensure consistent
> behavior on all platforms.

From jeremy.kloth at gmail.com  Tue Jun 25 00:08:01 2013
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 24 Jun 2013 16:08:01 -0600
Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: recv(),
 recvfrom(), send(), sendall() and sendto() methods
In-Reply-To: <3bfPJ14MyCz7LqX@mail.python.org>
References: <3bfPJ14MyCz7LqX@mail.python.org>
Message-ID: <CAGvrs3JDxc7trL_RCU9Rf7OB9zQLmf-Kh5K2W-d8Qm0nxnd+Ag@mail.gmail.com>

On Mon, Jun 24, 2013 at 3:48 PM, victor.stinner
<python-checkins at python.org> wrote:
> +#if defined(MS_WIN64) || defined(MS_WINDOWS)

This test could be reduced to simply `#ifdef MS_WINDOWS`.  See PC\pyconfig.h

From victor.stinner at gmail.com  Tue Jun 25 00:28:09 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 25 Jun 2013 00:28:09 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <20130624204316.028CB2500E3@webabinitio.net>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
	<20130624204316.028CB2500E3@webabinitio.net>
Message-ID: <CAMpsgwZdOHkqiaTwwsahFtQzufhwNGHUuiTXOqrb1gemZL4p-Q@mail.gmail.com>

2013/6/24 R. David Murray <rdmurray at bitdance.com>:
> There is one.  -X faulthandler.  I'm sure others would agree about
> -O, but that long predates -X.

FYI I didn't chose "-X" because it is specific to CPython, but just
because it becomes really hard to choose a random letter to add a new
option... I prefer long options, but Python only supports the long
option --help. IMO "python -X faulthandler" is more explicit than
"python -@" (or "python -f", or any other random letter).

If you didn't know, you can also write "python -X faulthandler=1", it
does also enable faulthandler... As "python -X faulthandler=0" and
"python -X faulthandler=False" :-) (PHP style!)

> So, the idea is that -X *can* be used by other implementations for their
> own purposes, but there is certainly no requirement that they do so.
> Our promise is that anything CPython uses it for is something we don't
> expect other implementations to support.

It would be nice to have faulthandler or something similar (dump
traces on a crash) in other Python implementations :-) faulthandler
implementation is very specific to CPython. It uses internal
structures and low-level OS features like signal handler and a
specific stack for its signal handler. It is better to have it
integrated directly into CPython. (But it is also available as a third
party module for older versions.)

Victor

From victor.stinner at gmail.com  Tue Jun 25 00:40:14 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 25 Jun 2013 00:40:14 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: recv(),
 recvfrom(), send(), sendall() and sendto() methods
In-Reply-To: <CAGvrs3JDxc7trL_RCU9Rf7OB9zQLmf-Kh5K2W-d8Qm0nxnd+Ag@mail.gmail.com>
References: <3bfPJ14MyCz7LqX@mail.python.org>
	<CAGvrs3JDxc7trL_RCU9Rf7OB9zQLmf-Kh5K2W-d8Qm0nxnd+Ag@mail.gmail.com>
Message-ID: <CAMpsgwbC40G46jRTA43u+kRRzgGKT-n7xSVVKWAaUc=9BPbg1A@mail.gmail.com>

Oh, I didn't know that."if defined(MS_WIN64) || defined(MS_WINDOWS)"
is a common pattern in the Python source code. I simplified the #ifdef
in many places:
http://hg.python.org/cpython/rev/dfc020b4b123

I also read that MS_WIN32 is always defined on Windows. "#ifdef
MS_WIN32" (used in many files, especially in ctypes) can probably be
replaced with "#ifdef MS_WINDOWS" for consistency. MS_WIN32 should
only be used to check if the API is Win64:

/* MSVC defines _WINxx to differentiate the windows platform types

   Note that for compatibility reasons _WIN32 is defined on Win32
   *and* on Win64. For the same reasons, in Python, MS_WIN32 is
   defined on Win32 *and* Win64. Win32 only code must therefore be
   guarded as follows:
       #if defined(MS_WIN32) && !defined(MS_WIN64)
   Some modules are disabled on Itanium processors, therefore we
   have MS_WINI64 set for those targets, otherwise MS_WINX64
*/

Victor

2013/6/25 Jeremy Kloth <jeremy.kloth at gmail.com>:
> On Mon, Jun 24, 2013 at 3:48 PM, victor.stinner
> <python-checkins at python.org> wrote:
>> +#if defined(MS_WIN64) || defined(MS_WINDOWS)
>
> This test could be reduced to simply `#ifdef MS_WINDOWS`.  See PC\pyconfig.h

From steve at pearwood.info  Tue Jun 25 01:01:15 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 25 Jun 2013 09:01:15 +1000
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
	options to python man page.
In-Reply-To: <CAMpsgwZdOHkqiaTwwsahFtQzufhwNGHUuiTXOqrb1gemZL4p-Q@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
	<20130624204316.028CB2500E3@webabinitio.net>
	<CAMpsgwZdOHkqiaTwwsahFtQzufhwNGHUuiTXOqrb1gemZL4p-Q@mail.gmail.com>
Message-ID: <20130624230115.GB10635@ando>

On Tue, Jun 25, 2013 at 12:28:09AM +0200, Victor Stinner wrote:
> 2013/6/24 R. David Murray <rdmurray at bitdance.com>:
> > There is one.  -X faulthandler.  I'm sure others would agree about
> > -O, but that long predates -X.
> 
> FYI I didn't chose "-X" because it is specific to CPython, but just
> because it becomes really hard to choose a random letter to add a new
> option... I prefer long options, but Python only supports the long
> option --help.

Is that a policy though? Couldn't Python support long options, or is it 
prohibited?


-- 
Steven

From ncoghlan at gmail.com  Tue Jun 25 01:43:40 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 25 Jun 2013 09:43:40 +1000
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <20130624230115.GB10635@ando>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
	<20130624204316.028CB2500E3@webabinitio.net>
	<CAMpsgwZdOHkqiaTwwsahFtQzufhwNGHUuiTXOqrb1gemZL4p-Q@mail.gmail.com>
	<20130624230115.GB10635@ando>
Message-ID: <CADiSq7ebTzUbJC5jMAKfqw5-L5w72ntGXOuRaTMT+ya4TDNvrA@mail.gmail.com>

On 25 Jun 2013 09:03, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> On Tue, Jun 25, 2013 at 12:28:09AM +0200, Victor Stinner wrote:
> > 2013/6/24 R. David Murray <rdmurray at bitdance.com>:
> > > There is one.  -X faulthandler.  I'm sure others would agree about
> > > -O, but that long predates -X.
> >
> > FYI I didn't chose "-X" because it is specific to CPython, but just
> > because it becomes really hard to choose a random letter to add a new
> > option... I prefer long options, but Python only supports the long
> > option --help.
>
> Is that a policy though? Couldn't Python support long options, or is it
> prohibited?

Just impractical. It will likely be more feasible to support them in a
maintainable way once PEP 432 is done and we can lean on the C API more
heavily while processing command line arguments.

Cheers,
Nick.

>
>
> --
> Steven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130625/06670c9d/attachment.html>

From raymond.hettinger at gmail.com  Tue Jun 25 04:47:51 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 24 Jun 2013 19:47:51 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
	blocksize to a power-of-two for better divmod
In-Reply-To: <CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
Message-ID: <A902328B-AE41-4733-951E-DF9380A86437@gmail.com>


On Jun 24, 2013, at 4:07 AM, Victor Stinner <victor.stinner at gmail.com> wrote:

> Out of curiosity, do you know (remember) how was the number 62 chosen?
> Is it a compromise between memory usage and performances? 62 is
> surprising because it is not a power of two :-)
> 
> Is it to just have 64 (2+62) pointers in the structure? 

Yes, the goal was to have the struct size be an exact multiple
of the cache line length (always a power-of-two, typically 64 bytes).
What was different then is that deques weren't indexable.
When indexing was added, the size of 62 became an 
unfavorable choice because it made the division and modulo 
calculation in deque_index() slower than for a power of two.

> For this specific case, I "hope" that nobody relies on the exact BLOCK
> structure (since it is a private structure).

I don't know what you're talking about.
This structure isn't externally visible.


Raymond









-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/9886627d/attachment.html>

From guido at python.org  Tue Jun 25 06:10:18 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 24 Jun 2013 21:10:18 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
	<A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
Message-ID: <CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>

On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
>
> On Jun 24, 2013, at 4:07 AM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
> Out of curiosity, do you know (remember) how was the number 62 chosen?
> Is it a compromise between memory usage and performances? 62 is
> surprising because it is not a power of two :-)
>
> Is it to just have 64 (2+62) pointers in the structure?
>
>
> Yes, the goal was to have the struct size be an exact multiple
> of the cache line length (always a power-of-two, typically 64 bytes).
> What was different then is that deques weren't indexable.
> When indexing was added, the size of 62 became an
> unfavorable choice because it made the division and modulo
> calculation in deque_index() slower than for a power of two.
>

A-ha! Finally an explanation of the change. It makes intuitive sense now. I
think the general feeling is that folks overreacted (perhaps confused by
your silence) and that the reversal will be rolled back. Benjamin?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/2d96ad22/attachment.html>

From greg at krypto.org  Tue Jun 25 07:07:30 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 24 Jun 2013 22:07:30 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
	<A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
	<CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
Message-ID: <CAGE7PN+ewaOt71+cG3nX8vhS6vw+6Y+L37ktwiiavvPYDH9Rsg@mail.gmail.com>

On Jun 24, 2013 9:11 PM, "Guido van Rossum" <guido at python.org> wrote:
>
> On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:
>>
>>
>>
>> On Jun 24, 2013, at 4:07 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:
>>
>>> Out of curiosity, do you know (remember) how was the number 62 chosen?
>>> Is it a compromise between memory usage and performances? 62 is
>>> surprising because it is not a power of two :-)
>>>
>>> Is it to just have 64 (2+62) pointers in the structure?
>>
>>
>> Yes, the goal was to have the struct size be an exact multiple
>> of the cache line length (always a power-of-two, typically 64 bytes).
>> What was different then is that deques weren't indexable.
>> When indexing was added, the size of 62 became an
>> unfavorable choice because it made the division and modulo
>> calculation in deque_index() slower than for a power of two.
>
>
> A-ha! Finally an explanation of the change. It makes intuitive sense now.
I think the general feeling is that folks overreacted (perhaps confused by
your silence) and that the reversal will be rolled back. Benjamin?

Seems likely to me.

>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130624/7e495412/attachment-0001.html>

From benjamin at python.org  Tue Jun 25 07:12:43 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 24 Jun 2013 22:12:43 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
	<A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
	<CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
Message-ID: <CAPZV6o9ZKwf-kZnxGk7cs0c3QHHNXCVqWpsA-xr1n89y3g4rmA@mail.gmail.com>

2013/6/24 Guido van Rossum <guido at python.org>:
> On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>>
>>
>> On Jun 24, 2013, at 4:07 AM, Victor Stinner <victor.stinner at gmail.com>
>> wrote:
>>
>> Out of curiosity, do you know (remember) how was the number 62 chosen?
>> Is it a compromise between memory usage and performances? 62 is
>> surprising because it is not a power of two :-)
>>
>> Is it to just have 64 (2+62) pointers in the structure?
>>
>>
>> Yes, the goal was to have the struct size be an exact multiple
>> of the cache line length (always a power-of-two, typically 64 bytes).
>> What was different then is that deques weren't indexable.
>> When indexing was added, the size of 62 became an
>> unfavorable choice because it made the division and modulo
>> calculation in deque_index() slower than for a power of two.
>
>
> A-ha! Finally an explanation of the change. It makes intuitive sense now. I
> think the general feeling is that folks overreacted (perhaps confused by
> your silence) and that the reversal will be rolled back. Benjamin?

Raymond, go ahead and reapply your change.


--
Regards,
Benjamin

From fijall at gmail.com  Tue Jun 25 11:12:23 2013
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 25 Jun 2013 11:12:23 +0200
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <20130624225629.72d9cf05@fsol>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
	<20130623161440.2587A250338@webabinitio.net>
	<CAK5idxRq8k12AYyxqKztTCgNgd8t_3N_pJHGLOwR7d3xkCuZ_g@mail.gmail.com>
	<20130624204316.028CB2500E3@webabinitio.net>
	<20130624225629.72d9cf05@fsol>
Message-ID: <CAK5idxTT5AGt0GasBi_MScrPkMoajUNvbHVJsPGm_=AVAnCnEQ@mail.gmail.com>

On Mon, Jun 24, 2013 at 10:56 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 24 Jun 2013 16:43:15 -0400
> "R. David Murray" <rdmurray at bitdance.com> wrote:
>> On Mon, 24 Jun 2013 22:14:46 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray <rdmurray at bitdance.com> wrote:
>> > > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> > >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon <brett at python.org> wrote:
>> > >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran
>> > >> > <python-checkins at python.org> wrote:
>> > >> >>  .TP
>> > >> >> +.BI "\-X " option
>> > >> >> +Set implementation specific option.
>> > >> >
>> > >> >
>> > >> > Should probably be "Set the implementation-specific option."
>> > >>
>> > >> Is there anyone respecting this notation? (I know pypy does not, it
>> > >> uses --jit and stuff)
>> > >
>> > > CPython does.  We introduced it for ourselves, it is up to other
>> > > implementations whether or not to use it, or use something else.
>> > >
>> > > --David
>> >
>> > you mean "CPython does not have any implementation-specific options"?
>> > I would claim -O behavior should be implementation-specific since it's
>> > nonsense in the optimizations sense, but other than that, it does not
>> > seem that there is any -X options?
>>
>> There is one.  -X faulthandler.  I'm sure others would agree about
>> -O, but that long predates -X.
>>
>> So, the idea is that -X *can* be used by other implementations for their
>> own purposes, but there is certainly no requirement that they do so.
>> Our promise is that anything CPython uses it for is something we don't
>> expect other implementations to support.
>
> Yes, basically -X is a private namespace for every implementation to
> use as it sees fit without fearing of conflicting with a future
> cross-implementation option.
>
> Regards
>
> Antoine.

I'm for one looking forward to the day where cpython conflicts on --jit ;-)

From brett at yvrsfo.ca  Tue Jun 25 17:45:00 2013
From: brett at yvrsfo.ca (Brett Cannon)
Date: Tue, 25 Jun 2013 11:45:00 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Issue #18081:
 Workaround "./python -m test_idle test_logging" failure
In-Reply-To: <3bfPyS4crdz7Lll@mail.python.org>
References: <3bfPyS4crdz7Lll@mail.python.org>
Message-ID: <CAP1=2W5q9gYUthGQJTq7wie_r-jPq3zgWoEjrqjAfo6KmWAe4w@mail.gmail.com>

This fix is still not correct as warnings.formatwarning should not be
overridden, only showwarning can be.
On Jun 24, 2013 6:18 PM, "victor.stinner" <python-checkins at python.org>
wrote:

> http://hg.python.org/cpython/rev/2a9e1eb3719c
> changeset:   84330:2a9e1eb3719c
> user:        Victor Stinner <victor.stinner at gmail.com>
> date:        Tue Jun 25 00:17:37 2013 +0200
> summary:
>   Issue #18081: Workaround "./python -m test_idle test_logging" failure
>
> "import idlelib" should not install hooks on the warning modules, hooks
> should
> only be installed when IDLE is started.
>
> files:
>   Lib/idlelib/PyShell.py |  5 +++--
>   Lib/idlelib/run.py     |  4 +++-
>   2 files changed, 6 insertions(+), 3 deletions(-)
>
>
> diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
> --- a/Lib/idlelib/PyShell.py
> +++ b/Lib/idlelib/PyShell.py
> @@ -61,7 +61,6 @@
>                                                lineno, line=line))
>          except OSError:
>              pass  ## file (probably __stderr__) is invalid, warning
> dropped.
> -    warnings.showwarning = idle_showwarning
>      def idle_formatwarning(message, category, filename, lineno,
> line=None):
>          """Format warnings the IDLE way"""
>          s = "\nWarning (from warnings module):\n"
> @@ -73,7 +72,6 @@
>              s += "    %s\n" % line
>          s += "%s: %s\n>>> " % (category.__name__, message)
>          return s
> -    warnings.formatwarning = idle_formatwarning
>
>  def extended_linecache_checkcache(filename=None,
>                                    orig_checkcache=linecache.checkcache):
> @@ -1425,6 +1423,9 @@
>  def main():
>      global flist, root, use_subprocess
>
> +    warnings.showwarning = idle_showwarning
> +    warnings.formatwarning = idle_formatwarning
> +
>      use_subprocess = True
>      enable_shell = False
>      enable_edit = False
> diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py
> --- a/Lib/idlelib/run.py
> +++ b/Lib/idlelib/run.py
> @@ -40,7 +40,6 @@
>              s += "    %s\n" % line
>          s += "%s: %s\n" % (category.__name__, message)
>          return s
> -    warnings.formatwarning = idle_formatwarning_subproc
>
>
>  tcl = tkinter.Tcl()
> @@ -82,6 +81,9 @@
>      global exit_now
>      global quitting
>      global no_exitfunc
> +
> +    warnings.formatwarning = idle_formatwarning_subproc
> +
>      no_exitfunc = del_exitfunc
>      #time.sleep(15) # test subprocess not responding
>      try:
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130625/28f21240/attachment.html>

From raymond.hettinger at gmail.com  Tue Jun 25 19:59:33 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 25 Jun 2013 10:59:33 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
	blocksize to a power-of-two for better divmod
In-Reply-To: <CAPZV6o9ZKwf-kZnxGk7cs0c3QHHNXCVqWpsA-xr1n89y3g4rmA@mail.gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
	<A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
	<CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
	<CAPZV6o9ZKwf-kZnxGk7cs0c3QHHNXCVqWpsA-xr1n89y3g4rmA@mail.gmail.com>
Message-ID: <70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com>


On Jun 24, 2013, at 10:12 PM, Benjamin Peterson <benjamin at python.org> wrote:

> Raymond, go ahead and reapply your change.

If you don't mind, I think you should be the one to undo your own reversions.

Thank you,


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130625/5ab07ae1/attachment.html>

From jdhardy at gmail.com  Tue Jun 25 20:05:15 2013
From: jdhardy at gmail.com (Jeff Hardy)
Date: Tue, 25 Jun 2013 11:05:15 -0700
Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X
 options to python man page.
In-Reply-To: <CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
References: <3bbSw10tm2z7Ljf@mail.python.org>
	<CAP1=2W6jxWzd7kP6f55LsLWjZoh-_r2_+3=ZD=BGnbQJiOwB=g@mail.gmail.com>
	<CAK5idxSt053PxaOwDHm9VH5M-NVnp0QbaMU9es+jDGwRoeV1Fw@mail.gmail.com>
Message-ID: <CAF7AXFHGv7j90zcrzx-MjGMMzeB9sqV_EPn8K57AhwCH_Xbu9w@mail.gmail.com>

On Sun, Jun 23, 2013 at 8:40 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> >> +.BI "\-X " option
> >> +Set implementation specific option.
> >
> >
> > Should probably be "Set the implementation-specific option."
>
> Is there anyone respecting this notation? (I know pypy does not, it
> uses --jit and stuff)
>
> IronPython does, for -X:Frames and a few others.

- Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130625/23486caa/attachment.html>

From benjamin at python.org  Tue Jun 25 20:32:44 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 25 Jun 2013 11:32:44 -0700
Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust
 blocksize to a power-of-two for better divmod
In-Reply-To: <70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com>
References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol>
	<kpf6u5$826$1@ger.gmane.org>
	<CAF-Rda_DksHmRmu0tcb-eE0iuDtQsOkjSVSjVq4BLV-zHpDbMQ@mail.gmail.com>
	<51BFF956.7040403@stoneleaf.us>
	<CAGE7PNLzTSAbb9WF7ehSjDdF1OH+yvCBv24UzrUgO+YryBxoJw@mail.gmail.com>
	<CAPZV6o_SMn3OEGJV0r_B=jnTPDbzf+nKjEaGUm6rEcjjkBH0Ow@mail.gmail.com>
	<51C60C80.1010207@scottdial.com>
	<2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com>
	<CAMpsgwZtB=vEV4p0Dw9uJTXDrBg3sRiGviv4YJJZaQpC8ZHEmw@mail.gmail.com>
	<A902328B-AE41-4733-951E-DF9380A86437@gmail.com>
	<CAP7+vJLuS_h2LVW+Jo6iUOTEqYs7cVb=uBge4RJ2iy+E3wfefA@mail.gmail.com>
	<CAPZV6o9ZKwf-kZnxGk7cs0c3QHHNXCVqWpsA-xr1n89y3g4rmA@mail.gmail.com>
	<70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com>
Message-ID: <CAPZV6o_LTU_LXv6xCMT6SBKvFCojT1bK3hZDDCYV51b5fOa5Kw@mail.gmail.com>

2013/6/25 Raymond Hettinger <raymond.hettinger at gmail.com>:
>
> On Jun 24, 2013, at 10:12 PM, Benjamin Peterson <benjamin at python.org> wrote:
>
> Raymond, go ahead and reapply your change.
>
>
> If you don't mind, I think you should be the one to undo your own
> reversions.

Done.


--
Regards,
Benjamin

From vitaly.murashev at gmail.com  Tue Jun 25 22:11:04 2013
From: vitaly.murashev at gmail.com (Vitaly Murashev)
Date: Wed, 26 Jun 2013 00:11:04 +0400
Subject: [Python-Dev] Relative path in co_filename for zipped modules
Message-ID: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>

Dear Python developers community,

Recently I found out that it not possible to debug python code if it
is a part of zip-module.
Python version being used is 3.3.0

Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to
start debugging and give the following warning:
---
pydev debugger: CRITICAL WARNING: This version of python seems to be
incorrectly compiled (internal generated filenames are not absolute)
pydev debugger: The debugger may still function, but it will work
slower and may miss breakpoints.
---
So I started my own investigation of this issue and results are the following.

At first I took traditional python debugger 'pdb' to analyze how it
behaves during debugging of zipped module.
'pdb' showed me some backtaces and filename part for stack entries
looks malformed.
I expected something like
'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py'
but realy it looks like 'full-path-to-current-dir/subdir/test_module.py'

Source code in pdb.py and bdb.py (which one are a part of python
stdlib) gave me the answer why it happens.

The root cause are inside Bdb.format_stack_entry() + Bdb.canonic()

Please take a look at the following line inside 'format_stack_entry' method:

    filename = self.canonic(frame.f_code.co_filename)

For zipped module variable 'frame.f_code.co_filename' holds _relative_
file path started from the root of zip archive like
'subdir/test_module.py'
And as result Bdb.canonic() method gives what we have -
'full-path-to-current-dir/subdir/test_module.py'
---
So my final question.
Could anyone confirm that it is a bug in python core subsystem which
one is responsible for loading zipped modules,
or something is wrong with my zipped module ?

If it is a bug could you please submit it into official python bugtracker ?
I never did it before and afraid to do something wrong.

Thanks,
Vitaly Murashev

From rdmurray at bitdance.com  Tue Jun 25 22:19:57 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 25 Jun 2013 16:19:57 -0400
Subject: [Python-Dev] Relative path in co_filename for zipped modules
In-Reply-To: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>
References: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>
Message-ID: <20130625201958.29E8B250B3F@webabinitio.net>

Please file this as a bug report on bugs.python.org so that it doesn't
get lost.

See also http://bugs.python.org/issue13328 (which looks like a different bug
but could also be causing you problems).

--David

On Wed, 26 Jun 2013 00:11:04 +0400, Vitaly Murashev <vitaly.murashev at gmail.com> wrote:
> Dear Python developers community,
> 
> Recently I found out that it not possible to debug python code if it
> is a part of zip-module.
> Python version being used is 3.3.0
> 
> Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to
> start debugging and give the following warning:
> ---
> pydev debugger: CRITICAL WARNING: This version of python seems to be
> incorrectly compiled (internal generated filenames are not absolute)
> pydev debugger: The debugger may still function, but it will work
> slower and may miss breakpoints.
> ---
> So I started my own investigation of this issue and results are the following.
> 
> At first I took traditional python debugger 'pdb' to analyze how it
> behaves during debugging of zipped module.
> 'pdb' showed me some backtaces and filename part for stack entries
> looks malformed.
> I expected something like
> 'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py'
> but realy it looks like 'full-path-to-current-dir/subdir/test_module.py'
> 
> Source code in pdb.py and bdb.py (which one are a part of python
> stdlib) gave me the answer why it happens.
> 
> The root cause are inside Bdb.format_stack_entry() + Bdb.canonic()
> 
> Please take a look at the following line inside 'format_stack_entry' method:
> 
>     filename = self.canonic(frame.f_code.co_filename)
> 
> For zipped module variable 'frame.f_code.co_filename' holds _relative_
> file path started from the root of zip archive like
> 'subdir/test_module.py'
> And as result Bdb.canonic() method gives what we have -
> 'full-path-to-current-dir/subdir/test_module.py'
> ---
> So my final question.
> Could anyone confirm that it is a bug in python core subsystem which
> one is responsible for loading zipped modules,
> or something is wrong with my zipped module ?
> 
> If it is a bug could you please submit it into official python bugtracker ?
> I never did it before and afraid to do something wrong.
> 
> Thanks,
> Vitaly Murashev
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com

From victor.stinner at gmail.com  Wed Jun 26 00:44:07 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 26 Jun 2013 00:44:07 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
Message-ID: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>

Hi,

One month ago, unit tests were added to IDLE (cool!) with a file
called @README.txt. The @ was used to see the name on top in a listing
of the directory.

Some developers began to get strange Mercurial errors like:
"abort: data/Lib/idlelib/idle_test/@README.txt.i at 7573717b9e6f: no match"
" 83941: empty or missing Lib/idlelib/idle_test/@README.txt "
etc.

Senthil reported the issue on python-committers mailing list:
http://mail.python.org/pipermail/python-committers/2013-May/002565.html

The @ character was discussed. Replacing it with "_" was proposed.
Guido asked to simply rename the name "README.txt", he wrote:

"I think we have a zen rule about this: Special cases aren't special
enough to break the rules."

Senthil was asked to upgrade its Mercurial version. Someone supposed
that it is a disk issue. Anyway, the issue disappeared with a fresh
clone.

I also had the issue on 3 different computers, and so I reported the
issue upstream:
http://bz.selenic.com/show_bug.cgi?id=3954

I discussed with a Mercurial developer, Matt, on IRC. He asked how the
server is managed, if we are using only one physical server, if
repositories are copied with rsync, etc. I was unable to answer, I
don't have access to hg.python.org server.

The issue was closed, but 20 days later (today) I reproduced the issue again.

I cloned the repository at a specific revision, tried to update to
another specific revision: no bug.

I also played with with hg bisect, because I suspected a bug in bisect: no bug.

I tried to update at each revision between 83900 and 84348 to check if
@README.txt disappears from .hg/store: still no bug.

I also ran fsck: no error (but the disk is mounted, so I don't know if
the report is reliable).

And then I ran "make distclean"...

Victor

From benjamin at python.org  Wed Jun 26 00:58:30 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 25 Jun 2013 15:58:30 -0700
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
Message-ID: <CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>

2013/6/25 Victor Stinner <victor.stinner at gmail.com>:
> And then I ran "make distclean"...

You've left us hanging...


--
Regards,
Benjamin

From senthil at uthcode.com  Wed Jun 26 03:33:05 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Tue, 25 Jun 2013 20:33:05 -0500
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
Message-ID: <CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>

On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2013/6/25 Victor Stinner <victor.stinner at gmail.com>:
> > And then I ran "make distclean"...
>
> You've left us hanging...
>
>
Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4
But still I have question as why hg complained about @README in the first
place.
Also, I hope make distclean is not working "inside" .hg folder.

-- 
Senthil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130625/cc6d4fb2/attachment.html>

From steve at pearwood.info  Wed Jun 26 03:34:11 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 Jun 2013 11:34:11 +1000
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
Message-ID: <51CA4513.5030001@pearwood.info>

On 26/06/13 08:44, Victor Stinner wrote:

> And then I ran "make distclean"...


Victor, you're a cruel, cruel man. You've told us everything except the solution to the mystery.


-- 
Steven

From dreamingforward at gmail.com  Wed Jun 26 03:44:25 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Tue, 25 Jun 2013 18:44:25 -0700
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMjeLr-8PReqKUmmQUpRW-0QbAknvfGcJJP1PT3KRCbRMHRctA@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAMjeLr-8PReqKUmmQUpRW-0QbAknvfGcJJP1PT3KRCbRMHRctA@mail.gmail.com>
Message-ID: <CAMjeLr-jd=-JhZywOcX1=9zHt4hZ996No027-mJ-ipNT2wVY6w@mail.gmail.com>

> It's like this.  Whenever you use special characters in a file name,
> you're asking for trouble.  The shell and the OS have negotiate how to
> interpret it.  It bigger than git, and not a bug.

Sorry, I meant mercurial, not git.
-- 
MarkJ
Tacoma, Washington

From dreamingforward at gmail.com  Wed Jun 26 03:43:27 2013
From: dreamingforward at gmail.com (Mark Janssen)
Date: Tue, 25 Jun 2013 18:43:27 -0700
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
Message-ID: <CAMjeLr-8PReqKUmmQUpRW-0QbAknvfGcJJP1PT3KRCbRMHRctA@mail.gmail.com>

> One month ago, unit tests were added to IDLE (cool!) with a file
> called @README.txt. The @ was used to see the name on top in a listing
> of the directory.

It's like this.  Whenever you use special characters in a file name,
you're asking for trouble.  The shell and the OS have negotiate how to
interpret it.  It bigger than git, and not a bug.  The issue is
between the file system, the kernel, and the shell.

> I also had the issue on 3 different computers, and so I reported the
> issue upstream:
> http://bz.selenic.com/show_bug.cgi?id=3954

Try it on different OS's on different machines (MacOS, Linux, Windows)
.  If my theory is right, it should be inconsistent across machines,
but consistent within the same machine.

-- 
MarkJ
Tacoma, Washington

From eric at trueblade.com  Wed Jun 26 04:11:04 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 25 Jun 2013 22:11:04 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
Message-ID: <51CA4DB8.8060802@trueblade.com>

On 6/25/2013 9:33 PM, Senthil Kumaran wrote:
> 
> 
> 
> On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson <benjamin at python.org
> <mailto:benjamin at python.org>> wrote:
> 
>     2013/6/25 Victor Stinner <victor.stinner at gmail.com
>     <mailto:victor.stinner at gmail.com>>:
>     > And then I ran "make distclean"...
> 
>     You've left us hanging...
> 
> 
> Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4
> But still I have question as why hg complained about @README in the
> first place.
> Also, I hope make distclean is not working "inside" .hg folder.

I think that's exactly what's happening.

>From the bug report:

	find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
			   -o -name '[@,#]*' -o -name '*.old' \
			   -o -name '*.orig' -o -name '*.rej' \
			   -o -name '*.bak' ')' \
			   -exec rm -f {} ';'

Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.

Just this week I saw someone use the logical equivalent of:

find $(srcdir)/* ...

to avoid this problem. It won't expand the .hg top-level directory.

-- 
Eric.

From phd at phdru.name  Wed Jun 26 12:24:01 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 26 Jun 2013 14:24:01 +0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CA4DB8.8060802@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
Message-ID: <20130626102401.GA11231@iskra.aviel.ru>

On Tue, Jun 25, 2013 at 10:11:04PM -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
> On 6/25/2013 9:33 PM, Senthil Kumaran wrote:
> > 
> > 
> > 
> > On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson <benjamin at python.org
> > <mailto:benjamin at python.org>> wrote:
> > 
> >     2013/6/25 Victor Stinner <victor.stinner at gmail.com
> >     <mailto:victor.stinner at gmail.com>>:
> >     > And then I ran "make distclean"...
> > 
> >     You've left us hanging...
> > 
> > 
> > Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4
> > But still I have question as why hg complained about @README in the
> > first place.
> > Also, I hope make distclean is not working "inside" .hg folder.
> 
> I think that's exactly what's happening.
> 
> >From the bug report:
> 
> 	find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
> 			   -o -name '[@,#]*' -o -name '*.old' \
> 			   -o -name '*.orig' -o -name '*.rej' \
> 			   -o -name '*.bak' ')' \
> 			   -exec rm -f {} ';'
> 
> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.
> 
> Just this week I saw someone use the logical equivalent of:
> 
> find $(srcdir)/* ...
> 
> to avoid this problem. It won't expand the .hg top-level directory.

   Or find \( -type d -name .hg -prune \) -o ...

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From a.cavallo at cavallinux.eu  Wed Jun 26 12:43:41 2013
From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu)
Date: Wed, 26 Jun 2013 12:43:41 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130626102401.GA11231@iskra.aviel.ru>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
Message-ID: <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>

.. or having hg "purging" unwanted build artifact (probably cleaning up 
the .hgignore file first)


>> find $(srcdir)/* ...
>>
>> to avoid this problem. It won't expand the .hg top-level directory.
>
>    Or find \( -type d -name .hg -prune \) -o ...
>
> Oleg.


From eric at trueblade.com  Wed Jun 26 14:18:27 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Jun 2013 08:18:27 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
Message-ID: <51CADC13.5060400@trueblade.com>

On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote:
> .. or having hg "purging" unwanted build artifact (probably cleaning up
> the .hgignore file first)

How would that work? How could hg purge the .bak, .orig, .rej, .old,
etc. files?

>>> find $(srcdir)/* ...
>>>
>>> to avoid this problem. It won't expand the .hg top-level directory.
>>
>>    Or find \( -type d -name .hg -prune \) -o ...

I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
all dot directories/files?

-- 
Eric.

From eric at trueblade.com  Wed Jun 26 14:21:19 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Jun 2013 08:21:19 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CADC13.5060400@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
Message-ID: <51CADCBF.1030503@trueblade.com>

On 6/26/2013 8:18 AM, Eric V. Smith wrote:
> On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote:
>> .. or having hg "purging" unwanted build artifact (probably cleaning up
>> the .hgignore file first)
> 
> How would that work? How could hg purge the .bak, .orig, .rej, .old,
> etc. files?
> 
>>>> find $(srcdir)/* ...
>>>>
>>>> to avoid this problem. It won't expand the .hg top-level directory.
>>>
>>>    Or find \( -type d -name .hg -prune \) -o ...
> 
> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
> all dot directories/files?
> 

Sorry for the mis-attribution. "Yours" refers to Oleg.

-- 
Eric.

From phd at phdru.name  Wed Jun 26 14:44:05 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 26 Jun 2013 16:44:05 +0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CADC13.5060400@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
Message-ID: <20130626124405.GA14843@iskra.aviel.ru>

On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
> >>> find $(srcdir)/* ...
> >>>
> >>> to avoid this problem. It won't expand the .hg top-level directory.
> >>
> >>    Or find \( -type d -name .hg -prune \) -o ...
> 
> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,

   How many of those dot-files/directories are there beside .*ignore?

> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
> all dot directories/files?

   On the other hand yes, I think it'd be enough.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From ronaldoussoren at mac.com  Wed Jun 26 14:57:28 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 26 Jun 2013 14:57:28 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CADC13.5060400@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
Message-ID: <F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>


On 26 Jun, 2013, at 14:18, Eric V. Smith <eric at trueblade.com> wrote:

> On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote:
>> .. or having hg "purging" unwanted build artifact (probably cleaning up
>> the .hgignore file first)
> 
> How would that work? How could hg purge the .bak, .orig, .rej, .old,
> etc. files?
> 
>>>> find $(srcdir)/* ...
>>>> 
>>>> to avoid this problem. It won't expand the .hg top-level directory.
>>> 
>>>   Or find \( -type d -name .hg -prune \) -o ...
> 
> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
> all dot directories/files?

Is the find command in the distclean target still needed? The comment for the distclean target says it is used to clean up the tree for distribution, but that's easier to accomplish by using a clean checkout.

The target is still useful to get a clean tree when you're building with srcdir == builddir, but you don't need the find command for that.

Ronald


From eric at trueblade.com  Wed Jun 26 15:02:46 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Jun 2013 09:02:46 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130626124405.GA14843@iskra.aviel.ru>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<20130626124405.GA14843@iskra.aviel.ru>
Message-ID: <51CAE676.1080501@trueblade.com>

On 6/26/2013 8:44 AM, Oleg Broytman wrote:
> On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
>>>>> find $(srcdir)/* ...
>>>>>
>>>>> to avoid this problem. It won't expand the .hg top-level directory.
>>>>
>>>>    Or find \( -type d -name .hg -prune \) -o ...
>>
>> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
> 
>    How many of those dot-files/directories are there beside .*ignore?

That's all I can think of. And the files don't matter in this case,
anyway. So probably just .svn, .hg, .git is all we'd need.

>> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
>> all dot directories/files?
> 
>    On the other hand yes, I think it'd be enough.

I can go either way.

-- 
Eric.

From eric at trueblade.com  Wed Jun 26 15:04:49 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Jun 2013 09:04:49 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>
Message-ID: <51CAE6F1.1040707@trueblade.com>

On 6/26/2013 8:57 AM, Ronald Oussoren wrote:
> 
> On 26 Jun, 2013, at 14:18, Eric V. Smith <eric at trueblade.com> wrote:
> 
>> On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote:
>>> .. or having hg "purging" unwanted build artifact (probably cleaning up
>>> the .hgignore file first)
>>
>> How would that work? How could hg purge the .bak, .orig, .rej, .old,
>> etc. files?
>>
>>>>> find $(srcdir)/* ...
>>>>>
>>>>> to avoid this problem. It won't expand the .hg top-level directory.
>>>>
>>>>   Or find \( -type d -name .hg -prune \) -o ...
>>
>> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
>> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
>> all dot directories/files?
> 
> Is the find command in the distclean target still needed? The comment for the distclean target says it is used to clean up the tree for distribution, but that's easier to accomplish by using a clean checkout.
> 
> The target is still useful to get a clean tree when you're building with srcdir == builddir, but you don't need the find command for that.

I run 'make distclean' fairly often, but maybe it's just out of habit.
If I'm adding/deleting modules, I want to make sure there are no build
artifacts. And since I have modified files, a clean checkout won't help
(easily, at least).

But me running distclean is not the same as answering your question
about the find command being needed, I realize.

-- 
Eric.

From a.cavallo at cavallinux.eu  Wed Jun 26 15:12:45 2013
From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu)
Date: Wed, 26 Jun 2013 15:12:45 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CADC13.5060400@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
Message-ID: <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu>


> How would that work? How could hg purge the .bak, .orig, .rej, .old,
> etc. files?

hg purge (it's an extension) removes anything that isn't tracked (and 
not ignored in the .hgignore): kind of distclean.

I hope this helps

From solipsis at pitrou.net  Wed Jun 26 15:28:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 Jun 2013 15:28:17 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu>
Message-ID: <20130626152817.4a19541f@pitrou.net>

Le Wed, 26 Jun 2013 15:12:45 +0200,
a.cavallo at cavallinux.eu a ?crit :
> 
> > How would that work? How could hg purge the .bak, .orig, .rej, .old,
> > etc. files?
> 
> hg purge (it's an extension) removes anything that isn't tracked (and 
> not ignored in the .hgignore): kind of distclean.

distclean removes only what we want to remove, while hg purge will
remove any untracked file, including any files that you may have wanted
to keep (notes, work-in-progress patches, personal data files, etc.).

Regards

Antoine.



From zachary.ware+pydev at gmail.com  Wed Jun 26 15:28:51 2013
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Wed, 26 Jun 2013 08:28:51 -0500
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu>
Message-ID: <CAKJDb-Ndq750sYiaWuTMEYu_pTNL3XXgaEKhE_8auXbACWc4pA@mail.gmail.com>

On Wed, Jun 26, 2013 at 8:12 AM,  <a.cavallo at cavallinux.eu> wrote:
>Eric V. Smith wrote:
>> How would that work? How could hg purge the .bak, .orig, .rej, .old,
>> etc. files?
>
>
> hg purge (it's an extension) removes anything that isn't tracked (and not
> ignored in the .hgignore): kind of distclean.
>
> I hope this helps

I've recently discovered purge and have started using it on Windows
since there is no `make distclean`, and it is very effective.  `hg
purge -p` shows what will be removed (which should match anything with
a ? in `hg status`), `hg purge` removes it, and `hg purge --all`
clears out everything that's not tracked (including things listed in
.hgignore) giving a fresh checkout without having to re-download.
Very convenient, especially since it's a built-in extension.

From barry at python.org  Wed Jun 26 15:39:54 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 26 Jun 2013 09:39:54 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CAE6F1.1040707@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>
	<51CAE6F1.1040707@trueblade.com>
Message-ID: <20130626093954.51fc6741@anarchist>

On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote:

>I run 'make distclean' fairly often, but maybe it's just out of habit.
>If I'm adding/deleting modules, I want to make sure there are no build
>artifacts. And since I have modified files, a clean checkout won't help
>(easily, at least).

As do I.  I think it still makes sense for us to include a working distclean,
especially since it's a very common target for make-based builds.

-Barry


From ronaldoussoren at mac.com  Wed Jun 26 16:05:53 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 26 Jun 2013 16:05:53 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130626093954.51fc6741@anarchist>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>
	<51CAE6F1.1040707@trueblade.com>
	<20130626093954.51fc6741@anarchist>
Message-ID: <78896606-9093-4A3A-BE5C-F1AD77528C70@mac.com>


On 26 Jun, 2013, at 15:39, Barry Warsaw <barry at python.org> wrote:

> On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote:
> 
>> I run 'make distclean' fairly often, but maybe it's just out of habit.
>> If I'm adding/deleting modules, I want to make sure there are no build
>> artifacts. And since I have modified files, a clean checkout won't help
>> (easily, at least).
> 
> As do I.  I think it still makes sense for us to include a working distclean,
> especially since it's a very common target for make-based builds.

Sure, but is it necessary to run the find command for removing backup files in make distclean?

When the find command is removed you'd still end up with a tree that's clean enough to perform a build from scratch, although the tree won't be perfectly clean. 

BTW. I usually build in a separate directory, that makes cleaning up even easier :-)

Ronald
> 
> -Barry
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com


From rdmurray at bitdance.com  Wed Jun 26 16:18:49 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 26 Jun 2013 10:18:49 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <20130626093954.51fc6741@anarchist>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<F8F33A61-D344-41BA-B059-AACB5E55E2AC@mac.com>
	<51CAE6F1.1040707@trueblade.com>
	<20130626093954.51fc6741@anarchist>
Message-ID: <20130626141849.987C4250B43@webabinitio.net>

On Wed, 26 Jun 2013 09:39:54 -0400, Barry Warsaw <barry at python.org> wrote:
> On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote:
> 
> >I run 'make distclean' fairly often, but maybe it's just out of habit.
> >If I'm adding/deleting modules, I want to make sure there are no build
> >artifacts. And since I have modified files, a clean checkout won't help
> >(easily, at least).
> 
> As do I.  I think it still makes sense for us to include a working distclean,
> especially since it's a very common target for make-based builds.

We also sometimes ask someone reporting an issue to do a make distclean
and recompile, and many of these reporters will be working from a tarball
rather than a checkout.  Sure, they could re-unpack the tarball (if they
haven't deleted it already), but make distclean is easier.

--David

From victor.stinner at gmail.com  Wed Jun 26 16:24:03 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 26 Jun 2013 16:24:03 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CA4DB8.8060802@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
Message-ID: <CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>

2013/6/26 Eric V. Smith <eric at trueblade.com>:
> I think that's exactly what's happening.
>
> From the bug report:
>
>         find $(srcdir) '(' -name '*.fdc' -o -name '*~' \
>                            -o -name '[@,#]*' -o -name '*.old' \
>                            -o -name '*.orig' -o -name '*.rej' \
>                            -o -name '*.bak' ')' \
>                            -exec rm -f {} ';'
>
> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg.

In my opinion, make distclean should only remove files generated by
configure and a build. It should not remove random files.

*~, .orig, .rej, .back should be kept. They are not generated by
configure nor make.

What are these "@*", ",*" and "#*" files? Why does "make distclean" remove them?

"make distclean" removes also the "tags" file which is generated by
the ctags program, useful tool to browse the C source code (ex: in
vim). Why does "make distclean" remove it?

In short, the whole "find ... -exec rm -f {} ';'" command should be
removed from "make distclean". (They are other commands to remove
Makefile, "*.o" files, etc.) If someone really need such cleanup,
another Makefile rule should be added.

Victor

From a.cavallo at cavallinux.eu  Wed Jun 26 16:33:50 2013
From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu)
Date: Wed, 26 Jun 2013 16:33:50 +0200
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
Message-ID: <a41fc6ebde73e9624ea24e0c3983373a@cavallinux.eu>

> *~, .orig, .rej, .back should be kept. They are not generated by
> configure nor make.

Ideally they should be left untracked not ignored.

While devs can certainly add them to the .hgignore list to make life 
easier, a repository should be clean of extra files (or shown as 
untracked).

I'd add that generated files (like the one generated from pgen) 
shouldn't be part of defaul make target but keept as fully tracked files 
(and regenerated on demand through a special make target).

I hope this helps.




From vitaly.murashev at gmail.com  Wed Jun 26 16:31:27 2013
From: vitaly.murashev at gmail.com (Vitaly Murashev)
Date: Wed, 26 Jun 2013 18:31:27 +0400
Subject: [Python-Dev] Relative path in co_filename for zipped modules
In-Reply-To: <20130625201958.29E8B250B3F@webabinitio.net>
References: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com>
	<20130625201958.29E8B250B3F@webabinitio.net>
Message-ID: <CALPW6hy2g3-O2_8g1ArwJ00qgBU=JYYepcGxHAqc8pgreDKU5w@mail.gmail.com>

Reported as http://bugs.python.org/issue18307

On Wed, Jun 26, 2013 at 12:19 AM, R. David Murray <rdmurray at bitdance.com> wrote:
> Please file this as a bug report on bugs.python.org so that it doesn't
> get lost.
>
> See also http://bugs.python.org/issue13328 (which looks like a different bug
> but could also be causing you problems).
>
> --David
>

From rdmurray at bitdance.com  Wed Jun 26 18:18:08 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 26 Jun 2013 12:18:08 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_peps=3A_Reject_PEP_3?=
	=?utf-8?b?MTUu?=
In-Reply-To: <3bgT1R1H5tz7LxL@mail.python.org>
References: <3bgT1R1H5tz7LxL@mail.python.org>
Message-ID: <20130626161809.788D7250B43@webabinitio.net>

On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa <python-checkins at python.org> wrote:
> -    Deferred; see
> +    Rejected; see
> +    http://mail.python.org/pipermail/python-ideas/2013-June/021610.html
> +
> +    This PEP has been deferred since 2006; see
>      http://mail.python.org/pipermail/python-dev/2006-February/060718.html
>  
>      Subsequent efforts to revive the PEP in April 2009 did not
>      meet with success because no syntax emerged that could
> -    compete with a while-True and an inner if-break.
> +    compete with the following form:
>  
> -    A syntax was found for a basic do-while loop but it found
> -    had little support because the condition was at the top:
> +        while True:
> +            <setup code>
> +            if not <condition>:
> +                break
> +            <loop body>
>  
> -        do ... while <cond>:
> -            <loop body>
> +    Users of the language are advised to use that form when a do-while
> +    loop would have been appropriate.

Why delete the information about what was found wanting?

--David

From lukasz at langa.pl  Wed Jun 26 19:10:03 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 26 Jun 2013 19:10:03 +0200
Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315.
In-Reply-To: <20130626161809.788D7250B43@webabinitio.net>
References: <3bgT1R1H5tz7LxL@mail.python.org>
	<20130626161809.788D7250B43@webabinitio.net>
Message-ID: <D9A73CE6-4121-4407-8DF7-A11E3643CCF4@langa.pl>

On 26 cze 2013, at 18:18, R. David Murray <rdmurray at bitdance.com> wrote:

> On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa <python-checkins at python.org> wrote:
>> -    Deferred; see
>> +    Rejected; see
>> +    http://mail.python.org/pipermail/python-ideas/2013-June/021610.html
>> +
>> +    This PEP has been deferred since 2006; see
>>     http://mail.python.org/pipermail/python-dev/2006-February/060718.html
>> 
>>     Subsequent efforts to revive the PEP in April 2009 did not
>>     meet with success because no syntax emerged that could
>> -    compete with a while-True and an inner if-break.
>> +    compete with the following form:
>> 
>> -    A syntax was found for a basic do-while loop but it found
>> -    had little support because the condition was at the top:
>> +        while True:
>> +            <setup code>
>> +            if not <condition>:
>> +                break
>> +            <loop body>
>> 
>> -        do ... while <cond>:
>> -            <loop body>
>> +    Users of the language are advised to use that form when a do-while
>> +    loop would have been appropriate.
> 
> Why delete the information about what was found wanting?

This was a mention of an alternative possible syntax. I don't feel it was worth keeping this particular one while not discussing any other alternatives at all. If you'd rather keep this information, it would be better off to have it to an "Alternative approaches" section.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130626/d466c8a6/attachment.html>

From turnbull at sk.tsukuba.ac.jp  Wed Jun 26 19:41:22 2013
From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 27 Jun 2013 02:41:22 +0900
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<CAMpsgwYST2OSOyKjowQaNjUwDpESHzstkc7LqqOYLRYh=6uabw@mail.gmail.com>
Message-ID: <87a9mczs59.fsf@uwakimon.sk.tsukuba.ac.jp>

Victor Stinner writes:

 > In my opinion, make distclean should only remove files generated by
 > configure and a build. It should not remove random files.

FWIW, the GNU standard for these targets is something like:

## make clean  or  make mostlyclean
##      Delete all files from the current directory that are normally
##      created by building the program.  Don't delete the files that
##      record the configuration.  Also preserve files that could be
##      made by building, but normally aren't because the distribution
##      comes with them.
##      Delete `.dvi' files here if they are not part of the
##      distribution.

## make distclean
##      Delete all files from the current directory that are created by
##      configuring or building the program.  If you have unpacked the
##      source and built the program without creating any other files,
##      `make distclean' should leave only the files that were in the
##      distribution.

## make realclean
##      Delete everything from the current directory that can be
##      reconstructed with this Makefile.  This typically includes
##      everything deleted by distclean, plus more: C source files
##      produced by Bison, tags tables, info files, and so on.

## make extraclean
##      Still more severe - delete backup and autosave files, too.

This is from the XEmacs Makefile.in.in, so it's not authoritative.
Still, it seems pretty intuitive and presumably is in wide use, not to
forget matching Victor's preferred usage for 'distclean'.


From victor.stinner at gmail.com  Thu Jun 27 00:07:11 2013
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 27 Jun 2013 00:07:11 +0200
Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check
	for overflow
Message-ID: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>

Hi,

I don't understand why the B, H, I, k formats of PyArg_ParseTupe() do
not check for overflow, whereas formats for signed numbers do check
for overflow. What is the useful of ignoring overflow?
http://docs.python.org/3/c-api/arg.html

I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
64-bit system:
http://bugs.python.org/issue18294

How should I implement that? Use "O" format and then use
PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?

Victor

From guido at python.org  Thu Jun 27 00:26:05 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 Jun 2013 15:26:05 -0700
Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and
 check for overflow
In-Reply-To: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
References: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
Message-ID: <CAP7+vJLudusi2GswCMoQ0Fb-O89TYM1aLwYq1rup_Hg7k72dOA@mail.gmail.com>

On Wed, Jun 26, 2013 at 3:07 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> I don't understand why the B, H, I, k formats of PyArg_ParseTupe() do
> not check for overflow, whereas formats for signed numbers do check
> for overflow. What is the useful of ignoring overflow?
> http://docs.python.org/3/c-api/arg.html

I think this is really old DNA. It comes from times when 0xffffffff
and -1 were equivalent -- we borrowed this cavalier attitude from old
C code.

I worry that "fixing" this would break some old libraries relying on
the feature, so I hope we can avoid changes in this area (the older
the DNA, the more hidden dependencies it has ;-).

> I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
> 64-bit system:
> http://bugs.python.org/issue18294
>
> How should I implement that? Use "O" format and then use
> PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?

Why can't you use the K format? It won't reject out-of-range values,
but it will convert them to in-range so there aren't any attacks
possible based on bypassing the range check. I'm probably
misunderstanding something -- I don't completely understand that bug
report. :-(

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Thu Jun 27 00:51:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Jun 2013 08:51:53 +1000
Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315.
In-Reply-To: <D9A73CE6-4121-4407-8DF7-A11E3643CCF4@langa.pl>
References: <3bgT1R1H5tz7LxL@mail.python.org>
	<20130626161809.788D7250B43@webabinitio.net>
	<D9A73CE6-4121-4407-8DF7-A11E3643CCF4@langa.pl>
Message-ID: <CADiSq7cb+O0WymgAWozBYVPXYQt49wxp7kv4N4-n-bb6603NGg@mail.gmail.com>

On 27 Jun 2013 03:12, "?ukasz Langa" <lukasz at langa.pl> wrote:
>
> On 26 cze 2013, at 18:18, R. David Murray <rdmurray at bitdance.com> wrote:
>
>> On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa <
python-checkins at python.org> wrote:
>>>
>>> -    Deferred; see
>>> +    Rejected; see
>>> +    http://mail.python.org/pipermail/python-ideas/2013-June/021610.html
>>> +
>>> +    This PEP has been deferred since 2006; see
>>>
http://mail.python.org/pipermail/python-dev/2006-February/060718.html
>>>
>>>     Subsequent efforts to revive the PEP in April 2009 did not
>>>     meet with success because no syntax emerged that could
>>> -    compete with a while-True and an inner if-break.
>>> +    compete with the following form:
>>>
>>> -    A syntax was found for a basic do-while loop but it found
>>> -    had little support because the condition was at the top:
>>> +        while True:
>>> +            <setup code>
>>> +            if not <condition>:
>>> +                break
>>> +            <loop body>
>>>
>>> -        do ... while <cond>:
>>> -            <loop body>
>>> +    Users of the language are advised to use that form when a do-while
>>> +    loop would have been appropriate.
>>
>>
>> Why delete the information about what was found wanting?
>
>
> This was a mention of an alternative possible syntax. I don't feel it was
worth keeping this particular one while not discussing any other
alternatives at all. If you'd rather keep this information, it would be
better off to have it to an "Alternative approaches" section.

It was relevant because it was the most acceptable alternative found, and
we *still* didn't think it was an improvement over the status quo.

We don't generally significantly edit PEPs when accepting or rejecting them.

Cheers,
Nick.

>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130627/cf67b1fc/attachment.html>

From eric at trueblade.com  Thu Jun 27 03:27:55 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Jun 2013 21:27:55 -0400
Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug"
In-Reply-To: <51CAE676.1080501@trueblade.com>
References: <CAMpsgwYOfTZtxyGv2Vd+7CYUo-=W2e26_tQQo-7vcGdDtC9OKg@mail.gmail.com>
	<CAPZV6o9vvjDVOzTzB0CgCECW9gEgmnSBa9_-XxJ-FO4guuDzNg@mail.gmail.com>
	<CAPOVWOTj8TBGsDvoium4cBXjerY5gkrPA3jOvVf9d8zB=9aFag@mail.gmail.com>
	<51CA4DB8.8060802@trueblade.com>
	<20130626102401.GA11231@iskra.aviel.ru>
	<3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu>
	<51CADC13.5060400@trueblade.com>
	<20130626124405.GA14843@iskra.aviel.ru>
	<51CAE676.1080501@trueblade.com>
Message-ID: <51CB951B.1000501@trueblade.com>

On 6/26/2013 9:02 AM, Eric V. Smith wrote:
> On 6/26/2013 8:44 AM, Oleg Broytman wrote:
>> On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" <eric at trueblade.com> wrote:
>>>>>> find $(srcdir)/* ...
>>>>>>
>>>>>> to avoid this problem. It won't expand the .hg top-level directory.
>>>>>
>>>>>    Or find \( -type d -name .hg -prune \) -o ...
>>>
>>> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git,
>>
>>    How many of those dot-files/directories are there beside .*ignore?
> 
> That's all I can think of. And the files don't matter in this case,
> anyway. So probably just .svn, .hg, .git is all we'd need.
> 
>>> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore
>>> all dot directories/files?
>>
>>    On the other hand yes, I think it'd be enough.
> 
> I can go either way.
> 

I created http://bugs.python.org/issue18312 to track this.

-- 
Eric.

From guido at python.org  Thu Jun 27 03:56:10 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 Jun 2013 18:56:10 -0700
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
	formatting (PEP 257)
Message-ID: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>

PEP 257 says this on the formatting of multi-line docstrings:

"""
Multi-line docstrings consist of a summary line just like a one-line
docstring, followed by a blank line, followed by a more elaborate
description. The summary line may be used by automatic indexing tools;
it is important that it fits on one line and is separated from the
rest of the docstring by a blank line. [...]
"""

I still like this rule, but it is violated frequently, in the stdlib
and elsewhere. I'd like to urge stdlib contributors and core devs to
heed it -- or explain why you can't.

-- 
--Guido van Rossum (python.org/~guido)

From skip at pobox.com  Thu Jun 27 04:08:27 2013
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 26 Jun 2013 21:08:27 -0500
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
Message-ID: <CANc-5Ux1emJecT8yySSt8Z0wwND9Gg_eUB6kUDnNsGNVpESd+Q@mail.gmail.com>

> I'd like to urge stdlib contributors and core devs to
> heed it -- or explain why you can't.

Where I can, I do, however I often find it difficult to come up with a
one-liner, especially one that mentions the parameters to functions.
If the one-line rule is going to be violated, I go whole hog and don't
bother with the blank line either.  (If I'm going to punch a hole in a
rule, I might as well create one big enough to comfortably walk
through.)

Skip

From ethan at stoneleaf.us  Thu Jun 27 04:14:01 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 26 Jun 2013 19:14:01 -0700
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CANc-5Ux1emJecT8yySSt8Z0wwND9Gg_eUB6kUDnNsGNVpESd+Q@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<CANc-5Ux1emJecT8yySSt8Z0wwND9Gg_eUB6kUDnNsGNVpESd+Q@mail.gmail.com>
Message-ID: <51CB9FE9.5050300@stoneleaf.us>

On 06/26/2013 07:08 PM, Skip Montanaro wrote:
>> I'd like to urge stdlib contributors and core devs to
>> heed it -- or explain why you can't.
>
> Where I can, I do, however I often find it difficult to come up with a
> one-liner, especially one that mentions the parameters to functions.
> If the one-line rule is going to be violated, I go whole hog and don't
> bother with the blank line either.  (If I'm going to punch a hole in a
> rule, I might as well create one big enough to comfortably walk
> through.)

Why do the parameters have to be in the summary?  You could do them better justice in the following paragraphs.

--
~Ethan~

From ben+python at benfinney.id.au  Thu Jun 27 07:29:00 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 27 Jun 2013 15:29:00 +1000
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
	formatting (PEP 257)
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<CANc-5Ux1emJecT8yySSt8Z0wwND9Gg_eUB6kUDnNsGNVpESd+Q@mail.gmail.com>
Message-ID: <7w1u7oktpf.fsf@benfinney.id.au>

Skip Montanaro <skip at pobox.com> writes:

> Where I can [conform to PEP 257], I do, however I often find it
> difficult to come up with a one-liner, especially one that mentions
> the parameters to functions.

I think it's far better to make the synopsis a summary of the purpose of
the function; no need to force the parameters in there if they don't fit
naturally in the sentence fragment.

A list of the parameters and return value can go in the detailed
description.

-- 
 \       ?Theology is the effort to explain the unknowable in terms of |
  `\                         the not worth knowing.? ?Henry L. Mencken |
_o__)                                                                  |
Ben Finney


From ericsnowcurrently at gmail.com  Thu Jun 27 08:03:05 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 27 Jun 2013 00:03:05 -0600
Subject: [Python-Dev] Classes with ordered namespaces
Message-ID: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>

There are two relevant class namespaces in this proposal: definition
namespace and __dict__.  Currently both are dicts.

For class definition namespaces, I'd like to make the default an
OrderedDict.  With the implementation in issue16991, the change is
trivial (basically 1-liners in 2 spots).  This change would make it
unnecessary to write a custom metaclass just for a __prepare__().  PEP
422 alleviates that problem somewhat.  However, I expect OrderedDict
is by far the most common type returned by __prepare__(), so having it
be the default would proportionately reduce the need for people to
write metaclasses or learn about the PEP 422 API.  You may ask what is
the point if they aren't using a metaclass.  That leads to the other
half of the proposal.

Once I have a class, I'd like to know the definition order without
needing a metaclass.  Unfortunately it's not as simple as using the C
OrderedDict (issue16991 again) for tp_dict, etc.  That change is
actually pretty trivial.  However, it causes problems because the
concrete C API (PyDict_*) doesn't play nice with subclasses and the
concrete API is used on class __dict__ all over the place.  The
alternative I propose is to add __definition_order__ or similar to
classes.  It would be a list of the names from the definition
namespace, in definition order (making use of the new default there).
Having a class's __dict__ be ordered isn't important if the definition
order is available somewhere.  Again, using the C OrderedDict, the
change to add __definition_order__ is pretty trivial.

FWIW, Guido already expressed some approval[1].

-eric


[1] http://mail.python.org/pipermail/python-ideas/2013-February/019704.html

From solipsis at pitrou.net  Thu Jun 27 10:21:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 Jun 2013 10:21:43 +0200
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
Message-ID: <20130627102143.65125bc3@pitrou.net>

Le Wed, 26 Jun 2013 18:56:10 -0700,
Guido van Rossum <guido at python.org> a ?crit :
> PEP 257 says this on the formatting of multi-line docstrings:
> 
> """
> Multi-line docstrings consist of a summary line just like a one-line
> docstring, followed by a blank line, followed by a more elaborate
> description. The summary line may be used by automatic indexing tools;
> it is important that it fits on one line and is separated from the
> rest of the docstring by a blank line. [...]
> """
> 
> I still like this rule, but it is violated frequently, in the stdlib
> and elsewhere. I'd like to urge stdlib contributors and core devs to
> heed it -- or explain why you can't.

I don't always find it easy to summarize a function in one line.

Regards

Antoine.



From ncoghlan at gmail.com  Thu Jun 27 10:48:47 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 Jun 2013 18:48:47 +1000
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
Message-ID: <CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>

On 27 June 2013 16:03, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> There are two relevant class namespaces in this proposal: definition
> namespace and __dict__.  Currently both are dicts.
>
> For class definition namespaces, I'd like to make the default an
> OrderedDict.  With the implementation in issue16991, the change is
> trivial (basically 1-liners in 2 spots).  This change would make it
> unnecessary to write a custom metaclass just for a __prepare__().  PEP
> 422 alleviates that problem somewhat.  However, I expect OrderedDict
> is by far the most common type returned by __prepare__(), so having it
> be the default would proportionately reduce the need for people to
> write metaclasses or learn about the PEP 422 API.  You may ask what is
> the point if they aren't using a metaclass.

I'd be tempted to kill PEP 422 as not worth the hassle if we did this.
Yes, I count that as a point in favour of the idea :)

> That leads to the other
> half of the proposal.
>
> Once I have a class, I'd like to know the definition order without
> needing a metaclass.  Unfortunately it's not as simple as using the C
> OrderedDict (issue16991 again) for tp_dict, etc.  That change is
> actually pretty trivial.  However, it causes problems because the
> concrete C API (PyDict_*) doesn't play nice with subclasses and the
> concrete API is used on class __dict__ all over the place.  The
> alternative I propose is to add __definition_order__ or similar to
> classes.  It would be a list of the names from the definition
> namespace, in definition order (making use of the new default there).
> Having a class's __dict__ be ordered isn't important if the definition
> order is available somewhere.  Again, using the C OrderedDict, the
> change to add __definition_order__ is pretty trivial.

I think the main concern I would have is whether other implementations
are happy they can provide a suitable ordered dictionary for class
namespace execution.

It's also worth considering what would have to happen for dynamically
created types where the namespace passed in *wasn't* ordered.

It's a plausible way forward, though.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From lukasz at langa.pl  Thu Jun 27 10:49:41 2013
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Thu, 27 Jun 2013 10:49:41 +0200
Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315.
In-Reply-To: <CADiSq7cb+O0WymgAWozBYVPXYQt49wxp7kv4N4-n-bb6603NGg@mail.gmail.com>
References: <3bgT1R1H5tz7LxL@mail.python.org>
	<20130626161809.788D7250B43@webabinitio.net>
	<D9A73CE6-4121-4407-8DF7-A11E3643CCF4@langa.pl>
	<CADiSq7cb+O0WymgAWozBYVPXYQt49wxp7kv4N4-n-bb6603NGg@mail.gmail.com>
Message-ID: <929D0643-3516-4133-AE53-C73E2C4E5C29@langa.pl>

On 27 cze 2013, at 00:51, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 Jun 2013 03:12, "?ukasz Langa" <lukasz at langa.pl> wrote:
> > This was a mention of an alternative possible syntax. I don't feel it was worth keeping this particular one while not discussing any other alternatives at all. If you'd rather keep this information, it would be better off to have it to an "Alternative approaches" section.
> 
> It was relevant because it was the most acceptable alternative found, and we *still* didn't think it was an improvement over the status quo.
> 
> We don't generally significantly edit PEPs when accepting or rejecting them.
> 


Comment restored, point taken. Thanks for clearing this up.


-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130627/5316fada/attachment.html>

From guido at python.org  Thu Jun 27 16:20:55 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 27 Jun 2013 07:20:55 -0700
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <20130627102143.65125bc3@pitrou.net>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<20130627102143.65125bc3@pitrou.net>
Message-ID: <CAP7+vJL0SvTRLt=O=4yzp4Sr-ZRgMHdk7SwjxjT2ftdGHLF6_A@mail.gmail.com>

On Thu, Jun 27, 2013 at 1:21 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I don't always find it easy to summarize a function in one line.

Neither do I. But I always manage to do it anyway. My reasoning is,
you can always leave out more detail and add it to the subsequent
paragraph.

-- 
--Guido van Rossum (python.org/~guido)

From larry at hastings.org  Thu Jun 27 17:09:13 2013
From: larry at hastings.org (Larry Hastings)
Date: Thu, 27 Jun 2013 10:09:13 -0500
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
Message-ID: <51CC5599.9010305@hastings.org>

On 06/26/2013 08:56 PM, Guido van Rossum wrote:
> PEP 257 says this on the formatting of multi-line docstrings:
>
> """
> Multi-line docstrings consist of a summary line just like a one-line
> docstring, followed by a blank line, followed by a more elaborate
> description. The summary line may be used by automatic indexing tools;
> it is important that it fits on one line and is separated from the
> rest of the docstring by a blank line. [...]
> """
>
> I still like this rule, but it is violated frequently, in the stdlib
> and elsewhere. I'd like to urge stdlib contributors and core devs to
> heed it -- or explain why you can't.

Argument Clinic could conceivably enforce this.  It could mandate that 
the first paragraph of the function docstring contain exactly one 
sentence (must end in a period, all embedded periods cannot be followed by
whitespace).  This would make some things nicer; I could automatically 
insert the per-parameter docstrings in after the summary.

Should it?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130627/721fe993/attachment.html>

From ericsnowcurrently at gmail.com  Thu Jun 27 17:18:30 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 27 Jun 2013 09:18:30 -0600
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
Message-ID: <CALFfu7BaXquuMBo1XMs-nGdV+a_gVctUB2kUOnesE4HZaxeLDg@mail.gmail.com>

On Thu, Jun 27, 2013 at 2:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I think the main concern I would have is whether other implementations
> are happy they can provide a suitable ordered dictionary for class
> namespace execution.
>
> It's also worth considering what would have to happen for dynamically
> created types where the namespace passed in *wasn't* ordered.

Good points.  In either case there is no definition order available.
I'm okay with that.  Would it be better to represent that as None (and
the attribute is always defined) or by having the attribute undefined?
 I'd rather always have the attribute, but I expect the significantly
simpler solution is to leave the attribute undefined when definition
order is not available.  So I'd go with the latter.

-eric

From guido at python.org  Thu Jun 27 17:57:32 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 27 Jun 2013 08:57:32 -0700
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <51CC5599.9010305@hastings.org>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<51CC5599.9010305@hastings.org>
Message-ID: <CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>

Yes on one line, capitalized, period. No on single sentence.

--Guido van Rossum (sent from Android phone)
On Jun 27, 2013 8:17 AM, "Larry Hastings" <larry at hastings.org> wrote:

>  On 06/26/2013 08:56 PM, Guido van Rossum wrote:
>
> PEP 257 says this on the formatting of multi-line docstrings:
>
> """
> Multi-line docstrings consist of a summary line just like a one-line
> docstring, followed by a blank line, followed by a more elaborate
> description. The summary line may be used by automatic indexing tools;
> it is important that it fits on one line and is separated from the
> rest of the docstring by a blank line. [...]
> """
>
> I still like this rule, but it is violated frequently, in the stdlib
> and elsewhere. I'd like to urge stdlib contributors and core devs to
> heed it -- or explain why you can't.
>
>
> Argument Clinic could conceivably enforce this.  It could mandate that the
> first paragraph of the function docstring contain exactly one sentence
> (must end in a period, all embedded periods cannot be followed by
> whitespace).  This would make some things nicer; I could automatically
> insert the per-parameter docstrings in after the summary.
>
> Should it?
>
>
> */arry*
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130627/65ec1b82/attachment.html>

From ethan at stoneleaf.us  Thu Jun 27 17:35:38 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 27 Jun 2013 08:35:38 -0700
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CALFfu7BaXquuMBo1XMs-nGdV+a_gVctUB2kUOnesE4HZaxeLDg@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
	<CALFfu7BaXquuMBo1XMs-nGdV+a_gVctUB2kUOnesE4HZaxeLDg@mail.gmail.com>
Message-ID: <51CC5BCA.7050804@stoneleaf.us>

On 06/27/2013 08:18 AM, Eric Snow wrote:
> On Thu, Jun 27, 2013 at 2:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I think the main concern I would have is whether other implementations
>> are happy they can provide a suitable ordered dictionary for class
>> namespace execution.
>>
>> It's also worth considering what would have to happen for dynamically
>> created types where the namespace passed in *wasn't* ordered.
>
> Good points.  In either case there is no definition order available.
> I'm okay with that.  Would it be better to represent that as None (and
> the attribute is always defined) or by having the attribute undefined?
>   I'd rather always have the attribute, but I expect the significantly
> simpler solution is to leave the attribute undefined when definition
> order is not available.  So I'd go with the latter.

So in Python space the options are either:

    if some_class.__definition_order__ is not None:

or

    if getattr(some_class, '__definition_order__', None) is not None:

?

Seems like always defined would be easier.

--
~Ethan~



From tjreedy at udel.edu  Thu Jun 27 18:40:14 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 27 Jun 2013 12:40:14 -0400
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<51CC5599.9010305@hastings.org>
	<CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>
Message-ID: <kqhptb$oms$1@ger.gmane.org>

On 6/27/2013 11:57 AM, Guido van Rossum wrote:
> Yes on one line, capitalized, period. No on single sentence.

Complete and correct docstrings are somewhat rare in idlelib.
About half are missing.
Single lines typically omit the period.
Multiple lines often omit the blank line after the first.

I will take your reminder as permission to make changes as I review the 
code. I wish there were more good summary lines already.

My understanding is that SubclassTests and test_methods should not have 
docstrings.

-- 
Terry Jan Reedy


From guido at python.org  Thu Jun 27 18:57:30 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 27 Jun 2013 09:57:30 -0700
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <kqhptb$oms$1@ger.gmane.org>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<51CC5599.9010305@hastings.org>
	<CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>
	<kqhptb$oms$1@ger.gmane.org>
Message-ID: <CAP7+vJLh0jEnE_KHW49xAeHy1cq9+42wjBEJJg74P1AV_MQs0Q@mail.gmail.com>

It was never my intention to enforce that everything has a docstring.
Just that if it does, it looks good.

On Thu, Jun 27, 2013 at 9:40 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/27/2013 11:57 AM, Guido van Rossum wrote:
>>
>> Yes on one line, capitalized, period. No on single sentence.
>
>
> Complete and correct docstrings are somewhat rare in idlelib.
> About half are missing.
> Single lines typically omit the period.
> Multiple lines often omit the blank line after the first.
>
> I will take your reminder as permission to make changes as I review the
> code. I wish there were more good summary lines already.
>
> My understanding is that SubclassTests and test_methods should not have
> docstrings.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From tjreedy at udel.edu  Thu Jun 27 20:25:21 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 27 Jun 2013 14:25:21 -0400
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJLh0jEnE_KHW49xAeHy1cq9+42wjBEJJg74P1AV_MQs0Q@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<51CC5599.9010305@hastings.org>
	<CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>
	<kqhptb$oms$1@ger.gmane.org>
	<CAP7+vJLh0jEnE_KHW49xAeHy1cq9+42wjBEJJg74P1AV_MQs0Q@mail.gmail.com>
Message-ID: <kqi02e$20h$1@ger.gmane.org>

On 6/27/2013 12:57 PM, Guido van Rossum wrote:
> It was never my intention to enforce that everything has a docstring.
> Just that if it does, it looks good.

Ok, I won't add them when a function's name actually makes what it does 
obvious. But when I have to spend at least a few minutes reading the 
body to make sense of it, I will try to add the summary line that I wish 
had been there already.

-- 
Terry Jan Reedy


From pje at telecommunity.com  Thu Jun 27 23:45:36 2013
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 27 Jun 2013 17:45:36 -0400
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
Message-ID: <CALeMXf6ngvWee+7WsGzYWfx9i0Or3KMD5r1o6jLmsLA=JEtENw@mail.gmail.com>

On Thu, Jun 27, 2013 at 4:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I'd be tempted to kill PEP 422 as not worth the hassle if we did this.

I assume you mean the "namespace" keyword part of PEP 422, since PEP
422's class initialization feature is entirely orthogonal to
definition order or namespace customization.  (Indeed, I cannot recall
a single instance of class initialization in my code that actually
*cared* about definition order.  Certainly I haven't had any
situations where a pre-existing definition order would've eliminated
the need for a class-level initialization.)

From ericsnowcurrently at gmail.com  Thu Jun 27 23:48:25 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 27 Jun 2013 15:48:25 -0600
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <51CC5BCA.7050804@stoneleaf.us>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
	<CALFfu7BaXquuMBo1XMs-nGdV+a_gVctUB2kUOnesE4HZaxeLDg@mail.gmail.com>
	<51CC5BCA.7050804@stoneleaf.us>
Message-ID: <CALFfu7Dj8A4qFhyokDSqBVCYfbiwfj1rMeqe7igJovh0Tqk+YQ@mail.gmail.com>

On Thu, Jun 27, 2013 at 9:35 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/27/2013 08:18 AM, Eric Snow wrote:
>> Good points.  In either case there is no definition order available.
>> I'm okay with that.  Would it be better to represent that as None (and
>> the attribute is always defined) or by having the attribute undefined?
>> I'd rather always have the attribute, but I expect the significantly
>> simpler solution is to leave the attribute undefined when definition
>> order is not available.  So I'd go with the latter.
>
>
> So in Python space the options are either:
>
>    if some_class.__definition_order__ is not None:
>
> or
>
>    if getattr(some_class, '__definition_order__', None) is not None:
>
> ?
>
> Seems like always defined would be easier.

Certainly it's easier to use if you need the definition order.
However, I expect it wouldn't be used often enough that that would
make a big difference.  Furthermore, if it's always there then it will
always show up when you dir(SomeClass) or look at SomeClass.__dict__.
People would see that on every class, whether it actually provided
definition order or not.  I'm not sure that is as helpful as only
having the attribute around when order is defined.

This got me thinking.  The attribute should be a class-only attribute
(i.e. a property on type) like __name__ is.  That would keep it from
showing up on instance lookups and on every vars(obj).

The big thing for me, though, is that optionally having the attribute
accommodates Python implementations that can't (or don't) preserve
definition order.  They don't have to worry about adding a dummy
attribute to every class just for the sake of complying with the spec.
 It also means the implementation for dynamic types like Nick
mentioned don't necesarily need to be touched.  The more I think about
it the more the optionally set attribute approach is the way to go.

-eric

From alexander.belopolsky at gmail.com  Thu Jun 27 23:51:44 2013
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 27 Jun 2013 17:51:44 -0400
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJL0SvTRLt=O=4yzp4Sr-ZRgMHdk7SwjxjT2ftdGHLF6_A@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<20130627102143.65125bc3@pitrou.net>
	<CAP7+vJL0SvTRLt=O=4yzp4Sr-ZRgMHdk7SwjxjT2ftdGHLF6_A@mail.gmail.com>
Message-ID: <CAP7h-xb3Efj3fd3MqemFyvYSX70MNG1NAOmL=Kirr2H94YQLVg@mail.gmail.com>

On Thu, Jun 27, 2013 at 10:20 AM, Guido van Rossum <guido at python.org> wrote:

> On Thu, Jun 27, 2013 at 1:21 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > I don't always find it easy to summarize a function in one line.
>
> Neither do I. But I always manage to do it anyway.


+1

If you cannot summarize what your function does in one line, chances are it
is time to split your function, not the summary line.  Ideally, the name of
the function should already give a good idea of what it does.  A summary
line can just rephrase the same in a complete sentence.

I took a quick look at the stdlib and in many cases a summary line is
already there.  It is just the issue of formatting.

 250<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l250>
class _ErrorHolder(object):
251<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l251>
"""
252<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l252>
Placeholder
for a TestCase inside a result. As far as a TestResult
253<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l253>
is
concerned, this looks exactly like a unit test. Used to insert
254<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l254>
arbitrary
errors into a test suite run.
255<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l255>
"""
*http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l250*

81 <http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l81>
class TestProgram(object):
82 <http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l82>
"""A
command-line program that runs a set of tests; this is primarily
83 <http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l83>
for
making test modules conveniently executable.
84 <http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l84>
"""
*http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l81*

109<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l109>
@failfast
110<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l110>
def addError(self, test, err):
111<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l111>
"""Called
when an error has occurred. 'err' is a tuple of values as
112<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l112>
returned
by sys.exc_info().
113<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l113>
"""
114<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l114>
self.errors.append((test, self._exc_info_to_string(err, test)))
115<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l115>
self._mirrorOutput = True
116<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l116>
117<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l117>
@failfast
118<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l118>
def addFailure(self, test, err):
119<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l119>
"""Called
when an error has occurred. 'err' is a tuple of values as
120<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l120>
returned
by sys.exc_info()."""
121<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l121>
self.failures.append((test, self._exc_info_to_string(err, test)))
122<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l122>
self._mirrorOutput = True
123<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l123>
124<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l124>
@failfast
125<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l125>
def addSubTest(self, test, subtest, err):
126<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l126>
"""Called
at the end of a subtest.
127<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l127>
'err'
is None if the subtest ended successfully, otherwise it's a
128<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l128>
tuple
of values as returned by sys.exc_info().
129<http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l129>
"""
*http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l109*
*
*
589 <http://hg.python.org/cpython/file/44f455e6163d/Lib/pdb.py#l589> def
do_break(self, arg, temporary = 0):
590 <http://hg.python.org/cpython/file/44f455e6163d/Lib/pdb.py#l590> """b(reak)
[ ([filename:]lineno | function) [, condition] ]
591 <http://hg.python.org/cpython/file/44f455e6163d/Lib/pdb.py#l591> Without
argument, list all breaks.
*http://hg.python.org/cpython/file/44f455e6163d/Lib/pdb.py#l589*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130627/e14af801/attachment-0001.html>

From ncoghlan at gmail.com  Fri Jun 28 00:02:59 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Jun 2013 08:02:59 +1000
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CALeMXf6ngvWee+7WsGzYWfx9i0Or3KMD5r1o6jLmsLA=JEtENw@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
	<CALeMXf6ngvWee+7WsGzYWfx9i0Or3KMD5r1o6jLmsLA=JEtENw@mail.gmail.com>
Message-ID: <CADiSq7f4XY72mUQJaqiwtf1oK8PS327PSdrD==_eza13Fe+G5A@mail.gmail.com>

On 28 Jun 2013 07:46, "PJ Eby" <pje at telecommunity.com> wrote:
>
> On Thu, Jun 27, 2013 at 4:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > I'd be tempted to kill PEP 422 as not worth the hassle if we did this.
>
> I assume you mean the "namespace" keyword part of PEP 422, since PEP
> 422's class initialization feature is entirely orthogonal to
> definition order or namespace customization.  (Indeed, I cannot recall
> a single instance of class initialization in my code that actually
> *cared* about definition order.  Certainly I haven't had any
> situations where a pre-existing definition order would've eliminated
> the need for a class-level initialization.)

That and the second argument to the new initialisation hook. So you're
right, the consequences for PEP 422 would be "greatly simplify" rather than
"kill".

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130628/b6925062/attachment.html>

From ncoghlan at gmail.com  Fri Jun 28 00:19:59 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Jun 2013 08:19:59 +1000
Subject: [Python-Dev] Classes with ordered namespaces
In-Reply-To: <CALFfu7Dj8A4qFhyokDSqBVCYfbiwfj1rMeqe7igJovh0Tqk+YQ@mail.gmail.com>
References: <CALFfu7BtvKySD4qgagiPGrFAZBJs391=bMp8MaXfECXXLbTwQw@mail.gmail.com>
	<CADiSq7eCHBGs7wenQT63Zpgu9CBma-_oq5nB2emjs6Zv+GuVYw@mail.gmail.com>
	<CALFfu7BaXquuMBo1XMs-nGdV+a_gVctUB2kUOnesE4HZaxeLDg@mail.gmail.com>
	<51CC5BCA.7050804@stoneleaf.us>
	<CALFfu7Dj8A4qFhyokDSqBVCYfbiwfj1rMeqe7igJovh0Tqk+YQ@mail.gmail.com>
Message-ID: <CADiSq7dEP8qsU300ryjsODF=DifFBVRSauZe3LPDVFsntLOfOg@mail.gmail.com>

On 28 Jun 2013 07:51, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>
> On Thu, Jun 27, 2013 at 9:35 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> > On 06/27/2013 08:18 AM, Eric Snow wrote:
> >> Good points.  In either case there is no definition order available.
> >> I'm okay with that.  Would it be better to represent that as None (and
> >> the attribute is always defined) or by having the attribute undefined?
> >> I'd rather always have the attribute, but I expect the significantly
> >> simpler solution is to leave the attribute undefined when definition
> >> order is not available.  So I'd go with the latter.
> >
> >
> > So in Python space the options are either:
> >
> >    if some_class.__definition_order__ is not None:
> >
> > or
> >
> >    if getattr(some_class, '__definition_order__', None) is not None:
> >
> > ?
> >
> > Seems like always defined would be easier.
>
> Certainly it's easier to use if you need the definition order.
> However, I expect it wouldn't be used often enough that that would
> make a big difference.  Furthermore, if it's always there then it will
> always show up when you dir(SomeClass) or look at SomeClass.__dict__.
> People would see that on every class, whether it actually provided
> definition order or not.  I'm not sure that is as helpful as only
> having the attribute around when order is defined.
>
> This got me thinking.  The attribute should be a class-only attribute
> (i.e. a property on type) like __name__ is.  That would keep it from
> showing up on instance lookups and on every vars(obj).
>
> The big thing for me, though, is that optionally having the attribute
> accommodates Python implementations that can't (or don't) preserve
> definition order.  They don't have to worry about adding a dummy
> attribute to every class just for the sake of complying with the spec.
>  It also means the implementation for dynamic types like Nick
> mentioned don't necesarily need to be touched.  The more I think about
> it the more the optionally set attribute approach is the way to go.

My experience with maybe set, maybe not attributes for modules is that
they're *always* a PITA to deal with and just not worth the hassle.

Better to just expose it as a read-only attribute on class objects (that
isn't accessible through instances) and set it to None on construction if
order info is not available. This will also block accidental inheritance of
the attribute from an ordered base class.

A new keyword-only argument for the dynamic type construction APIs in the
"types" module would also be appropriate, and would allow the creation of
ordered classes even if the implementation used unordered namespaces for
class syntax.

Cheers,
Nick.

>
> -eric
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130628/9abeaac1/attachment.html>

From ben+python at benfinney.id.au  Fri Jun 28 03:25:00 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 28 Jun 2013 11:25:00 +1000
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
	formatting (PEP 257)
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<20130627102143.65125bc3@pitrou.net>
Message-ID: <7wppv7kowj.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

> I don't always find it easy to summarize a function in one line.

Difficulty in coming up with an accurate one-line summary of the intent
of a function is a code smell: the function probably needs to be
simplified so it is easier to describe accurately.

This is, IMO, one of the primary benefits of having a convention of
expecting a one-line summary for every function (and module and class).

-- 
 \          ?Courage is not the absence of fear, but the decision that |
  `\     something else is more important than fear.? ?Ambrose Redmoon |
_o__)                                                                  |
Ben Finney


From stephen at xemacs.org  Fri Jun 28 11:14:41 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 28 Jun 2013 18:14:41 +0900
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <kqi02e$20h$1@ger.gmane.org>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
	<51CC5599.9010305@hastings.org>
	<CAP7+vJKENEEp0vwx=fNdsm-eNvMN3=neYVF_L6obJDYkVpa+WA@mail.gmail.com>
	<kqhptb$oms$1@ger.gmane.org>
	<CAP7+vJLh0jEnE_KHW49xAeHy1cq9+42wjBEJJg74P1AV_MQs0Q@mail.gmail.com>
	<kqi02e$20h$1@ger.gmane.org>
Message-ID: <87li5uy4u6.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > Ok, I won't add them when a function's name actually makes what it does 
 > obvious. But when I have to spend at least a few minutes reading the 
 > body to make sense of it, I will try to add the summary line that I wish 
 > had been there already.

+1  That's a *great* rule-of-thumb!

From pynix.wang at gmail.com  Fri Jun 28 16:31:29 2013
From: pynix.wang at gmail.com (Pynix Wang)
Date: Fri, 28 Jun 2013 22:31:29 +0800
Subject: [Python-Dev] add new lambda syntax
Message-ID: <CAOSo4hNFYZjddQJTCZ3Ns5UVSqX1jC=dD9Mni0Rk8-PvvniAvQ@mail.gmail.com>

I want use coffeescript function syntax to write python lambda expression
so I modified the Grammar file.

```
atom: ('(' [yield_expr|testlist_comp|vararglist] ')' |
       '[' [testlist_comp] ']' |
       '{' [dictorsetmaker] '}' |
       NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text
```

but when I write
```
(x,y=1)->x+y
```
the parser doesn't go into vararglist.

any help?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130628/60ed2cc9/attachment.html>

From status at bugs.python.org  Fri Jun 28 18:07:36 2013
From: status at bugs.python.org (Python tracker)
Date: Fri, 28 Jun 2013 18:07:36 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20130628160736.1193F56A4C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2013-06-21 - 2013-06-28)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4069 (+12)
  closed 26027 (+34)
  total  30096 (+46)

Open issues with patches: 1804 


Issues opened (40)
==================

#7732: imp.find_module crashes Python if there exists a directory nam
http://bugs.python.org/issue7732  reopened by haypo

#18135: _ssl module: possible integer overflow for very long strings (
http://bugs.python.org/issue18135  reopened by pitrou

#18267: xmlrpc.client documentation multicall example missleading for 
http://bugs.python.org/issue18267  reopened by ber

#18279: IDLE Unit test for RstripExtension.py
http://bugs.python.org/issue18279  opened by philwebster

#18280: Documentation is too personalized
http://bugs.python.org/issue18280  opened by serhiy.storchaka

#18281: tarfile defines stat constants
http://bugs.python.org/issue18281  opened by christian.heimes

#18283: shutil.which() should support bytes
http://bugs.python.org/issue18283  opened by Arfrever

#18286: Python 3.3 - Slowing down computer
http://bugs.python.org/issue18286  opened by jon_irenicus

#18287: PyType_Ready() should sanity-check the tp_name field
http://bugs.python.org/issue18287  opened by nkoep

#18288: Idle 2.7: Run Module does not set __file__
http://bugs.python.org/issue18288  opened by terry.reedy

#18289: python.org Interactive interpreter linked with libedit can seg
http://bugs.python.org/issue18289  opened by Max.Kaye

#18290: json encoder does not support JSONP/JavaScript safe escaping
http://bugs.python.org/issue18290  opened by Ztane

#18291: codecs.open interprets space as line ends
http://bugs.python.org/issue18291  opened by wpk

#18292: IDLE Improvements: Unit test for AutoExpand.py
http://bugs.python.org/issue18292  opened by JayKrish

#18293: ssl.wrap_socket (cert_reqs=...), getpeercert, and unvalidated 
http://bugs.python.org/issue18293  opened by mpb

#18294: zlib module is not completly 64-bit safe
http://bugs.python.org/issue18294  opened by haypo

#18295: Possible integer overflow in PyCode_New()
http://bugs.python.org/issue18295  opened by haypo

#18296: test_os.test_trailers() is failing on AMD64 FreeBSD 9.0 dtrace
http://bugs.python.org/issue18296  opened by haypo

#18299: Change script_helper to use universal_newlines=True in _assert
http://bugs.python.org/issue18299  opened by r.david.murray

#18300: script_helper._asert_python should set TERM='' by default.
http://bugs.python.org/issue18300  opened by r.david.murray

#18301: In itertools.chain.from_iterable() there is no cls argument
http://bugs.python.org/issue18301  opened by py.user

#18302: test_multiprocessing: test.support.import_module() does not ig
http://bugs.python.org/issue18302  opened by haypo

#18304: ElementTree gets awkward to use if there is an xmlns
http://bugs.python.org/issue18304  opened by brycenesbitt

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305  opened by Sergey

#18306: os.stat gives exception for Windows junctions in v3.3
http://bugs.python.org/issue18306  opened by John.Jefferies

#18307: Relative path in co_filename for zipped modules
http://bugs.python.org/issue18307  opened by vmurashev

#18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures)
http://bugs.python.org/issue18308  opened by David.Edelsohn

#18309: Make python slightly more relocatable
http://bugs.python.org/issue18309  opened by mathias

#18312: "make distclean" deletes files under .hg directory
http://bugs.python.org/issue18312  opened by eric.smith

#18313: In itertools recipes repeatfunc() defines a non-keyword argume
http://bugs.python.org/issue18313  opened by py.user

#18314: Have os.unlink remove junction points
http://bugs.python.org/issue18314  opened by Kim.Gr??sman

#18315: bufsize parameter not documented in 2.7.5
http://bugs.python.org/issue18315  opened by Terrel.Shumway

#18316: Idle 2.7: update to simplify cross-version patches
http://bugs.python.org/issue18316  opened by terry.reedy

#18317: gettext: DoS via crafted Plural-Forms
http://bugs.python.org/issue18317  opened by jwilk

#18318: Idle: stop depending on console output
http://bugs.python.org/issue18318  opened by terry.reedy

#18319: gettext() cannot find translations with plural forms
http://bugs.python.org/issue18319  opened by jwilk

#18320: idle, pydoc search python on wrong path at make altinstall wit
http://bugs.python.org/issue18320  opened by cosmicduck

#18321: Multivolume support in tarfile module
http://bugs.python.org/issue18321  opened by edulix

#18322: test_stat nits
http://bugs.python.org/issue18322  opened by pitrou

#18323: 'GzipFile' object has no attribute 'extrastart'
http://bugs.python.org/issue18323  opened by Laurent.Gautier



Most recent 15 issues with no replies (15)
==========================================

#18320: idle, pydoc search python on wrong path at make altinstall wit
http://bugs.python.org/issue18320

#18319: gettext() cannot find translations with plural forms
http://bugs.python.org/issue18319

#18318: Idle: stop depending on console output
http://bugs.python.org/issue18318

#18307: Relative path in co_filename for zipped modules
http://bugs.python.org/issue18307

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305

#18304: ElementTree gets awkward to use if there is an xmlns
http://bugs.python.org/issue18304

#18302: test_multiprocessing: test.support.import_module() does not ig
http://bugs.python.org/issue18302

#18300: script_helper._asert_python should set TERM='' by default.
http://bugs.python.org/issue18300

#18299: Change script_helper to use universal_newlines=True in _assert
http://bugs.python.org/issue18299

#18292: IDLE Improvements: Unit test for AutoExpand.py
http://bugs.python.org/issue18292

#18273: Simplify calling and discovery of json test package
http://bugs.python.org/issue18273

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258

#18254: Accessing attr dict at definition time of a class from a metac
http://bugs.python.org/issue18254

#18243: mktime_tz documentation out-of-date
http://bugs.python.org/issue18243



Most recent 15 issues waiting for review (15)
=============================================

#18322: test_stat nits
http://bugs.python.org/issue18322

#18321: Multivolume support in tarfile module
http://bugs.python.org/issue18321

#18317: gettext: DoS via crafted Plural-Forms
http://bugs.python.org/issue18317

#18315: bufsize parameter not documented in 2.7.5
http://bugs.python.org/issue18315

#18312: "make distclean" deletes files under .hg directory
http://bugs.python.org/issue18312

#18309: Make python slightly more relocatable
http://bugs.python.org/issue18309

#18305: [patch] Fast sum() for non-numbers
http://bugs.python.org/issue18305

#18301: In itertools.chain.from_iterable() there is no cls argument
http://bugs.python.org/issue18301

#18294: zlib module is not completly 64-bit safe
http://bugs.python.org/issue18294

#18287: PyType_Ready() should sanity-check the tp_name field
http://bugs.python.org/issue18287

#18283: shutil.which() should support bytes
http://bugs.python.org/issue18283

#18279: IDLE Unit test for RstripExtension.py
http://bugs.python.org/issue18279

#18273: Simplify calling and discovery of json test package
http://bugs.python.org/issue18273

#18266: Fix test discovery for test_largefile.py
http://bugs.python.org/issue18266

#18258: Fix test discovery for test_codecmaps*.py
http://bugs.python.org/issue18258



Top 10 most discussed issues (10)
=================================

#16487: Allow ssl certificates to be specified from memory rather than
http://bugs.python.org/issue16487  17 msgs

#17206: Py_XDECREF() expands its argument multiple times
http://bugs.python.org/issue17206  15 msgs

#18234: Unicodedata module should provide access to codepoint aliases
http://bugs.python.org/issue18234  15 msgs

#11016: Re-implementation of the stat module in C
http://bugs.python.org/issue11016  14 msgs

#18236: str.isspace should use the Unicode White_Space property
http://bugs.python.org/issue18236  11 msgs

#18264: enum.IntEnum is not compatible with JSON serialisation
http://bugs.python.org/issue18264  10 msgs

#17621: Create a lazy import loader mixin
http://bugs.python.org/issue17621   9 msgs

#9566: Compilation warnings under x64 Windows
http://bugs.python.org/issue9566   8 msgs

#18135: _ssl module: possible integer overflow for very long strings (
http://bugs.python.org/issue18135   8 msgs

#18081: test_logging failure in WarningsTest on buildbots
http://bugs.python.org/issue18081   7 msgs



Issues closed (34)
==================

#1542: Ship 32 and 64bit libs with MSI installer
http://bugs.python.org/issue1542  closed by christian.heimes

#1953: Compact int and float freelists
http://bugs.python.org/issue1953  closed by christian.heimes

#4610: Unicode case mappings are incorrect
http://bugs.python.org/issue4610  closed by belopolsky

#9369: const char* for PyObject_CallMethod and PyObject_CallFunction
http://bugs.python.org/issue9369  closed by serhiy.storchaka

#11390: doctest: add cmdline parameters
http://bugs.python.org/issue11390  closed by r.david.murray

#11454: email.message import time
http://bugs.python.org/issue11454  closed by r.david.murray

#13483: Use VirtualAlloc to allocate memory arenas
http://bugs.python.org/issue13483  closed by loewis

#14264: Comparison bug in distutils2.version
http://bugs.python.org/issue14264  closed by eric.araujo

#14360: email.encoders.encode_quopri doesn't work with python 3.2
http://bugs.python.org/issue14360  closed by r.david.murray

#15223: datetime instances lack __module__ attribute
http://bugs.python.org/issue15223  closed by belopolsky

#15818: multiprocessing documentation of Process.exitcode
http://bugs.python.org/issue15818  closed by sbt

#16114: incorrect path in subprocess.Popen() FileNotFoundError message
http://bugs.python.org/issue16114  closed by gregory.p.smith

#17037: Use a test.support helper to wrap the PEP 399 boilerplate
http://bugs.python.org/issue17037  closed by eric.snow

#17985: multiprocessing Queue.qsize() and Queue.empty() with different
http://bugs.python.org/issue17985  closed by sbt

#18137: format(float, str): integer overflow for huge precision
http://bugs.python.org/issue18137  closed by haypo

#18143: ssl.get_default_verify_paths()
http://bugs.python.org/issue18143  closed by christian.heimes

#18164: Embedding Python doc incorrectly refers to LINKFORSHARED
http://bugs.python.org/issue18164  closed by ned.deily

#18179: SMTP.local_hostname is undocumented
http://bugs.python.org/issue18179  closed by r.david.murray

#18184: Add range check  for %c in PyUnicode_FromFormat
http://bugs.python.org/issue18184  closed by serhiy.storchaka

#18220: Expand itertools.islice docstring signature to 2 lines
http://bugs.python.org/issue18220  closed by python-dev

#18222: os.path.abspath should accept multiple path parts and join the
http://bugs.python.org/issue18222  closed by terry.reedy

#18246: tkinter.Text() add a newline to the content - bug?
http://bugs.python.org/issue18246  closed by terry.reedy

#18260: configparser: TypeError occurs when handling errors in files w
http://bugs.python.org/issue18260  closed by lukasz.langa

#18277: Queue is empty right after put from the same process/thread
http://bugs.python.org/issue18277  closed by sbt

#18278: Clarify the loader requirements for importlib.machinery.FileFi
http://bugs.python.org/issue18278  closed by brett.cannon

#18282: Ugly behavior of binary and unicode handling on reading unknow
http://bugs.python.org/issue18282  closed by r.david.murray

#18284: Doc make html: KeyError: 'howto/advocacy'
http://bugs.python.org/issue18284  closed by r.david.murray

#18285: In itertools.product() add argument repeat to the docstring
http://bugs.python.org/issue18285  closed by python-dev

#18297: In random.sample() correct the ValueError message for negative
http://bugs.python.org/issue18297  closed by mark.dickinson

#18298: pythonw.exe fails with redirected stderr
http://bugs.python.org/issue18298  closed by amaury.forgeotdarc

#18303: json.dumps() claims numpy.ndarray and numpy.bool_ are not seri
http://bugs.python.org/issue18303  closed by ethan.furman

#18310: itertools.tee() can't accept keyword arguments
http://bugs.python.org/issue18310  closed by rhettinger

#18311: Typo in SSL documentation
http://bugs.python.org/issue18311  closed by r.david.murray

#1040439: Missing documentation on how to link with libpython
http://bugs.python.org/issue1040439  closed by ned.deily

From amauryfa at gmail.com  Fri Jun 28 19:01:47 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 28 Jun 2013 19:01:47 +0200
Subject: [Python-Dev] add new lambda syntax
In-Reply-To: <CAOSo4hNFYZjddQJTCZ3Ns5UVSqX1jC=dD9Mni0Rk8-PvvniAvQ@mail.gmail.com>
References: <CAOSo4hNFYZjddQJTCZ3Ns5UVSqX1jC=dD9Mni0Rk8-PvvniAvQ@mail.gmail.com>
Message-ID: <CAGmFidbY4qaKDhmGMsnS02ZhC4C8n-F5MZgf=jXTFVzJgzBapw@mail.gmail.com>

2013/6/28 Pynix Wang <pynix.wang at gmail.com>

> I want use coffeescript function syntax to write python lambda expression
> so I modified the Grammar file.
>
> ```
> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' |
>        '[' [testlist_comp] ']' |
>        '{' [dictorsetmaker] '}' |
>        NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text
> ```
>
> but when I write
> ```
> (x,y=1)->x+y
> ```
> the parser doesn't go into vararglist.
>

This grammar is not LL(1) anymore (it's probably LALR now)
when seeing "x", it has the choice between testlist_comp and vararglist,
and the first one is picked.
Python's parser generator only supports LL(1) grammars.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130628/a191a82a/attachment.html>

From thomas at python.org  Fri Jun 28 19:39:02 2013
From: thomas at python.org (Thomas Wouters)
Date: Fri, 28 Jun 2013 19:39:02 +0200
Subject: [Python-Dev] add new lambda syntax
In-Reply-To: <CAGmFidbY4qaKDhmGMsnS02ZhC4C8n-F5MZgf=jXTFVzJgzBapw@mail.gmail.com>
References: <CAOSo4hNFYZjddQJTCZ3Ns5UVSqX1jC=dD9Mni0Rk8-PvvniAvQ@mail.gmail.com>
	<CAGmFidbY4qaKDhmGMsnS02ZhC4C8n-F5MZgf=jXTFVzJgzBapw@mail.gmail.com>
Message-ID: <CAPdQG2oW6u0hKFTV6tQo+wr1PGhdGf=X2Ah3_ixN01WVEdrkwA@mail.gmail.com>

On Fri, Jun 28, 2013 at 7:01 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> 2013/6/28 Pynix Wang <pynix.wang at gmail.com>
>
>> I want use coffeescript function syntax to write python lambda expression
>> so I modified the Grammar file.
>>
>> ```
>> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' |
>>        '[' [testlist_comp] ']' |
>>        '{' [dictorsetmaker] '}' |
>>        NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
>> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text
>> ```
>>
>> but when I write
>> ```
>> (x,y=1)->x+y
>> ```
>> the parser doesn't go into vararglist.
>>
>
> This grammar is not LL(1) anymore (it's probably LALR now)
> when seeing "x", it has the choice between testlist_comp and vararglist,
> and the first one is picked.
> Python's parser generator only supports LL(1) grammars.
>

Indeed. You may be able to make this work, but you'd have to fold the bits
of vararglist you need into testlist_comp, then reject invalid syntax that
matches the grammar (like "(x=expr for ...)" or "((x+1)=expr)" or
"(*foo=1)") in the compiler. Something like (untested):

testlist_comp: (test|star_expr) ['=' test]( comp_for | (','
(test|star_expr) ['=' test])* [','] )

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130628/21268a54/attachment.html>

From jimjjewett at gmail.com  Fri Jun 28 22:07:50 2013
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Fri, 28 Jun 2013 13:07:50 -0700 (PDT)
Subject: [Python-Dev]  backported Enum
In-Reply-To: <20130618230403.4cf9e657@anarchist>
Message-ID: <51cded16.4267320a.7d7d.0586@mx.google.com>


 
(On June 19, 2013) Barry Warsaw wrote about porting mailman from
flufl.enum to the stdlib.enum:


> Switching from call syntax to getitem syntax for looking up an
> enum member by name, e.g.

>    -        delivery_mode = DeliveryMode(data['delivery_mode'])
>    +        delivery_mode = DeliveryMode[data['delivery_mode']]

> Switching from getitem syntax to call syntax for looking up an
> enum member by value, e.g.

>    -        return self._enum[value]
>    +        return self._enum(value)

> Interesting that these two were exactly opposite from flufl.enum.

Is there a reason why these were reversed?

I can sort of convince myself that it makes sense because dicts
work better with strings than with ints, but ... it seems like
such a minor win that I'm not sure it is worth backwards
incompatibility.  (Of course, I also don't know how much use
stdlib.enum has already gotten with the current syntax.)



> Switching from int() to .value to get the integer value of an
> enum member, e.g.

>    -    return (member.list_id, member.address.email, int(member.role))
>    +    return (member.list_id, member.address.email, member.role.value)

Is just this a style preference?

Using a .value attribute certainly makes sense, but I don't see it
mentioned in the PEP as even optional, let alone recommended.  If
you care that the value be specifically an int (as opposed to any
object), then a int constructor may be better.

> [Some additional changes that mean there will be *some* changes,
> which does reduce the pressure for backwards compatibility.] ...


> An unexpected difference is that failing name lookups raise a
> KeyError instead of a ValueError.

I could understand either, as well as AttributeError, since the
instance that would represent that value isn't a class attribute.

Looking at Enum creation, I think ValueError would be better than
TypeError for complaints about duplicate names.  Was TypeError
chosen because it should only happen during setup?

I would also not be shocked if some people expect failed value
lookups to raise an IndexError, though I expect they would
adapt if they get something else that makes sense.

Would it be wrong to create an EnumError that subclasses
(ValueError, KeyError, AttributeError) and to raise that
subclass from everything but _StealthProperty and _get_mixins?


-jJ

-- 

If there are still threading problems with my replies, please 
email me with details, so that I can try to resolve them.  -jJ


From barry at python.org  Fri Jun 28 22:20:27 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 28 Jun 2013 16:20:27 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <51cded16.4267320a.7d7d.0586@mx.google.com>
References: <20130618230403.4cf9e657@anarchist>
	<51cded16.4267320a.7d7d.0586@mx.google.com>
Message-ID: <20130628162027.44a0607b@anarchist>

On Jun 28, 2013, at 01:07 PM, Jim J. Jewett wrote:

>> Switching from getitem syntax to call syntax for looking up an
>> enum member by value, e.g.
>
>>    -        return self._enum[value]
>>    +        return self._enum(value)
>
>> Interesting that these two were exactly opposite from flufl.enum.
>
>Is there a reason why these were reversed?

The individual decisions made (begrudging) sense at the time, although I don't
think it was noticed that the combined effect was to switch the meanings.

Call syntax makes sense for converting a value to an enum because that mirrors
calling built-in types, e.g. int('7').  Getitem syntax makes sense for
lookup-by-name.

>> Switching from int() to .value to get the integer value of an
>> enum member, e.g.
>
>>    -    return (member.list_id, member.address.email, int(member.role))
>>    +    return (member.list_id, member.address.email, member.role.value)
>
>Is just this a style preference?

No.  I'm not using IntEnums and regular enums don't have an __int__, which
makes sense because their values can be anything.  .value is the way to get at
the value.  (Note that even though I don't use IntEnums, my values are all
integers.  It's just not worth it to change all my imports.)

-Barry

From ethan at stoneleaf.us  Fri Jun 28 22:30:14 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 28 Jun 2013 13:30:14 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <51cded16.4267320a.7d7d.0586@mx.google.com>
References: <51cded16.4267320a.7d7d.0586@mx.google.com>
Message-ID: <51CDF256.30104@stoneleaf.us>

On 06/28/2013 01:07 PM, Jim J. Jewett wrote:
> (On June 19, 2013) Barry Warsaw wrote about porting mailman from
> flufl.enum to the stdlib.enum:
>>
>> Switching from call syntax to getitem syntax for looking up an
>> enum member by name, e.g.
>
>>     -        delivery_mode = DeliveryMode(data['delivery_mode'])
>>     +        delivery_mode = DeliveryMode[data['delivery_mode']]
>>
>> Switching from getitem syntax to call syntax for looking up an
>> enum member by value, e.g.
>>
>>     -        return self._enum[value]
>>     +        return self._enum(value)
>>
>> Interesting that these two were exactly opposite from flufl.enum.
>
> Is there a reason why these were reversed?

Originally the call syntax was used for both value and name lookup.  Various folks were unhappy with that arrangement, 
and since the use-case that I was thinking of at the time was getting enum members back from databases, etc., I went 
with EnumClass(value); I still wanted to be able to use name lookups, so lobbied for getitem syntax for that.  Nobody 
noticed this was the opposite of flufl.enum.

Oh, I say "I", and it is certainly my reasons, but I no longer remember if was me that initially proposed those specific 
ideas, and there were certainly many others that agreed.


>> Switching from int() to .value to get the integer value of an
>> enum member, e.g.
>>
>>     -    return (member.list_id, member.address.email, int(member.role))
>>     +    return (member.list_id, member.address.email, member.role.value)
>
> Is just this a style preference?

Nope.  If you want the exact value, accessing `.value` is the way to get it.


> Using a .value attribute certainly makes sense, but I don't see it
> mentioned in the PEP as even optional, let alone recommended.

I'll look again at the PEP and the docs and make sure that point is clear.

> If you care that the value be specifically an int (as opposed to any
> object), then a int constructor may be better.

Not entirely sure what you mean by this?

Had it been me, I would have subclassed Enum (as, say, FluflEnum) and added `__int__` to it so that those lines would 
have remained the same.


>> [Some additional changes that mean there will be *some* changes,
>> which does reduce the pressure for backwards compatibility.] ...
>>
>>
>> An unexpected difference is that failing name lookups raise a
>> KeyError instead of a ValueError.
>
> I could understand either, as well as AttributeError, since the
> instance that would represent that value isn't a class attribute.
>
> Looking at Enum creation, I think ValueError would be better than
> TypeError for complaints about duplicate names.  Was TypeError
> chosen because it should only happen during setup?

Yes.  That particular error can only happen during setup.


> I would also not be shocked if some people expect failed value
> lookups to raise an IndexError, though I expect they would
> adapt if they get something else that makes sense.
>
> Would it be wrong to create an EnumError that subclasses
> (ValueError, KeyError, AttributeError) and to raise that
> subclass from everything but _StealthProperty and _get_mixins?

Wouldn't bother me; I'm not sure what the bar is for adding new exceptions, though.

--
~Ethan~

From guido at python.org  Fri Jun 28 23:11:59 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 28 Jun 2013 14:11:59 -0700
Subject: [Python-Dev] backported Enum
In-Reply-To: <51CDF256.30104@stoneleaf.us>
References: <51cded16.4267320a.7d7d.0586@mx.google.com>
	<51CDF256.30104@stoneleaf.us>
Message-ID: <CAP7+vJLcDsGfuU9duPOQv+Pgz8K+OyupLK29ii-RmUaWfW7TXg@mail.gmail.com>

On Fri, Jun 28, 2013 at 1:30 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 06/28/2013 01:07 PM, Jim J. Jewett wrote:
>> (On June 19, 2013) Barry Warsaw wrote about porting mailman from
>>> Switching from getitem syntax to call syntax for looking up an
>>> enum member by value, e.g.
>>>
>>>     -        return self._enum[value]
>>>     +        return self._enum(value)
>>>
>>> Interesting that these two were exactly opposite from flufl.enum.

>> Is there a reason why these were reversed?

> Originally the call syntax was used for both value and name lookup.  Various
> folks were unhappy with that arrangement, and since the use-case that I was
> thinking of at the time was getting enum members back from databases, etc.,
> I went with EnumClass(value); I still wanted to be able to use name lookups,
> so lobbied for getitem syntax for that.  Nobody noticed this was the
> opposite of flufl.enum.
>
> Oh, I say "I", and it is certainly my reasons, but I no longer remember if
> was me that initially proposed those specific ideas, and there were
> certainly many others that agreed.

I also have a strong preference for the new way of doing this. In
Python, type(arg) is usually a constructor or conversion/cast, and
that maps reasonably well on what we're doing here with enums --
taking a value and turning it into the corresponding enum (which still
shares some properties with that value, and more if it's an IntEnum).
Also, this is a no-op (technically, the identity function) if the arg
is already a member of that enum.

On the other hand, Color['red'] feels more like a lookup -- this is
not so much rhyming with the behavior of other classes (classes are
rarely mappings) but still the mnemonic tip "foo[bar] is a lookup"
should help in remembering the meaning.

I have no idea why it was the other way around in flufl.enum, but
admittedly neither of these rules are absolute, and it's possible that
flufl.enum just evolved that way without conscious decisions, or that
it came from taking a different perspective.

-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Fri Jun 28 23:18:30 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 28 Jun 2013 17:18:30 -0400
Subject: [Python-Dev] backported Enum
In-Reply-To: <CAP7+vJLcDsGfuU9duPOQv+Pgz8K+OyupLK29ii-RmUaWfW7TXg@mail.gmail.com>
References: <51cded16.4267320a.7d7d.0586@mx.google.com>
	<51CDF256.30104@stoneleaf.us>
	<CAP7+vJLcDsGfuU9duPOQv+Pgz8K+OyupLK29ii-RmUaWfW7TXg@mail.gmail.com>
Message-ID: <20130628171830.4640d897@anarchist>

On Jun 28, 2013, at 02:11 PM, Guido van Rossum wrote:

>I have no idea why it was the other way around in flufl.enum, but
>admittedly neither of these rules are absolute, and it's possible that
>flufl.enum just evolved that way without conscious decisions, or that
>it came from taking a different perspective.

The latter, but I think PEP 435 made the right choices, at least for the
stdlib version.  It doesn't bother me that I had to switch them when porting.

-Barry

From tjreedy at udel.edu  Sat Jun 29 00:45:06 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 28 Jun 2013 18:45:06 -0400
Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring
 formatting (PEP 257)
In-Reply-To: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
References: <CAP7+vJKyONB1GK9tN=SuLiH=C9Uf7HQR1zm2m+5VMYY4pPN_Yw@mail.gmail.com>
Message-ID: <kql3lg$cv4$1@ger.gmane.org>

On 6/26/2013 9:56 PM, Guido van Rossum wrote:
> PEP 257 says this on the formatting of multi-line docstrings:
>
> """
> Multi-line docstrings consist of a summary line just like a one-line
> docstring, followed by a blank line, followed by a more elaborate
> description.

fileinput has docstrings like

     """
     Return the file number of the current file. When no file is currently
     opened, returns -1.
     """
and, perhaps neater,
     """
     Return the name of the file currently being read.
     Before the first line has been read, returns None.
     """

 From the above, I presume these should become

     """
     Return the file number of the current file.

     When no file is currently opened, returns -1.
     """
and
     """
     Return the name of the file currently being read.

     Before the first line has been read, returns None.
     """

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Sat Jun 29 12:21:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jun 2013 20:21:36 +1000
Subject: [Python-Dev] backported Enum
In-Reply-To: <51CDF256.30104@stoneleaf.us>
References: <51cded16.4267320a.7d7d.0586@mx.google.com>
	<51CDF256.30104@stoneleaf.us>
Message-ID: <CADiSq7cZNSzmuWENXk6mWktRW-98G8OwoUYroQZLNvzRR9Bg6Q@mail.gmail.com>

On 29 June 2013 06:30, Ethan Furman <ethan at stoneleaf.us> wrote:
>> I would also not be shocked if some people expect failed value
>> lookups to raise an IndexError, though I expect they would
>> adapt if they get something else that makes sense.
>>
>> Would it be wrong to create an EnumError that subclasses
>> (ValueError, KeyError, AttributeError) and to raise that
>> subclass from everything but _StealthProperty and _get_mixins?
>
>
> Wouldn't bother me; I'm not sure what the bar is for adding new exceptions,
> though.

I'd actually do something a bit more complex, but also cleaner from a
type system perspective:

    class EnumError(Exception): pass
    class EnumValueError(EnumError, ValueError): pass
    class EnumAttributeError(EnumError, AttributeError): pass
    class EnumKeyError(EnumError, KeyError): pass

However, it's probably not necessary. The value lookup API should just
document clearly which exception it throws.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From pynix.wang at gmail.com  Sat Jun 29 09:05:28 2013
From: pynix.wang at gmail.com (Pynix Wang)
Date: Sat, 29 Jun 2013 15:05:28 +0800
Subject: [Python-Dev] add new lambda syntax
In-Reply-To: <CAPdQG2oW6u0hKFTV6tQo+wr1PGhdGf=X2Ah3_ixN01WVEdrkwA@mail.gmail.com>
References: <CAOSo4hNFYZjddQJTCZ3Ns5UVSqX1jC=dD9Mni0Rk8-PvvniAvQ@mail.gmail.com>
	<CAGmFidbY4qaKDhmGMsnS02ZhC4C8n-F5MZgf=jXTFVzJgzBapw@mail.gmail.com>
	<CAPdQG2oW6u0hKFTV6tQo+wr1PGhdGf=X2Ah3_ixN01WVEdrkwA@mail.gmail.com>
Message-ID: <CAOSo4hP7PW78SP8n+jwk-sEWN9juP9H22Hyh7rODd4NyX8y2gw@mail.gmail.com>

http://aruld.info/lambda-expressions-in-java-8-adopts-c-style-syntax/

Lambda expressions in Java 8 adopts C# style syntax.


On Sat, Jun 29, 2013 at 1:39 AM, Thomas Wouters <thomas at python.org> wrote:

>
>
>
> On Fri, Jun 28, 2013 at 7:01 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:
>
>> 2013/6/28 Pynix Wang <pynix.wang at gmail.com>
>>
>>> I want use coffeescript function syntax to write python lambda
>>> expression so I modified the Grammar file.
>>>
>>> ```
>>> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' |
>>>        '[' [testlist_comp] ']' |
>>>        '{' [dictorsetmaker] '}' |
>>>        NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
>>> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text
>>> ```
>>>
>>> but when I write
>>> ```
>>> (x,y=1)->x+y
>>> ```
>>> the parser doesn't go into vararglist.
>>>
>>
>> This grammar is not LL(1) anymore (it's probably LALR now)
>> when seeing "x", it has the choice between testlist_comp and vararglist,
>> and the first one is picked.
>> Python's parser generator only supports LL(1) grammars.
>>
>
> Indeed. You may be able to make this work, but you'd have to fold the bits
> of vararglist you need into testlist_comp, then reject invalid syntax that
> matches the grammar (like "(x=expr for ...)" or "((x+1)=expr)" or
> "(*foo=1)") in the compiler. Something like (untested):
>
> testlist_comp: (test|star_expr) ['=' test]( comp_for | (','
> (test|star_expr) ['=' test])* [','] )
>
> --
> Thomas Wouters <thomas at python.org>
>
> Hi! I'm an email virus! Think twice before sending your email to help me
> spread!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130629/8f592588/attachment.html>

From nadeem.vawda at gmail.com  Sat Jun 29 17:16:46 2013
From: nadeem.vawda at gmail.com (Nadeem Vawda)
Date: Sat, 29 Jun 2013 17:16:46 +0200
Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and
 check for overflow
In-Reply-To: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
References: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
Message-ID: <CANF4RMnTiQAMne9RF8LYVj2s8XC3KenySK4k0Le8p_qpR7CLGA@mail.gmail.com>

On Thu, Jun 27, 2013 at 12:07 AM, Victor Stinner
<victor.stinner at gmail.com>wrote:

> I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
> 64-bit system:
> http://bugs.python.org/issue18294
>
> How should I implement that? Use "O" format and then use
> PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?
>

I ran into the same problem in the _lzma module. My solution was to define
a custom converter that does an explicit check before returning the value
(see http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134).

On Thu, Jun 27, 2013 at 12:26 AM, Guido van Rossum <guido at python.org> wrote:

> > I would to parse an integer in [0; UINT_MAX] to fix the zlib module on
> > 64-bit system:
> > http://bugs.python.org/issue18294
> >
> > How should I implement that? Use "O" format and then use
> > PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX?
>
> Why can't you use the K format? It won't reject out-of-range values,
> but it will convert them to in-range so there aren't any attacks
> possible based on bypassing the range check. I'm probably
> misunderstanding something -- I don't completely understand that bug
> report. :-(


The point is not to protect against deliberate attacks, but rather to fail
loudly (instead of silently) when the caller provides an input that the
underlying C library cannot handle.

- Nadeem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130629/59b000a4/attachment.html>

From storchaka at gmail.com  Sat Jun 29 18:24:11 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 29 Jun 2013 19:24:11 +0300
Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and
	check for overflow
In-Reply-To: <CANF4RMnTiQAMne9RF8LYVj2s8XC3KenySK4k0Le8p_qpR7CLGA@mail.gmail.com>
References: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
	<CANF4RMnTiQAMne9RF8LYVj2s8XC3KenySK4k0Le8p_qpR7CLGA@mail.gmail.com>
Message-ID: <kqn1n8$863$1@ger.gmane.org>

29.06.13 18:16, Nadeem Vawda ???????(??):
> I ran into the same problem in the _lzma module. My solution was to
> define a custom converter that does an explicit check before returning
> the value (see
> http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134).

Definitely Argument Clinic should use converters for unsigned integers 
without wraparound by default.



From tamir.fri at gmail.com  Sat Jun 29 17:59:13 2013
From: tamir.fri at gmail.com (Tamir Friedman)
Date: Sat, 29 Jun 2013 18:59:13 +0300
Subject: [Python-Dev] winsound.c fix to support python3
Message-ID: <CAA4a4fzr1Pn01MpOMN36iCgn7kzgPRUJ23k7+Tz=8UDtZHmrNw@mail.gmail.com>

Hello,
My name is Tamir Friedman, and I suggest to fix a bug in PlaySound in
winsound library. It's doesn't support the SND_MEMORY feature because its
accepts only "str" and rejects "bytes" type.
therefore i include the fixed source file:
OLD:
----------------------------------------------------------------------------
static PyObject *
sound_playsound(PyObject *s, PyObject *args)
{
    wchar_t *wsound;
    int flags;
    int ok;

    if (PyArg_ParseTuple(args, "Zi:PlaySound", &wsound, &flags)) {
        if (flags & SND_ASYNC && flags & SND_MEMORY) {
            /* Sidestep reference counting headache; unfortunately this also
               prevent SND_LOOP from memory. */
            PyErr_SetString(PyExc_RuntimeError, "Cannot play asynchronously
from memory");
            return NULL;
        }
        Py_BEGIN_ALLOW_THREADS
        ok = PlaySoundW(wsound, NULL, flags);
        Py_END_ALLOW_THREADS
        if (!ok) {
            PyErr_SetString(PyExc_RuntimeError, "Failed to play sound");
            return NULL;
        }
        Py_INCREF(Py_None);
        return Py_None;
    }
    return NULL;
}
----------------------------------------------------------------------------
NEW:
----------------------------------------------------------------------------
static PyObject *
sound_playsound(PyObject *s, PyObject *args)
{
    wchar_t *wsound;
    int flags;
    int ok;

    if (PyArg_ParseTuple(args, "z*i:PlaySound", &wsound, &flags)) {
        if (flags & SND_ASYNC && flags & SND_MEMORY) {
            /* Sidestep reference counting headache; unfortunately this also
               prevent SND_LOOP from memory. */
            PyErr_SetString(PyExc_RuntimeError, "Cannot play asynchronously
from memory");
            return NULL;
        }
        Py_BEGIN_ALLOW_THREADS
        ok = PlaySoundW(wsound, NULL, flags);
        Py_END_ALLOW_THREADS
        if (!ok) {
            PyErr_SetString(PyExc_RuntimeError, "Failed to play sound");
            return NULL;
        }
        Py_INCREF(Py_None);
        return Py_None;
    }
    return NULL;
}
----------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130629/8eb5f988/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OLD_winsound_OLD.c
Type: text/x-csrc
Size: 5581 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130629/8eb5f988/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winsound.c
Type: text/x-csrc
Size: 5637 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130629/8eb5f988/attachment-0001.c>

From rdmurray at bitdance.com  Sat Jun 29 21:29:09 2013
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 29 Jun 2013 15:29:09 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_cpython=3A_Fix_memor?=
	=?utf-8?q?y_leak_in_pyexpat_PyUnknownEncodingHandler?=
In-Reply-To: <3bjNyf4jvKz7Llt@mail.python.org>
References: <3bjNyf4jvKz7Llt@mail.python.org>
Message-ID: <20130629192910.333AD2500C5@webabinitio.net>

I don't know if this is the commit at fault or not, but we are seeing
segfaults in test_xml_etree on the buildbots now.

--David

On Sat, 29 Jun 2013 20:43:22 +0200, christian.heimes <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/bd0834b59828
> changeset:   84375:bd0834b59828
> user:        Christian Heimes <christian at cheimes.de>
> date:        Sat Jun 29 20:43:13 2013 +0200
> summary:
>   Fix memory leak in pyexpat PyUnknownEncodingHandler
> CID 1040367 (#1 of 1): Resource leak (RESOURCE_LEAK)
> leaked_storage: Variable u going out of scope leaks the storage it points to.
> 
> files:
>   Modules/pyexpat.c |  4 +++-
>   1 files changed, 3 insertions(+), 1 deletions(-)
> 
> 
> diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c
> --- a/Modules/pyexpat.c
> +++ b/Modules/pyexpat.c
> @@ -1128,8 +1128,10 @@
>      }
>  
>      u = PyUnicode_Decode((char*) template_buffer, 256, name, "replace");
> -    if (u == NULL || PyUnicode_READY(u))
> +    if (u == NULL || PyUnicode_READY(u)) {
> +        Py_DECREF(u);
>          return XML_STATUS_ERROR;
> +    }
>  
>      if (PyUnicode_GET_LENGTH(u) != 256) {
>          Py_DECREF(u);
> 
> -- 
> Repository URL: http://hg.python.org/cpython
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins

From christian at python.org  Sat Jun 29 21:51:19 2013
From: christian at python.org (Christian Heimes)
Date: Sat, 29 Jun 2013 21:51:19 +0200
Subject: [Python-Dev] cpython: Fix memory leak in pyexpat
	PyUnknownEncodingHandler
In-Reply-To: <20130629192910.333AD2500C5@webabinitio.net>
References: <3bjNyf4jvKz7Llt@mail.python.org>
	<20130629192910.333AD2500C5@webabinitio.net>
Message-ID: <51CF3AB7.8020308@python.org>

Am 29.06.2013 21:29, schrieb R. David Murray:
> I don't know if this is the commit at fault or not, but we are seeing
> segfaults in test_xml_etree on the buildbots now.

Yeah, it's my fault. Thanks!

>>      u = PyUnicode_Decode((char*) template_buffer, 256, name, "replace");
>> -    if (u == NULL || PyUnicode_READY(u))wr
>> +        Py_DECREF(u);
>>          return XML_STATUS_ERROR;
>> +    }

Py_DECREF() is wrong as u can be NULL. I have changed the code to use
Py_XDECREF().


From tjreedy at udel.edu  Sat Jun 29 22:20:59 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 29 Jun 2013 16:20:59 -0400
Subject: [Python-Dev] winsound.c fix to support python3
In-Reply-To: <CAA4a4fzr1Pn01MpOMN36iCgn7kzgPRUJ23k7+Tz=8UDtZHmrNw@mail.gmail.com>
References: <CAA4a4fzr1Pn01MpOMN36iCgn7kzgPRUJ23k7+Tz=8UDtZHmrNw@mail.gmail.com>
Message-ID: <kqnfjb$gln$1@ger.gmane.org>

On 6/29/2013 11:59 AM, Tamir Friedman wrote:
> Hello,
> My name is Tamir Friedman, and I suggest to fix a bug in PlaySound in
> winsound library. It's doesn't support the SND_MEMORY feature because
> its accepts only "str" and rejects "bytes" type.
> therefore i include the fixed source file:

Thank you for tracking down the source of a problem. Please make an 
account at bugs.python.org and open a tracker issue. A post here will 
get lost. On the issue, either upload a patch file or describe the 
change. As near as I can tell, the description for your change might be:

"winsound.c, line NN in sound_playsound function, is currently
       if (PyArg_ParseTuple(args, "Zi:PlaySound", &wsound, &flags)) {
I think 'Zi' should be changed for 'z*i' because ..."


-- 
Terry Jan Reedy


From amk at amk.ca  Sun Jun 30 00:00:23 2013
From: amk at amk.ca (A.M. Kuchling)
Date: Sat, 29 Jun 2013 18:00:23 -0400
Subject: [Python-Dev] Issue 4199: combining assignment with global & nonlocal
Message-ID: <20130629220023.GA66048@datlandrewk.home>

Issue 4199 begins with a self-explanatory comment:

      PEP 3104 says that the nonlocal and global statements should
      allow a shorthand. ("global x; x = 3" == "global x = 3") This
      patch implements that.

Benjamin posted his patch on 2008-10-24.  It got postponed to 3.2
because it was too late for 3.1, and then to 3.3 because 3.2 still
fell under the language moratorium, and then was forgotten for 3.3.

Georg Brandl and Jeremy Hylton raised some issues with the patch and
with the PEP's specification:
http://bugs.python.org/issue4199#msg100015 

We've gone five years in 3.x without supporting this statement form.
Is it still worth implementing it for 3.4?  Or should the issue simply
be closed (and maybe a note added to the PEP that "we didn't do this
after all")?

--amk

From larry at hastings.org  Sun Jun 30 00:03:25 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 30 Jun 2013 00:03:25 +0200
Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and
 check for overflow
In-Reply-To: <kqn1n8$863$1@ger.gmane.org>
References: <CAMpsgwYSd_q25Nb3mOUrWrSr8POGcqC3SP3mxNx72ezy4pLyaA@mail.gmail.com>
	<CANF4RMnTiQAMne9RF8LYVj2s8XC3KenySK4k0Le8p_qpR7CLGA@mail.gmail.com>
	<kqn1n8$863$1@ger.gmane.org>
Message-ID: <51CF59AD.2070301@hastings.org>



On 06/29/2013 06:24 PM, Serhiy Storchaka wrote:
> Definitely Argument Clinic should use converters for unsigned integers 
> without wraparound by default.

Argument Clinic 1.0 is a thin layer over PyArg_ParseTuple.  But it will 
support these sorts of converters, and if people have bright ideas about 
converters to include in-the-box that's totally fine.

but-nothing-has-been-formally-accepted-yet-ly yours,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130630/a4549dcb/attachment.html>

From guido at python.org  Sun Jun 30 02:15:05 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 29 Jun 2013 17:15:05 -0700
Subject: [Python-Dev] Issue 4199: combining assignment with global &
	nonlocal
In-Reply-To: <20130629220023.GA66048@datlandrewk.home>
References: <20130629220023.GA66048@datlandrewk.home>
Message-ID: <CAP7+vJ+idJvjwnBwUoktvqR46fxTEEMrCXRfC8O2u-=AsfrUDw@mail.gmail.com>

I agree that there are problems with the idea of combining assignment
and global statements, because the ambiguity of "global x, y = p, q"
is just too much to handle. The only case would be if there was only a
single variable, but that fails the test of easy generalization. So
let's close it.

--Guido

On Sat, Jun 29, 2013 at 3:00 PM, A.M. Kuchling <amk at amk.ca> wrote:
> Issue 4199 begins with a self-explanatory comment:
>
>       PEP 3104 says that the nonlocal and global statements should
>       allow a shorthand. ("global x; x = 3" == "global x = 3") This
>       patch implements that.
>
> Benjamin posted his patch on 2008-10-24.  It got postponed to 3.2
> because it was too late for 3.1, and then to 3.3 because 3.2 still
> fell under the language moratorium, and then was forgotten for 3.3.
>
> Georg Brandl and Jeremy Hylton raised some issues with the patch and
> with the PEP's specification:
> http://bugs.python.org/issue4199#msg100015
>
> We've gone five years in 3.x without supporting this statement form.
> Is it still worth implementing it for 3.4?  Or should the issue simply
> be closed (and maybe a note added to the PEP that "we didn't do this
> after all")?
>
> --amk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From benjamin at python.org  Sun Jun 30 02:22:44 2013
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 29 Jun 2013 17:22:44 -0700
Subject: [Python-Dev] Issue 4199: combining assignment with global &
	nonlocal
In-Reply-To: <CAP7+vJ+idJvjwnBwUoktvqR46fxTEEMrCXRfC8O2u-=AsfrUDw@mail.gmail.com>
References: <20130629220023.GA66048@datlandrewk.home>
	<CAP7+vJ+idJvjwnBwUoktvqR46fxTEEMrCXRfC8O2u-=AsfrUDw@mail.gmail.com>
Message-ID: <CAPZV6o8uHr7U=jO3NMNG_TnYo57q=f0VK2j=wf=pJPFL2qZ+WA@mail.gmail.com>

Agreed. The currently supported syntax (a global/nonlocal line and an
assignment) is not too painful to figure out the ambiguity of the
proposed syntax.

2013/6/29 Guido van Rossum <guido at python.org>:
> I agree that there are problems with the idea of combining assignment
> and global statements, because the ambiguity of "global x, y = p, q"
> is just too much to handle. The only case would be if there was only a
> single variable, but that fails the test of easy generalization. So
> let's close it.
>
> --Guido
>
> On Sat, Jun 29, 2013 at 3:00 PM, A.M. Kuchling <amk at amk.ca> wrote:
>> Issue 4199 begins with a self-explanatory comment:
>>
>>       PEP 3104 says that the nonlocal and global statements should
>>       allow a shorthand. ("global x; x = 3" == "global x = 3") This
>>       patch implements that.
>>
>> Benjamin posted his patch on 2008-10-24.  It got postponed to 3.2
>> because it was too late for 3.1, and then to 3.3 because 3.2 still
>> fell under the language moratorium, and then was forgotten for 3.3.
>>
>> Georg Brandl and Jeremy Hylton raised some issues with the patch and
>> with the PEP's specification:
>> http://bugs.python.org/issue4199#msg100015
>>
>> We've gone five years in 3.x without supporting this statement form.
>> Is it still worth implementing it for 3.4?  Or should the issue simply
>> be closed (and maybe a note added to the PEP that "we didn't do this
>> after all")?
>>
>> --amk
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org



-- 
Regards,
Benjamin