From ericsnowcurrently at gmail.com  Sun Mar  1 02:30:55 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 28 Feb 2015 18:30:55 -0700
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
Message-ID: <CALFfu7DaLakdgwZZviM2xmCTJfmde+He=dJ9XBCSjWFeoNyqTQ@mail.gmail.com>

On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> That leaves my one concern being the "namespace" keyword parameter.
> I'd started reconsidering that for PEP 422, and I think the argument
> holds at least as strongly for your improved design:
> https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity
>
> Since speed usually isn't a critical concern for class definitions,
> it's likely better to just make them ordered by default, which can be
> its own PEP, rather than being part of this one.

FWIW, Guido already told me a while back that OrderedDict by default
for class definition namespace was okay.  It's just been waiting on me
to get back to the C OrderedDict implementation (which I've started to
recently).

-eric

From ncoghlan at gmail.com  Sun Mar  1 07:48:10 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Mar 2015 16:48:10 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CALFfu7DaLakdgwZZviM2xmCTJfmde+He=dJ9XBCSjWFeoNyqTQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CALFfu7DaLakdgwZZviM2xmCTJfmde+He=dJ9XBCSjWFeoNyqTQ@mail.gmail.com>
Message-ID: <CADiSq7cPatpZdZ3M93eeWFHqmhDPtmUS8CckgejhZ48s2yTcAA@mail.gmail.com>

On 1 March 2015 at 11:30, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> That leaves my one concern being the "namespace" keyword parameter.
>> I'd started reconsidering that for PEP 422, and I think the argument
>> holds at least as strongly for your improved design:
>> https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity
>>
>> Since speed usually isn't a critical concern for class definitions,
>> it's likely better to just make them ordered by default, which can be
>> its own PEP, rather than being part of this one.
>
> FWIW, Guido already told me a while back that OrderedDict by default
> for class definition namespace was okay.  It's just been waiting on me
> to get back to the C OrderedDict implementation (which I've started to
> recently).

Yeah, that was the conversation that prompted me to question the
utility of the namespace keyword idea. The prototype class and class
extension examples are cute, but fall into the "don't do this in
production" category for me, making them prime candidates for custom
metaclasses.

The one really useful one is using an ordered dictionary for namespace
execution, and your solution addresses that better by just making it
the normal behaviour.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From lkb.teichmann at gmail.com  Sun Mar  1 10:27:43 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Sun, 1 Mar 2015 10:27:43 +0100
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
Message-ID: <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>

> Aye, I like this design, it just wasn't until I saw it written out
> fully that it finally clicked why restricting it to subclasses let
> both zero-argument super() and explicit namespace access to the
> defining class both work. Getting the latter to also work is a
> definite strength of your approach over mine.

That said, I would love if it was just possible to use super() in
classmethods called from the __init__ of a metaclass. Until
now, the __class__ of a method is only initialized after
the metaclasses __init__ has run.

Fortunately, this is easily changeable: it is possible to move the
initialization of __class__ into type.__new__. As the class is
(typically) created there, that is about as early as possible.

I put the necessary changes to github:
https://github.com/tecki/cpython/commits/cell

> That leaves my one concern being the "namespace" keyword parameter.
> I'd started reconsidering that for PEP 422, and I think the argument
> holds at least as strongly for your improved design:
> https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity
>
> Since speed usually isn't a critical concern for class definitions,
> it's likely better to just make them ordered by default, which can be
> its own PEP, rather than being part of this one.

I feel the same way as you, so I am happy to drop the namespace
idea from PEP 487, but I agree that then ordered namespaces
would simply be great, but reading Eric's comment, that seems
just around the corner.

Greetings

Martin

From ncoghlan at gmail.com  Mon Mar  2 10:45:38 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Mar 2015 19:45:38 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
Message-ID: <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>

On 1 Mar 2015 19:28, "Martin Teichmann" <lkb.teichmann at gmail.com> wrote:
>
> > Aye, I like this design, it just wasn't until I saw it written out
> > fully that it finally clicked why restricting it to subclasses let
> > both zero-argument super() and explicit namespace access to the
> > defining class both work. Getting the latter to also work is a
> > definite strength of your approach over mine.
>
> That said, I would love if it was just possible to use super() in
> classmethods called from the __init__ of a metaclass. Until
> now, the __class__ of a method is only initialized after
> the metaclasses __init__ has run.
>
> Fortunately, this is easily changeable: it is possible to move the
> initialization of __class__ into type.__new__. As the class is
> (typically) created there, that is about as early as possible.

Unfortunately, it's too early - we can't control whether or not subclasses
actually return the result of type.__new__ from their own __new__
implementation, so that object could have been replaced by something else
by the time __init__ gets called.

However, it may be feasible to move it into type.__init__. That way class
methods would start working properly as soon as the metaclass __init__
implementation had called up into type.__init__.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150302/9593fbb3/attachment.html>

From lkb.teichmann at gmail.com  Mon Mar  2 11:57:09 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Mon, 2 Mar 2015 11:57:09 +0100
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
Message-ID: <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>

>> Fortunately, this is easily changeable: it is possible to move the
>> initialization of __class__ into type.__new__. As the class is
>> (typically) created there, that is about as early as possible.
>
> Unfortunately, it's too early - we can't control whether or not subclasses
> actually return the result of type.__new__ from their own __new__
> implementation, so that object could have been replaced by something else by
> the time __init__ gets called.

Well, this is actually just a question of definition and documentation.
Sure, a metaclasses __new__ may return whatever it pleases, but this
is not in contradiction of setting __class__ of the methods in type.__new__.

There are two possible scenarios:

1. metaclass.__new__ does not call type.__new__. Once we document that
it is type.__new__'s job to set __class__, nobody can complain if __class__
is not set if type.__new__ is not called, as this would be documented behavior.
In my changes I actually still left in the old code, so __build_class__ would
still try to set __class__ even if type.__new__ did not get called.

2. somehow type.__new__ does get called (most likely by metaclass.__new__).
It will do its job and set the __class__ of the methods of the class being
created. Whatever metaclass.__new__ returns then doesn't matter much,
because what sense would it make to set __class__ of that object it creates?

To give an example:
Imagine a soliton-generating metaclass:

    class Soliton(type):
       def __new__(cls, name, bases, ns):
           self = super().__new__(name, bases, ns)
           return self()

And generate such a soliton:

    class A(metaclass=Soliton):
        def f(self):
            print(__class__)

As of now, writing "A.f()" interestingly prints "<__main__.A object>", so
__class__ is indeed set to what Soliton.__new__ returns, the object,
not the class.

This is currently correct behavior, but I think it actually is not what one
expects, nor what one desires. (Does anyone out there make use of
such a construct? Please speak up!) super() certainly won't work.
So I think it would actually be a change for the better to let
type.__new__ set the __class__ of the generated class.

I know that it technically breaks backward compatibility. But I ran
all the tests and none failed, so apparently until now nobody got
weird ideas like me... I even tried to remove the old code from
__build_class__ and still all tests run. (just a note, if anyone else
here is trying to do so, I tampered with the compiler, so you better
delete your .pyc files before running the tests)

Greetings

Martin

From steve at pearwood.info  Mon Mar  2 12:46:59 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Mar 2015 22:46:59 +1100
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
Message-ID: <20150302114658.GF7655@ando.pearwood.info>

On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote:

> Imagine a soliton-generating metaclass:
> 
>     class Soliton(type):
>        def __new__(cls, name, bases, ns):
>            self = super().__new__(name, bases, ns)
>            return self()

You have to pass cls as an explicit argument to __new__, otherwise you 
get a TypeError:

TypeError: type.__new__(X): X is not a type object (str)

(that's in Python 3.3). The line should be:

self = super().__new__(cls, name, bases, ns)


 
> And generate such a soliton:
> 
>     class A(metaclass=Soliton):
>         def f(self):
>             print(__class__)
> 
> As of now, writing "A.f()" interestingly prints "<__main__.A object>", so
> __class__ is indeed set to what Soliton.__new__ returns, the object,
> not the class.

Is "soliton" the usual terminology for this? Do you perhaps mean 
singleton? I've googled for "soliton" but nothing relevant is coming up.

 
> This is currently correct behavior, but I think it actually is not what one
> expects, nor what one desires. (Does anyone out there make use of
> such a construct? Please speak up!) 

I do now! Seriously, I think you have just solved a major problem for 
me. I'll need to do some further experimentation, but that is almost 
exactly what I have been looking for. I've been looking for a way to 
have a class statement return a custom instance and I think this might 
just do it.


-- 
Steve

From ncoghlan at gmail.com  Mon Mar  2 13:21:44 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Mar 2015 22:21:44 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
Message-ID: <CADiSq7ezTcCeJ5itVm_2y5_UMnBLRqWw3hDqxrixQJxYi6vCcw@mail.gmail.com>

On 2 March 2015 at 20:57, Martin Teichmann <lkb.teichmann at gmail.com> wrote:
>>> Fortunately, this is easily changeable: it is possible to move the
>>> initialization of __class__ into type.__new__. As the class is
>>> (typically) created there, that is about as early as possible.
>>
>> Unfortunately, it's too early - we can't control whether or not subclasses
>> actually return the result of type.__new__ from their own __new__
>> implementation, so that object could have been replaced by something else by
>> the time __init__ gets called.
>
> Well, this is actually just a question of definition and documentation.
> Sure, a metaclasses __new__ may return whatever it pleases, but this
> is not in contradiction of setting __class__ of the methods in type.__new__.
>
> There are two possible scenarios:
>
> 1. metaclass.__new__ does not call type.__new__. Once we document that
> it is type.__new__'s job to set __class__, nobody can complain if __class__
> is not set if type.__new__ is not called, as this would be documented behavior.
> In my changes I actually still left in the old code, so __build_class__ would
> still try to set __class__ even if type.__new__ did not get called.
>
> 2. somehow type.__new__ does get called (most likely by metaclass.__new__).
> It will do its job and set the __class__ of the methods of the class being
> created. Whatever metaclass.__new__ returns then doesn't matter much,
> because what sense would it make to set __class__ of that object it creates?
>
> To give an example:
> Imagine a soliton-generating metaclass:
>
>     class Soliton(type):
>        def __new__(cls, name, bases, ns):
>            self = super().__new__(name, bases, ns)
>            return self()
>
> And generate such a soliton:
>
>     class A(metaclass=Soliton):
>         def f(self):
>             print(__class__)
>
> As of now, writing "A.f()" interestingly prints "<__main__.A object>", so
> __class__ is indeed set to what Soliton.__new__ returns, the object,
> not the class.
>
> This is currently correct behavior, but I think it actually is not what one
> expects, nor what one desires. (Does anyone out there make use of
> such a construct? Please speak up!) super() certainly won't work.
> So I think it would actually be a change for the better to let
> type.__new__ set the __class__ of the generated class.

Thinking about this a bit more, I think a good way to look at it is to
consider the impact of:

1. Using explicit super() rather than implicit super()
2. Using explicit decorators

Currently, in your example, "__class__" and "A" will both refer to an
*instance* of the class.

If, instead, you used a "singleton" explicit decorator, then
"__class__ " would refer to the actual class object, while "A" would
refer to the instance.

With your proposed change, then "__class__" will always refer to the
actual class object, even if the metaclass __new__ plays implicit
decoration games.

It also means that class methods that rely on __class__ (whether
explicitly or implicitly through super()) will "just work" in
metaclass __init__ and __new__ methods, although only after the call
up to the base class implementation in the latter case.

> I know that it technically breaks backward compatibility. But I ran
> all the tests and none failed, so apparently until now nobody got
> weird ideas like me... I even tried to remove the old code from
> __build_class__ and still all tests run. (just a note, if anyone else
> here is trying to do so, I tampered with the compiler, so you better
> delete your .pyc files before running the tests)

Aye, and the discrepancy I was concerned about can already happen when
using explicit decorators. That suggest to me that even if someone
*was* relying on __class__ pointing to the same object as the bound
name when a metaclass does something odd, they can likely switch to
referring to it by name instead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Mar  2 13:25:31 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Mar 2015 22:25:31 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <20150302114658.GF7655@ando.pearwood.info>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <20150302114658.GF7655@ando.pearwood.info>
Message-ID: <CADiSq7df0_fqEoDFTNxZExjxfBFYJCbG+iQHS4JoMcoPuzcbqA@mail.gmail.com>

On 2 March 2015 at 21:46, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote:
>> This is currently correct behavior, but I think it actually is not what one
>> expects, nor what one desires. (Does anyone out there make use of
>> such a construct? Please speak up!)
>
> I do now! Seriously, I think you have just solved a major problem for
> me. I'll need to do some further experimentation, but that is almost
> exactly what I have been looking for. I've been looking for a way to
> have a class statement return a custom instance and I think this might
> just do it.

Martin's question wasn't about using __new__ on the metaclass to do
factory function magic (that's "normal usage" as far as metaclasses
go). The "construct" in question here is:

1. Returning something from a metaclass __new__ implementation other
than the result of type.__new__; AND
2. Relying on __class__ to refer to the returned object, rather than
the result of type.__new__

That seems likely to be a vanishingly small subset of Python usage,
and even if it does happen, the subset that couldn't switch to using
explicit super() via the bound name is likely to be even smaller.

That seems like a risk well worth taking when the pay-off is having
class methods that use implicit super() just work when called from the
metaclass.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Mon Mar  2 13:45:22 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Mar 2015 22:45:22 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
Message-ID: <CADiSq7ei6HCyiQXdBPcjF=giK2zqpnAe1iWBY+G3C+2aYhzT+A@mail.gmail.com>

On 28 February 2015 at 02:56, Martin Teichmann <lkb.teichmann at gmail.com> wrote:
> Connection to other PEP
> =======================
>
> This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban.
> It shares both most of the PEP text and proposed code, but has major
> differences in how to achieve its goals.

I've now withdrawn PEP 422 in favour of this proposal:
https://hg.python.org/peps/rev/4db0c1a3e363 (main site hadn't updated
at time of writing, but it should propagate before too long)

> Proposal
> ========
>
> This PEP proposes that a new mechanism to customise class creation be
> added to Python 3.5 that meets the following criteria:
>
> 1. Integrates nicely with class inheritance structures (including mixins and
>    multiple inheritance),
> 2. Integrates nicely with the implicit ``__class__`` reference and
>    zero-argument ``super()`` syntax introduced by PEP 3135,
> 3. Can be added to an existing base class without a significant risk of
>    introducing backwards compatibility problems, and
> 4. Restores the ability for class namespaces to have some influence on the
>    class creation process (above and beyond populating the namespace itself),
>    but potentially without the full flexibility of the Python 2 style
>    ``__metaclass__`` hook.
>
> Those goals can be achieved by adding two functionalities:
>
> 1. A ``__init_subclass__`` hook that initializes all subclasses of a
>    given class, and

In writing the withdrawal notice for PEP 422, I realised there's a
case that needs to be more clearly specified here: the handling of
cases where the base class already defines the hook. Since object has
a default implementation, we should be able formally define this as
the type creation machinery calling:

    super(cls, cls).__init_subclass__(ns, **kwargs)

Rather than:

    cls.__init_subclass__(ns, **kwargs)

We may also want to consider defining object.__init_subclass__ as
accepting "cls, ns, *, metaclass=type" so we don't need to implicitly
pop metaclass from the kwargs dict when it gets specified.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From g.rodola at gmail.com  Tue Mar  3 01:46:51 2015
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Tue, 3 Mar 2015 01:46:51 +0100
Subject: [Python-ideas] Show deprecation warnings in the interactive
	interpreter
In-Reply-To: <mcjuuk$2pt$1@ger.gmane.org>
References: <mcjuuk$2pt$1@ger.gmane.org>
Message-ID: <CAFYqXL9Ai2Y1XwTFbrC63DZQTON_26QBTt_rx7Wfem61-0nFnw@mail.gmail.com>

On Wed, Feb 25, 2015 at 8:51 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> What you are think about turning deprecation warnings on by default in the
> interactive interpreter?
>
> Deprecation warnings are silent by default because they just annoys end
> user that uses old Python program with new Python. End user usually is not
> a programmer and can't do something with warnings. But in the interactive
> interpreter the user is the author of executed code and can avoid to use
> deprecated functions if warned.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>

Am I the only one thinking that disabling deprecation warnings by default
(2.6?) was a bad idea?
It's particularly counterproductive for library vendors and their users
because the latters have no visibility of what APIs are going to be
removed: they will just wake up one day, upgrade to the next major version
of their favorite lib and have their code broken.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/f346b02d/attachment-0001.html>

From guettliml at thomas-guettler.de  Tue Mar  3 09:33:18 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Tue, 03 Mar 2015 09:33:18 +0100
Subject: [Python-ideas] unittest.failureExeption(msg,
 data=None): add optional data kwarg to failureException() of
 unittest
Message-ID: <54F571CE.3040901@thomas-guettler.de>

Hi,

what do you think about the idea to add an optional data kwarg to unittest.failureExeption()?

Up to now a the exception class "failureException" does only get a message which is a string.

I would like to log the failed Exceptions to a structured logging back end.

Parsing the string message to extract the real value is possible, but somehow ugly and error prone.

My high level use case: Have a system (maybe ELK-Stack) to investigate if a test failure was seen before.

Doing this on strings could work, but I would like to keep the original (json serializable) values.

What do you think?

   Thomas G?ttler

From ncoghlan at gmail.com  Tue Mar  3 13:58:18 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 3 Mar 2015 22:58:18 +1000
Subject: [Python-ideas] Show deprecation warnings in the interactive
	interpreter
In-Reply-To: <CAFYqXL9Ai2Y1XwTFbrC63DZQTON_26QBTt_rx7Wfem61-0nFnw@mail.gmail.com>
References: <mcjuuk$2pt$1@ger.gmane.org>
 <CAFYqXL9Ai2Y1XwTFbrC63DZQTON_26QBTt_rx7Wfem61-0nFnw@mail.gmail.com>
Message-ID: <CADiSq7cJwWKZvarK9o=e3H7G_XMPP9Mp_kr5axZP9AAKYQisxw@mail.gmail.com>

On 3 March 2015 at 10:46, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
>
> On Wed, Feb 25, 2015 at 8:51 AM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
>>
>> What you are think about turning deprecation warnings on by default in the
>> interactive interpreter?
>>
>> Deprecation warnings are silent by default because they just annoys end
>> user that uses old Python program with new Python. End user usually is not a
>> programmer and can't do something with warnings. But in the interactive
>> interpreter the user is the author of executed code and can avoid to use
>> deprecated functions if warned.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> Am I the only one thinking that disabling deprecation warnings by default
> (2.6?) was a bad idea?
> It's particularly counterproductive for library vendors and their users
> because the latters have no visibility of what APIs are going to be removed:
> they will just wake up one day, upgrade to the next major version of their
> favorite lib and have their code broken.

Not really, because if you don't have automated regression tests, not
seeing the deprecation warnings from your dependencies is the least of
your worries - you can break your own stuff completely and have no
idea until after you ship the broken version.

If I ever find time to implement the PEP 432 changes to the startup
sequence (or a volunteer interested in trying to unknot some of the
most tangled C code we have left in the interpreter), then it might be
feasible to introduce a system wide config file that redistributors
and end users could use to decide whether to show deprecation warnings
by default or not.

In the meantime though, while the status quo has its flaws, I think
its less flawed than the previous situation where end users were
getting cryptic warnings they couldn't do anything about just because
they were running a tool that happened to be written in Python (e.g.
consider Mercurial users getting deprecation warnings on Python 2.6
when Python 2.7 was still over a year away).

The previous approach was leading to things like tools explicitly
silencing the warnings (so end users didn't see them), but then their
tests weren't seeing them either. The flip in moving to the status quo
(including adjusting test frameworks to turn the deprecation warnings
back on) improved the situation for folks doing things right (i.e.
using automated regression testing), while making it slightly worse
for folks that are trusting to luck (i.e. they may not even get the
deprecation warnings before things outright break on them).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Tue Mar  3 16:52:59 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 03 Mar 2015 17:52:59 +0200
Subject: [Python-ideas] IntFlags
Message-ID: <md4ldf$1a9$1@ger.gmane.org>

Enum and IntEnum classes allow constants to have nice str() and repr() 
representations.

 >>> socket.AF_INET
<AddressFamily.AF_INET: 2>
 >>> socket.socket()
<socket.socket fd=3, family=AddressFamily.AF_INET, 
type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>

But when integer constants are flags that should be ORed, IntEnum 
doesn't help, because the result of bitwise OR of two IntEnum instances 
is int, and this value can't be represented as IntEnum.

We need new type IntFlags. It is like IntEnum, but has differences:

1. The value of an instance should be not limited to the set of 
predefined constants. It can be a combination of predefined constants or 
even arbitrary integer.

2. The result of "|", "&" and "~" operators for IntFlags arguments 
should be an instance of the same IntFlags subclass.

3. It should have nice str() and repr().

 >>> print(stat.S_IROTH | stat.S_IWOTH)
stat.S_IROTH|stat.S_IWOTH
 >>> stat.S_IROTH | stat.S_IWOTH
<StatFlags.S_IROTH|S_IWOTH: 6>

Any thoughts?


From victor.stinner at gmail.com  Tue Mar  3 17:11:39 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 3 Mar 2015 17:11:39 +0100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <CAMpsgwaYj75bztwavud2kSjwsRpTVZi-bxTE4i7XzSYrdsOc-A@mail.gmail.com>

2015-03-03 16:52 GMT+01:00 Serhiy Storchaka <storchaka at gmail.com>:
> Enum and IntEnum classes allow constants to have nice str() and repr()
> representations.
>
>>>> socket.AF_INET
> <AddressFamily.AF_INET: 2>
>>>> socket.socket()
> <socket.socket fd=3, family=AddressFamily.AF_INET,
> type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>

By the way, since the PEP 446, sock.type may contain the flag SOCK_CLOEXEC ;-)

Extract of test_socket.py:

        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.assertEqual(sock.family, socket.AF_INET)
        if hasattr(socket, 'SOCK_CLOEXEC'):
            self.assertIn(sock.type,
                          (socket.SOCK_STREAM | socket.SOCK_CLOEXEC,
                           socket.SOCK_STREAM))

Victor

From ethan at stoneleaf.us  Tue Mar  3 17:40:11 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 03 Mar 2015 08:40:11 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <54F5E3EB.6080506@stoneleaf.us>

On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>
> Enum and IntEnum classes allow constants to have nice str() and repr() representations.
> 
> But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two
> IntEnum instances is int, and this value can't be represented as IntEnum.
> 
> We need new type IntFlags. It is like IntEnum, but has differences:
> 
> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
> predefined constants or even arbitrary integer.
> 
> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
> 
> 3. It should have nice str() and repr().

This was in my original Enum code, but stripped out as not being needed at the time.

If there is sufficient interest (and use-cases) I can add it back in.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/0fe9c732/attachment.sig>

From jsbueno at python.org.br  Tue Mar  3 17:44:15 2015
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 3 Mar 2015 13:44:15 -0300
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F5E3EB.6080506@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org>
	<54F5E3EB.6080506@stoneleaf.us>
Message-ID: <CAH0mxTSnC3HZfESAeoMqotP76Oo6E90EF5+3U9FRQbwsy9wStw@mail.gmail.com>

On 3 March 2015 at 13:40, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>>
>> Enum and IntEnum classes allow constants to have nice str() and repr() representations.
>>
>> But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two
>> IntEnum instances is int, and this value can't be represented as IntEnum.
>>
>> We need new type IntFlags. It is like IntEnum, but has differences:
>>
>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
>> predefined constants or even arbitrary integer.
>>
>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
>>
>> 3. It should have nice str() and repr().
>
> This was in my original Enum code, but stripped out as not being needed at the time.
>
> If there is sufficient interest (and use-cases) I can add it back in.

I think that would be nice.
The problem is that it would be too late now for changing the behavior
of "IntEnum" itself, right?

>
> --
> ~Ethan~
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From me at the-compiler.org  Tue Mar  3 17:55:35 2015
From: me at the-compiler.org (Florian Bruhin)
Date: Tue, 3 Mar 2015 17:55:35 +0100
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F5E3EB.6080506@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org>
 <54F5E3EB.6080506@stoneleaf.us>
Message-ID: <20150303165535.GJ11094@tonks>

* Ethan Furman <ethan at stoneleaf.us> [2015-03-03 08:40:11 -0800]:
> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
> >
> > Enum and IntEnum classes allow constants to have nice str() and repr() representations.
> > 
> > But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two
> > IntEnum instances is int, and this value can't be represented as IntEnum.
> > 
> > We need new type IntFlags. It is like IntEnum, but has differences:
> > 
> > 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
> > predefined constants or even arbitrary integer.
> > 
> > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
> > 
> > 3. It should have nice str() and repr().
> 
> This was in my original Enum code, but stripped out as not being needed at the time.
> 
> If there is sufficient interest (and use-cases) I can add it back in.

I'd gladly use it - for example when communicating with embedded
electronic devices.

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/

From storchaka at gmail.com  Tue Mar  3 18:27:22 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 03 Mar 2015 19:27:22 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F5E3EB.6080506@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54F5E3EB.6080506@stoneleaf.us>
Message-ID: <md4qtr$7aq$1@ger.gmane.org>

On 03.03.15 18:40, Ethan Furman wrote:
> This was in my original Enum code, but stripped out as not being needed at the time.
>
> If there is sufficient interest (and use-cases) I can add it back in.

This would be good. I need IntFlags even more than IntEnum.

I made an implementation based on early implementations of IntEnum, but 
then IntEnum was changed too much and became incompatible with my extension.

IntFlags should be a separate class, IntEnum is good with its restrictions.



From ethan at stoneleaf.us  Tue Mar  3 18:28:05 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 03 Mar 2015 09:28:05 -0800
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
Message-ID: <54F5EF25.4030002@stoneleaf.us>

On 03/02/2015 02:57 AM, Martin Teichmann wrote:
>     class Soliton(type):
>        def __new__(cls, name, bases, ns):
>            self = super().__new__(name, bases, ns)
>            return self()
> 
>     class A(metaclass=Soliton):
>         def f(self):
>             print(__class__)

This should be `self.__class__`, yes?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/92ccf739/attachment.sig>

From rosuav at gmail.com  Tue Mar  3 19:48:51 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 4 Mar 2015 05:48:51 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4qtr$7aq$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org> <54F5E3EB.6080506@stoneleaf.us>
 <md4qtr$7aq$1@ger.gmane.org>
Message-ID: <CAPTjJmpdg7keseSx6ZPngPGEXXCZRbYBbOXR4mPqRAePeNNxGg@mail.gmail.com>

On Wed, Mar 4, 2015 at 4:27 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> IntFlags should be a separate class, IntEnum is good with its restrictions.

Also, some operations that make sense for an IntEnum don't make sense
for IntFlags (or FlagEnum or whatever it gets called), such as
greater-than/less-than comparisons. Definitely a separate class IMO.

ChrisA

From cs at zip.com.au  Tue Mar  3 23:04:01 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 4 Mar 2015 09:04:01 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4qtr$7aq$1@ger.gmane.org>
References: <md4qtr$7aq$1@ger.gmane.org>
Message-ID: <20150303220401.GA13067@cskk.homeip.net>

On 03Mar2015 19:27, Serhiy Storchaka <storchaka at gmail.com> wrote:
>On 03.03.15 18:40, Ethan Furman wrote:
>>This was in my original Enum code, but stripped out as not being needed at the time.
>>
>>If there is sufficient interest (and use-cases) I can add it back in.
>
>This would be good. I need IntFlags even more than IntEnum.

I am also +1.

I also probably use IntFlags (euqivalents) more often than IntEnums. It would 
have been nice for my PEP-418 demo implementation:

  https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.py

I rolled my own _Clock_Flags there for this very reason.

>I made an implementation based on early implementations of IntEnum, 
>but then IntEnum was changed too much and became incompatible with my 
>extension.
>
>IntFlags should be a separate class, IntEnum is good with its restrictions.

I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that make 
for a TypeError or ValueError and one be expected to start with IntFlags?

Cheers,
Cameron Simpson <cs at zip.com.au>

Trust the computer... the computer is your friend.
        - Richard Dominelli <dominel at panix.com>

From rosuav at gmail.com  Tue Mar  3 23:16:47 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 4 Mar 2015 09:16:47 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <20150303220401.GA13067@cskk.homeip.net>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
Message-ID: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>

On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson <cs at zip.com.au> wrote:
> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that
> make for a TypeError or ValueError and one be expected to start with
> IntFlags?

That should be a sanity error, but since an IntEnum devolves to an
int, it just produces a nonsensical integer. It definitely shouldn't
become an IntFlags. There are two distinctly different use-cases here:

class Color(IntEnum):
    black = 0
    red = 1
    green = 2
    orange = 3
    blue = 4
    magenta = 5
    cyan = 6
    white = 7

class FileMode(IntFlags):
    owner_read = 0o400
    owner_write= 0o200
    owner_exec = 0o100
    group_read = 0o040
    group_write= 0o020
    group_exec = 0o010
    other_read = 0o004
    other_write= 0o002
    other_exec = 0o001

With colors, it makes no sense to combine them in any way (addition,
bitwise or, etc). You can't put blue and cyan together and expect to
get something usable. File modes are meant to be combined, and their
values are deliberately chosen to make this possible. So
IntEnum|IntEnum is most likely going to be combining values that were
assigned sequentially or arbitrarily, which isn't likely to be very
useful.

ChrisA

From ethan at stoneleaf.us  Tue Mar  3 23:29:34 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 03 Mar 2015 14:29:34 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
 <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
Message-ID: <54F635CE.6070508@stoneleaf.us>

On 03/03/2015 02:16 PM, Chris Angelico wrote:
> On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson <cs at zip.com.au> wrote:
>> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that
>> make for a TypeError or ValueError and one be expected to start with
>> IntFlags?
> 
> That should be a sanity error, but since an IntEnum devolves to an
> int, it just produces a nonsensical integer. It definitely shouldn't
> become an IntFlags.

Agreed.  IntEnums are just fancy ints, and will continue to behave as ints with the exception of how they are displayed.

IntFlags, should such a thing come into being, can be more proprietary with its behavior.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/e72f246e/attachment.sig>

From mistersheik at gmail.com  Wed Mar  4 02:45:38 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Tue, 3 Mar 2015 17:45:38 -0800 (PST)
Subject: [Python-ideas] unittest.failureExeption(msg,
 data=None): add optional data kwarg to failureException() of
 unittest
In-Reply-To: <54F571CE.3040901@thomas-guettler.de>
References: <54F571CE.3040901@thomas-guettler.de>
Message-ID: <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com>

Says it right in the documentation:

failureException 
<https://docs.python.org/2/library/unittest.html#unittest.TestCase.failureException>

This class attribute gives the exception raised by the test method. If a 
test framework needs to use a specialized exception, possibly to carry 
additional information, it must subclass this exception in order to ?play 
fair? with the framework. The initial value of this attribute is 
AssertionError 
<https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError>
.

On Tuesday, March 3, 2015 at 3:34:02 AM UTC-5, Thomas G?ttler wrote:
>
> Hi,
>
> what do you think about the idea to add an optional data kwarg to 
> unittest.failureExeption()?
>
> Up to now a the exception class "failureException" does only get a message 
> which is a string.
>
> I would like to log the failed Exceptions to a structured logging back end.
>
> Parsing the string message to extract the real value is possible, but 
> somehow ugly and error prone.
>
> My high level use case: Have a system (maybe ELK-Stack) to investigate if 
> a test failure was seen before.
>
> Doing this on strings could work, but I would like to keep the original 
> (json serializable) values.
>
> What do you think?
>
>    Thomas G?ttler
> _______________________________________________
> Python-ideas mailing list
> Python... at python.org <javascript:>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/c07e90b3/attachment-0001.html>

From cs at zip.com.au  Wed Mar  4 04:42:47 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 4 Mar 2015 14:42:47 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
References: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
Message-ID: <20150304034247.GA77820@cskk.homeip.net>

On 04Mar2015 09:16, Chris Angelico <rosuav at gmail.com> wrote:
>On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson <cs at zip.com.au> wrote:
>> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that
>> make for a TypeError or ValueError and one be expected to start with
>> IntFlags?
>
>That should be a sanity error, but since an IntEnum devolves to an
>int, it just produces a nonsensical integer. It definitely shouldn't
>become an IntFlags.

Yes, agreed. Brain fade. Sorry,
Cameron Simpson <cs at zip.com.au>

From mistersheik at gmail.com  Wed Mar  4 05:55:23 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Tue, 3 Mar 2015 20:55:23 -0800 (PST)
Subject: [Python-ideas] IntFlags
In-Reply-To: <20150303220401.GA13067@cskk.homeip.net>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
Message-ID: <77e8ae02-ed09-4771-9da7-c881f0321f6b@googlegroups.com>

Also +1.

I think IntFlags should also have a named constructor that constructs an 
IntFlags instance from a set:

reduce(lambda x, y: x | y, (c[key] for key in some_set))

Best,

Neil

On Tuesday, March 3, 2015 at 5:04:50 PM UTC-5, Cameron Simpson wrote:
>
> On 03Mar2015 19:27, Serhiy Storchaka <stor... at gmail.com <javascript:>> 
> wrote: 
> >On 03.03.15 18:40, Ethan Furman wrote: 
> >>This was in my original Enum code, but stripped out as not being needed 
> at the time. 
> >> 
> >>If there is sufficient interest (and use-cases) I can add it back in. 
> > 
> >This would be good. I need IntFlags even more than IntEnum. 
>
> I am also +1. 
>
> I also probably use IntFlags (euqivalents) more often than IntEnums. It 
> would 
> have been nice for my PEP-418 demo implementation: 
>
>   
> https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.py 
>
> I rolled my own _Clock_Flags there for this very reason. 
>
> >I made an implementation based on early implementations of IntEnum, 
> >but then IntEnum was changed too much and became incompatible with my 
> >extension. 
> > 
> >IntFlags should be a separate class, IntEnum is good with its 
> restrictions. 
>
> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that 
> make 
> for a TypeError or ValueError and one be expected to start with IntFlags? 
>
> Cheers, 
> Cameron Simpson <c... at zip.com.au <javascript:>> 
>
> Trust the computer... the computer is your friend. 
>         - Richard Dominelli <dom... at panix.com <javascript:>> 
> _______________________________________________ 
> Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150303/f144ce51/attachment.html>

From guettliml at thomas-guettler.de  Wed Mar  4 06:55:38 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Wed, 04 Mar 2015 06:55:38 +0100
Subject: [Python-ideas] unittest.failureExeption(msg,
 data=None): add optional data kwarg to failureException() of
 unittest
In-Reply-To: <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com>
References: <54F571CE.3040901@thomas-guettler.de>
 <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com>
Message-ID: <54F69E5A.1020907@thomas-guettler.de>

Am 04.03.2015 um 02:45 schrieb Neil Girdhar:
> Says it right in the documentation:
> 
> failureException 
> <https://docs.python.org/2/library/unittest.html#unittest.TestCase.failureException>
> 
> This class attribute gives the exception raised by the test method. If a 
> test framework needs to use a specialized exception, possibly to carry 
> additional information, it must subclass this exception in order to ?play 
> fair? with the framework. The initial value of this attribute is 
> AssertionError 
> <https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError>

You are right.

And if you want to assertEqual(a, b) to capture a and b,
you need to replace assertEqual to forward the values to
your own failureException.

I guess we will go this way: replace failureException and
assertEqual(), assertIn(), .... 

No change in Python's unittest lib is needed.

Thank you for your reply.

  Thomas G?ttler

-- 
http://www.thomas-guettler.de/

From mistersheik at gmail.com  Wed Mar  4 06:59:32 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Wed, 4 Mar 2015 00:59:32 -0500
Subject: [Python-ideas] unittest.failureExeption(msg,
 data=None): add optional data kwarg to failureException() of
 unittest
In-Reply-To: <54F69E5A.1020907@thomas-guettler.de>
References: <54F571CE.3040901@thomas-guettler.de>
 <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com>
 <54F69E5A.1020907@thomas-guettler.de>
Message-ID: <CAA68w_nBUYHyveS1dmXmSEKhNbQWjX=348VT669f7d32WxKs0w@mail.gmail.com>

You got it.

On Wed, Mar 4, 2015 at 12:55 AM, Thomas G?ttler <
guettliml at thomas-guettler.de> wrote:

> Am 04.03.2015 um 02:45 schrieb Neil Girdhar:
> > Says it right in the documentation:
> >
> > failureException
> > <
> https://docs.python.org/2/library/unittest.html#unittest.TestCase.failureException
> >
> >
> > This class attribute gives the exception raised by the test method. If a
> > test framework needs to use a specialized exception, possibly to carry
> > additional information, it must subclass this exception in order to ?play
> > fair? with the framework. The initial value of this attribute is
> > AssertionError
> > <
> https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError
> >
>
> You are right.
>
> And if you want to assertEqual(a, b) to capture a and b,
> you need to replace assertEqual to forward the values to
> your own failureException.
>
> I guess we will go this way: replace failureException and
> assertEqual(), assertIn(), ....
>
> No change in Python's unittest lib is needed.
>
> Thank you for your reply.
>
>   Thomas G?ttler
>
> --
> http://www.thomas-guettler.de/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150304/d9e8ae51/attachment.html>

From abarnert at yahoo.com  Wed Mar  4 16:17:49 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 4 Mar 2015 07:17:49 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>

One of the big questions that (IIRC) derailed this last time and got it dropped from the enum stdlib design was: what does ~ do? Does it give you the 2's complement negative integer? What does that display as in the str and repr? And, if you add in conversion from an IntFlags to/from a set of separate values, as has been suggested again in this thread, how does that work? All of this is trivial when you're dealing with C fixed-size unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) are set.

Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)

Sent from a random iPhone

On Mar 3, 2015, at 7:52, Serhiy Storchaka <storchaka at gmail.com> wrote:

> Enum and IntEnum classes allow constants to have nice str() and repr() representations.
> 
> >>> socket.AF_INET
> <AddressFamily.AF_INET: 2>
> >>> socket.socket()
> <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
> 
> But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two IntEnum instances is int, and this value can't be represented as IntEnum.
> 
> We need new type IntFlags. It is like IntEnum, but has differences:
> 
> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of predefined constants or even arbitrary integer.
> 
> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
> 
> 3. It should have nice str() and repr().
> 
> >>> print(stat.S_IROTH | stat.S_IWOTH)
> stat.S_IROTH|stat.S_IWOTH
> >>> stat.S_IROTH | stat.S_IWOTH
> <StatFlags.S_IROTH|S_IWOTH: 6>
> 
> Any thoughts?
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rosuav at gmail.com  Wed Mar  4 16:31:25 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 5 Mar 2015 02:31:25 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
Message-ID: <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>

On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
>

That could probably be handled by going through the flags in iteration
order. If the flag is present, emit it and move on. Something like
this:

from enum import IntEnum

class Flags(IntEnum):
    RDWR = 3
    RDONLY = 1
    WRONLY = 2
    CLOEXEC = 4

def flag_str(flg):
    names = []
    for flag in Flags:
        if (flg&flag) == flag:
            flg -= flag
            names.append(str(flag))
    return "|".join(names)

print(flag_str(Flags.RDWR|Flags.CLOEXEC))
print(flag_str(Flags.RDONLY|Flags.CLOEXEC))


As long as the combined versions come up ahead of the others, they'll
be used. Alternatively, if you prefer them _not_ to be used, just put
them after the individual forms, and then the str() will expand them
out.

ChrisA

From mistersheik at gmail.com  Wed Mar  4 16:44:33 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Wed, 4 Mar 2015 10:44:33 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
Message-ID: <CAA68w_mOxpSQo8=4rtP7+HTE-74MHZh5P5XrS_dT9K9wLPU1zQ@mail.gmail.com>

Why do you need these composite flags?

On Wed, Mar 4, 2015 at 10:31 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
> > Another issue that came up was that C flags often have "combined" names
> that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a
> repr (in C, it's just going to print 3); does it have to be smart enough to
> show RDWR? (Or, worse, RDWR | CLOEXEC.)
> >
>
> That could probably be handled by going through the flags in iteration
> order. If the flag is present, emit it and move on. Something like
> this:
>
> from enum import IntEnum
>
> class Flags(IntEnum):
>     RDWR = 3
>     RDONLY = 1
>     WRONLY = 2
>     CLOEXEC = 4
>
> def flag_str(flg):
>     names = []
>     for flag in Flags:
>         if (flg&flag) == flag:
>             flg -= flag
>             names.append(str(flag))
>     return "|".join(names)
>
> print(flag_str(Flags.RDWR|Flags.CLOEXEC))
> print(flag_str(Flags.RDONLY|Flags.CLOEXEC))
>
>
> As long as the combined versions come up ahead of the others, they'll
> be used. Alternatively, if you prefer them _not_ to be used, just put
> them after the individual forms, and then the str() will expand them
> out.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150304/fe22189d/attachment.html>

From mistersheik at gmail.com  Wed Mar  4 16:51:21 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Wed, 4 Mar 2015 07:51:21 -0800 (PST)
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com>

For ~, I suggest either having an extra bit on the object that remembers 
negation or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b 
with a.without_flags(b) or something like that.

On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote:
>
> Enum and IntEnum classes allow constants to have nice str() and repr() 
> representations. 
>
>  >>> socket.AF_INET 
> <AddressFamily.AF_INET: 2> 
>  >>> socket.socket() 
> <socket.socket fd=3, family=AddressFamily.AF_INET, 
> type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> 
>
> But when integer constants are flags that should be ORed, IntEnum 
> doesn't help, because the result of bitwise OR of two IntEnum instances 
> is int, and this value can't be represented as IntEnum. 
>
> We need new type IntFlags. It is like IntEnum, but has differences: 
>
> 1. The value of an instance should be not limited to the set of 
> predefined constants. It can be a combination of predefined constants or 
> even arbitrary integer. 
>
> 2. The result of "|", "&" and "~" operators for IntFlags arguments 
> should be an instance of the same IntFlags subclass. 
>
> 3. It should have nice str() and repr(). 
>
>  >>> print(stat.S_IROTH | stat.S_IWOTH) 
> stat.S_IROTH|stat.S_IWOTH 
>  >>> stat.S_IROTH | stat.S_IWOTH 
> <StatFlags.S_IROTH|S_IWOTH: 6> 
>
> Any thoughts? 
>
> _______________________________________________ 
> Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150304/66ee3ce4/attachment-0001.html>

From storchaka at gmail.com  Wed Mar  4 16:54:29 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 04 Mar 2015 17:54:29 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
Message-ID: <md79rl$ohf$1@ger.gmane.org>

On 04.03.15 17:17, Andrew Barnert wrote:
> One of the big questions that (IIRC) derailed this last time and got it dropped from the enum stdlib design was: what does ~ do? Does it give you the 2's complement negative integer? What does that display as in the str and repr? And, if you add in conversion from an IntFlags to/from a set of separate values, as has been suggested again in this thread, how does that work? All of this is trivial when you're dealing with C fixed-size unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) are set.

IntFlags is just fancy int. int(~flags) == int(~int(flags)). Python 
supports ~ for arbitrary integers. ~x == -x-1

There are no problems with conversions from a set to IntFlags, but the 
conversion from IntFlags to set is not always possible.

> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)

This problem is nor so hard. My implementation was smart enough.

 >>> print(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC)
OpenMode.RDWR|OpenMode.CLOEXEC
 >>> print(~(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC))
~(OpenMode.RDWR|OpenMode.CLOEXEC)

Of course the repr can be senseless if the value is senseless (such as 
RDONLY | ~WRONLY).



From storchaka at gmail.com  Wed Mar  4 16:58:37 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 04 Mar 2015 17:58:37 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
Message-ID: <md7a3d$t1m$1@ger.gmane.org>

On 04.03.15 17:31, Chris Angelico wrote:
> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
> That could probably be handled by going through the flags in iteration
> order. If the flag is present, emit it and move on. Something like
> this:

Yes, something like this, but with iterating flags in descended sorted 
order, and with special case for negative value.



From rosuav at gmail.com  Wed Mar  4 17:04:09 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 5 Mar 2015 03:04:09 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md7a3d$t1m$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org>
Message-ID: <CAPTjJmpssP-7YD4UMsTTQZwKdHE1-E5_r7zsKV4+Sxx61vLcBQ@mail.gmail.com>

On Thu, Mar 5, 2015 at 2:58 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 04.03.15 17:31, Chris Angelico wrote:
>>
>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
>> <abarnert at yahoo.com.dmarc.invalid> wrote:
>>>
>>> Another issue that came up was that C flags often have "combined" names
>>> that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a
>>> repr (in C, it's just going to print 3); does it have to be smart enough to
>>> show RDWR? (Or, worse, RDWR | CLOEXEC.)
>>
>> That could probably be handled by going through the flags in iteration
>> order. If the flag is present, emit it and move on. Something like
>> this:
>
>
> Yes, something like this, but with iterating flags in descended sorted
> order, and with special case for negative value.

Sure. Going automatically like that is a way of guaranteeing that the
combined flags will be used, which is probably what you want most of
the time anyway. And yes, I hadn't coped with negatives in that.

ChrisA

From python at 2sn.net  Wed Mar  4 19:37:38 2015
From: python at 2sn.net (Alexander Heger)
Date: Thu, 5 Mar 2015 05:37:38 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
 <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
Message-ID: <CAN3CYHyQiC0THBk3pK4h38CQg041h=tZTKis3G8FWp0LBNFa1A@mail.gmail.com>

> That should be a sanity error, but since an IntEnum devolves to an
> int, it just produces a nonsensical integer. It definitely shouldn't
> become an IntFlags. There are two distinctly different use-cases here:
>
> class Color(IntEnum):
>     black = 0
>     red = 1
>     green = 2
>     orange = 3
>     blue = 4
>     magenta = 5
>     cyan = 6
>     white = 7
>
> class FileMode(IntFlags):
>     owner_read = 0o400
>     owner_write= 0o200
>     owner_exec = 0o100
>     group_read = 0o040
>     group_write= 0o020
>     group_exec = 0o010
>     other_read = 0o004
>     other_write= 0o002
>     other_exec = 0o001
>
> With colors, it makes no sense to combine them in any way (addition,
> bitwise or, etc). You can't put blue and cyan together and expect to
> get something usable.

I don't see your point.  With colours it would be exactly the same:

red | blue == magenta

and for your example:

cyan | blue = cyan

As with the later discussion on file flags, cyan is just a combination
flag for green and blue.

-Alexander

From python at 2sn.net  Wed Mar  4 19:49:25 2015
From: python at 2sn.net (Alexander Heger)
Date: Thu, 5 Mar 2015 05:49:25 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md79rl$ohf$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <md79rl$ohf$1@ger.gmane.org>
Message-ID: <CAN3CYHyrfNgGyhxxebWwBdEdi7kkq9r8JCokzVq5coKKudJ1Ew@mail.gmail.com>

>> One of the big questions that (IIRC) derailed this last time and got it
>> dropped from the enum stdlib design was: what does ~ do? Does it give you
>> the 2's complement negative integer? What does that display as in the str
>> and repr? And, if you add in conversion from an IntFlags to/from a set of
>> separate values, as has been suggested again in this thread, how does that
>> work? All of this is trivial when you're dealing with C fixed-size unsigned
>> ints: ~READ means 15 of the 16 bits (all except the READ bit) are set.
>
>
> IntFlags is just fancy int. int(~flags) == int(~int(flags)). Python supports
> ~ for arbitrary integers. ~x == -x-1
>
> There are no problems with conversions from a set to IntFlags, but the
> conversion from IntFlags to set is not always possible.
>
>> Another issue that came up was that C flags often have "combined" names
>> that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a
>> repr (in C, it's just going to print 3); does it have to be smart enough to
>> show RDWR? (Or, worse, RDWR | CLOEXEC.)
>
>
> This problem is nor so hard. My implementation was smart enough.
>
>>>> print(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC)
> OpenMode.RDWR|OpenMode.CLOEXEC
>>>> print(~(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC))
> ~(OpenMode.RDWR|OpenMode.CLOEXEC)
>
> Of course the repr can be senseless if the value is senseless (such as
> RDONLY | ~WRONLY).

Wouldn't it be possible to for the class to determine what bits are
all used from the constants defined, and then for the ~ operator to
just invert those?  It may require some sanity check for users only
defining "combination constants" in the class such that the result
could not be represented.

class Stupid(IntFlags):
    CAT = 3
    DOG = 6

in which case ~CAT would not make sense, but neither could CAT | DOG
be represented.  I suppose in such cases an error should be raised by
the metaclass on class definition.

-Alexander

From random832 at fastmail.us  Wed Mar  4 20:45:07 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 04 Mar 2015 14:45:07 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com>

On Tue, Mar 3, 2015, at 10:52, Serhiy Storchaka wrote:
> 3. It should have nice str() and repr().
> 
>  >>> print(stat.S_IROTH | stat.S_IWOTH)
> stat.S_IROTH|stat.S_IWOTH
>  >>> stat.S_IROTH | stat.S_IWOTH
> <StatFlags.S_IROTH|S_IWOTH: 6>
> 
> Any thoughts?

Any reason StatFlags shouldn't have a representation like "rwxr-x---"
for 0750?

From skip.montanaro at gmail.com  Wed Mar  4 20:47:17 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 4 Mar 2015 13:47:17 -0600
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com>
Message-ID: <CANc-5Uy7Ow37tgvsvAwJ-7xPPsGPyXAZ4MhauhXe52C14D+5_g@mail.gmail.com>

On Wed, Mar 4, 2015 at 1:45 PM, <random832 at fastmail.us> wrote:

> Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750?

I've never considered this before, but is this sort of thing portable
to/meaningful on Windows?

Skip

From random832 at fastmail.us  Wed Mar  4 20:48:26 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 04 Mar 2015 14:48:26 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
 <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
Message-ID: <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com>

On Tue, Mar 3, 2015, at 17:16, Chris Angelico wrote:
> class Color(IntEnum):
>     black = 0
>     red = 1
>     green = 2
>     orange = 3
>     blue = 4
>     magenta = 5
>     cyan = 6
>     white = 7
> 
> With colors, it makes no sense to combine them in any way (addition,
> bitwise or, etc). You can't put blue and cyan together and expect to
> get something usable.

Except for the part where green|blue==cyan... and if only you'd said
yellow instead of orange it'd hold across the board. I assume you took
these values from ANSI colors.

From random832 at fastmail.us  Wed Mar  4 20:52:50 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 04 Mar 2015 14:52:50 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
Message-ID: <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>

On Wed, Mar 4, 2015, at 10:17, Andrew Barnert wrote:
> One of the big questions that (IIRC) derailed this last time and got it
> dropped from the enum stdlib design was: what does ~ do? Does it give you
> the 2's complement negative integer? What does that display as in the str
> and repr? And, if you add in conversion from an IntFlags to/from a set of
> separate values, as has been suggested again in this thread, how does
> that work? All of this is trivial when you're dealing with C fixed-size
> unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit)
> are set.

str should be ~(READ) obviously. And more generally
~(all|bits|that|are|not|set).

> Another issue that came up was that C flags often have "combined" names
> that are ambiguous: RDWR = RDONLY | WRONLY), 

Nope. You've actually got a different, much worse, issue: traditionally,
RDONLY is 0 (and should be printed if and only if 1 and 2 are not set),
WRONLY is 1, RDWR is 2, and 3 is traditionally invalid and may have a
platform-dependent meaning.

From random832 at fastmail.us  Wed Mar  4 20:54:43 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 04 Mar 2015 14:54:43 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAN3CYHyQiC0THBk3pK4h38CQg041h=tZTKis3G8FWp0LBNFa1A@mail.gmail.com>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
 <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
 <CAN3CYHyQiC0THBk3pK4h38CQg041h=tZTKis3G8FWp0LBNFa1A@mail.gmail.com>
Message-ID: <1425498883.1920337.235571721.245D6167@webmail.messagingengine.com>

On Wed, Mar 4, 2015, at 13:37, Alexander Heger wrote:
> I don't see your point.  With colours it would be exactly the same:
> 
> red | blue == magenta
> 
> and for your example:
> 
> cyan | blue = cyan
> 
> As with the later discussion on file flags, cyan is just a combination
> flag for green and blue.

I think it was a badly chosen example - someone easily _could_ have a
color enum that doesn't have these properties. Definitely there are
enums that it doesn't make sense to combine together with bitwise ops.

From p.f.moore at gmail.com  Wed Mar  4 21:17:27 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 4 Mar 2015 20:17:27 +0000
Subject: [Python-ideas] IntFlags
In-Reply-To: <CANc-5Uy7Ow37tgvsvAwJ-7xPPsGPyXAZ4MhauhXe52C14D+5_g@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com>
 <CANc-5Uy7Ow37tgvsvAwJ-7xPPsGPyXAZ4MhauhXe52C14D+5_g@mail.gmail.com>
Message-ID: <CACac1F9WkmZ84YHMzqZ2nV8YwQsTxiN1y6y6hbqoBV2JmJwY7w@mail.gmail.com>

On 4 March 2015 at 19:47, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> On Wed, Mar 4, 2015 at 1:45 PM, <random832 at fastmail.us> wrote:
>
>> Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750?
>
> I've never considered this before, but is this sort of thing portable
> to/meaningful on Windows?

Not really. Windows' actual ACL system is far too complex to fit into
9 bits. But the readable and writable bits are use to reflect the
readonly flag (which is a separate value from the ACLs) of a file (so
S_IREAD is always true, and S_IWRITE is true unless the file is
readonly). So you only ever get "'-rw-rw-rw-'" or "'-r--r--r--'"

Paul

From rosuav at gmail.com  Wed Mar  4 21:58:38 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 5 Mar 2015 07:58:38 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com>
References: <md4qtr$7aq$1@ger.gmane.org>
 <20150303220401.GA13067@cskk.homeip.net>
 <CAPTjJmqWSavhdNfyTvcY_JGRORPHCpu2KZdPnoywjJaO4Ttrsg@mail.gmail.com>
 <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com>
Message-ID: <CAPTjJmq9+=mdr99zUTkZ49mXL-56LcYvkbzNUFW6QNM73t86fA@mail.gmail.com>

On Thu, Mar 5, 2015 at 6:48 AM,  <random832 at fastmail.us> wrote:
> On Tue, Mar 3, 2015, at 17:16, Chris Angelico wrote:
>> class Color(IntEnum):
>>     black = 0
>>     red = 1
>>     green = 2
>>     orange = 3
>>     blue = 4
>>     magenta = 5
>>     cyan = 6
>>     white = 7
>>
>> With colors, it makes no sense to combine them in any way (addition,
>> bitwise or, etc). You can't put blue and cyan together and expect to
>> get something usable.
>
> Except for the part where green|blue==cyan... and if only you'd said
> yellow instead of orange it'd hold across the board. I assume you took
> these values from ANSI colors.

Those are indeed the ANSI colors, but even though you might think that
you can combine them, they don't really combine usefully in all cases.
But sure. If you're bothered by the fact that blue+green == cyan, use
a different example. Turn 'em into animals, cars, CPU models, whatever
you like; the point of an IntEnum is usually just that the values are
unique, _not_ that they can be combined in any meaningful way.

ChrisA

From random832 at fastmail.us  Wed Mar  4 23:31:55 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 04 Mar 2015 17:31:55 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CANc-5Uy7Ow37tgvsvAwJ-7xPPsGPyXAZ4MhauhXe52C14D+5_g@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com>
 <CANc-5Uy7Ow37tgvsvAwJ-7xPPsGPyXAZ4MhauhXe52C14D+5_g@mail.gmail.com>
Message-ID: <1425508315.1963424.235639341.7EF9350A@webmail.messagingengine.com>

On Wed, Mar 4, 2015, at 14:47, Skip Montanaro wrote:
> On Wed, Mar 4, 2015 at 1:45 PM, <random832 at fastmail.us> wrote:
> 
> > Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750?
> 
> I've never considered this before, but is this sort of thing portable
> to/meaningful on Windows?

No, but neither are 90% of the flags themselves, especially as actually
implemented by the C runtime library (they're basically all hardcoded,
except the write flags are turned off if a file is readonly, the exec
flags are set based on a hardcoded list of filename extensions, and a
handful of the file type bit states are implemented)

If someone wants to do something meaningful on windows, they'll need a
richer API than the one currently implemented in the os module.

From abarnert at yahoo.com  Wed Mar  4 23:30:07 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 4 Mar 2015 14:30:07 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_mOxpSQo8=4rtP7+HTE-74MHZh5P5XrS_dT9K9wLPU1zQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <CAA68w_mOxpSQo8=4rtP7+HTE-74MHZh5P5XrS_dT9K9wLPU1zQ@mail.gmail.com>
Message-ID: <B2D9C6D9-F48C-4E14-98D7-E1812FF4E9F3@yahoo.com>

On Mar 4, 2015, at 7:44, Neil Girdhar <mistersheik at gmail.com> wrote:

> Why do you need these composite flags?

Because the whole point of this proposal is to deal with C types (otherwise, who cares about the int value?). And most such C types define combined values--the motivating example, stat, has S_IRWXU, etc. If the Python code is less readable than the equivalent C code...

> On Wed, Mar 4, 2015 at 10:31 AM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
>> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> > Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
>> >
>> 
>> That could probably be handled by going through the flags in iteration
>> order. If the flag is present, emit it and move on. Something like
>> this:
>> 
>> from enum import IntEnum
>> 
>> class Flags(IntEnum):
>>     RDWR = 3
>>     RDONLY = 1
>>     WRONLY = 2
>>     CLOEXEC = 4
>> 
>> def flag_str(flg):
>>     names = []
>>     for flag in Flags:
>>         if (flg&flag) == flag:
>>             flg -= flag
>>             names.append(str(flag))
>>     return "|".join(names)
>> 
>> print(flag_str(Flags.RDWR|Flags.CLOEXEC))
>> print(flag_str(Flags.RDONLY|Flags.CLOEXEC))
>> 
>> 
>> As long as the combined versions come up ahead of the others, they'll
>> be used. Alternatively, if you prefer them _not_ to be used, just put
>> them after the individual forms, and then the str() will expand them
>> out.
>> 
>> ChrisA
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> --
>> 
>> ---
>> You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150304/d6c0e0ab/attachment-0001.html>

From abarnert at yahoo.com  Wed Mar  4 23:59:48 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 4 Mar 2015 14:59:48 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com>
Message-ID: <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com>

On Mar 4, 2015, at 7:51, Neil Girdhar <mistersheik at gmail.com> wrote:

> For ~, I suggest either having an extra bit on the object that remembers negation

Then what exactly happens when you & or | two values, one of which has the negation bit set? For example, using stat values, what is S_IRUSR & ~S_IXUSR? Or, worse, with |. You can end up with some bits set, some negatively set, and some set neither way. How are you going to represent that?

In C, because the values are stored as unsigned fixed-size (say, 16-bit) ints and the ~ is just the usual 1's complement operator on unsigned int types, ~S_IXUSR is all the bits except 0o400 (that is 0o1377).

There are multiple ways to solve this: 

 * Use signed 1's complement (as I suggested in my original message, which Serhily ignored and suggested the same thing), but then the value of ~S_IXUSR is -0o401, which isn't likely to match == to a value you got from C (or to be obvious to someone using/debugging the code).

 * Scan the values and assume the highest bit seen is the max bit (as two other people suggested) and manually unsigned-1's-comp, but this is not only more complicated, it only provides the same values as C if you've used all 8/16/32/64 bits or you explicitly mask off & ALL_BITS at the end.

 * Require specifying the max bit somewhere in the definition, maybe defaulting to 32, or maybe using C++ enum rules (round the highest value up to 8, 16, 32, or 64 bits).

 * Store two separate ints, one for negated and one for non-negated.

 * Don't allow ~ at all (as you suggest at the end of the paragraph), using methods instead (although you only gave mutating methods, which is going to make a lot of C 1-liners turn into verbose 3-liners in Python).

And if you're wondering why I keep harping on having the same value as C: if you don't care about that, you don't need to use an int as a bitset in the first place; the only reason this proposal is useful in the first place is that functions like stat return ints that have well-known meanings from C, and everyone knows how to manipulate them in C, and we want to be able to do the same thing in Python but with readable reprs and all the other benefits of a real enum type.

> or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b with a.without_flags(b) or something like that.
> 
> On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote:
>> 
>> Enum and IntEnum classes allow constants to have nice str() and repr() 
>> representations. 
>> 
>>  >>> socket.AF_INET 
>> <AddressFamily.AF_INET: 2> 
>>  >>> socket.socket() 
>> <socket.socket fd=3, family=AddressFamily.AF_INET, 
>> type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> 
>> 
>> But when integer constants are flags that should be ORed, IntEnum  
>> doesn't help, because the result of bitwise OR of two IntEnum instances 
>> is int, and this value can't be represented as IntEnum. 
>> 
>> We need new type IntFlags. It is like IntEnum, but has differences: 
>> 
>> 1. The value of an instance should be not limited to the set of 
>> predefined constants. It can be a combination of predefined constants or 
>> even arbitrary integer. 
>> 
>> 2. The result of "|", "&" and "~" operators for IntFlags arguments 
>> should be an instance of the same IntFlags subclass. 
>> 
>> 3. It should have nice str() and repr(). 
>> 
>>  >>> print(stat.S_IROTH | stat.S_IWOTH) 
>> stat.S_IROTH|stat.S_IWOTH 
>>  >>> stat.S_IROTH | stat.S_IWOTH 
>> <StatFlags.S_IROTH|S_IWOTH: 6> 
>> 
>> Any thoughts? 
>> 
>> _______________________________________________ 
>> Python-ideas mailing list 
>> Python... at python.org 
>> https://mail.python.org/mailman/listinfo/python-ideas 
>> Code of Conduct: http://python.org/psf/codeofconduct/ 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150304/9b5f4d67/attachment.html>

From abarnert at yahoo.com  Thu Mar  5 00:19:00 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 4 Mar 2015 15:19:00 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
Message-ID: <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>

On Mar 4, 2015, at 11:52, random832 at fastmail.us wrote:

> On Wed, Mar 4, 2015, at 10:17, Andrew Barnert wrote:
>> One of the big questions that (IIRC) derailed this last time and got it
>> dropped from the enum stdlib design was: what does ~ do? Does it give you
>> the 2's complement negative integer? What does that display as in the str
>> and repr? And, if you add in conversion from an IntFlags to/from a set of
>> separate values, as has been suggested again in this thread, how does
>> that work? All of this is trivial when you're dealing with C fixed-size
>> unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit)
>> are set.
> 
> str should be ~(READ) obviously. And more generally
> ~(all|bits|that|are|not|set).

Think about how that extends to the result of |. Of course the answer depends on how you store ~ in the first place, but for most choices, str is not obvious. For example, using fixed-size unsigned with automatic highest-bit detection, for an enum with READ, WRITE, EXEC, STICKY, ~(READ) is the same value as (WRITE|EXEC|STICKY), so how does str know which to print? The one with the fewest flags? Some other rule? (The signed 1's comp choice actually has a reasonable answer here, it just means that you get silly results for silly values, which is fine...)

>> Another issue that came up was that C flags often have "combined" names
>> that are ambiguous: RDWR = RDONLY | WRONLY),
> 
> Nope. You've actually got a different, much worse, issue: traditionally,
> RDONLY is 0 (and should be printed if and only if 1 and 2 are not set),
> WRONLY is 1, RDWR is 2, and 3 is traditionally invalid and may have a
> platform-dependent meaning.

Yeah, open flags are especially screwy, where flags & 3 has a special non-bitmapped meaning but the rest of the bits are flags. There are other cases where multiple ints, only some of which are bitmaps, are packed as separate bitfields into the same int (e.g., TCP/IP headers), but I don't know of any others that pretend to be a single bitmap even though they aren't, so that's really a unique problem, which can be ignored. Just think about stat results or mmap prot flags or anything else where read and write are separate bits. (PS, IIRC, Linux treats 3 as "open for fstat only", and you have that access on most files even if you can't do anything else to them, which was an accident left in place because lilo used it, which made lilo a pain to port to FreeBSD as part of a Linux repair kit...)

> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Thu Mar  5 00:42:03 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 4 Mar 2015 15:42:03 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <md7a3d$t1m$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org>
Message-ID: <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>

On Mar 4, 2015, at 7:58, Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 04.03.15 17:31, Chris Angelico wrote:
>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
>> <abarnert at yahoo.com.dmarc.invalid> wrote:
>>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
>> That could probably be handled by going through the flags in iteration
>> order. If the flag is present, emit it and move on. Something like
>> this:
> 
> Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value.

I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious.

For the record, I think your version is better, because usually the C definitions define the combined values after the individual ones, and it would be nice to be able to mirror the C definitions (or, even better, auto generate the Python from the C header*) and get the desired results.

I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course).

But regardless, the point is that these questions don't have a single obvious answer; you have to think about them, decide what makes sense, and explain the tradeoffs and why one should win (and implement it). That's why this "simple" proposal didn't make it into 3.4's enum: because it's not actually simple, and everyone who insisted that it was disagreed, and someone (Guido or Eli?) finally told everyone to shut up, flags weren't going into 3.4, and put your competing implementations on PyPI and see which one people use. Do we really want to rehash all those arguments from scratch?

---

* The parenthetical brings up another issue: if you look in your platform's sys/stat.h (or whichever header actually defines these things), S_IRWXG is probably not defined as 0o700, but as S_IRGRP | S_IWGRP | S_IXGRP. Can we do that in the enum definition? If not, it may be less readable than the C. (In fact, on many platforms, S_IRGRP is itself defined as something like _S_IREAD << _S_GRP, and S_IRWXG may be defined as (_S_IREAD | _S_IWRITE | _S_IEXEC) << _S_GRP. I think Linux took this even further and defined it as S_IRWXU >> (_S_USR - _S_GRP) or something silly. But at that point, not being able to clone the C no longer looks like a loss of readability...)


From rosuav at gmail.com  Thu Mar  5 00:56:57 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 5 Mar 2015 10:56:57 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org>
 <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
Message-ID: <CAPTjJmq2X_+KwEVGFtd3hqmDPOeYkRGrPW5+7nF2FCee5JZswQ@mail.gmail.com>

On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Mar 4, 2015, at 7:58, Serhiy Storchaka <storchaka at gmail.com> wrote:
>
>> On 04.03.15 17:31, Chris Angelico wrote:
>>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
>>> <abarnert at yahoo.com.dmarc.invalid> wrote:
>>>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
>>> That could probably be handled by going through the flags in iteration
>>> order. If the flag is present, emit it and move on. Something like
>>> this:
>>
>> Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value.
>
> I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious.

Actually, we chose the same thing, only in slightly different ways.
Serhiy suggested (in effect) sorting the flags by value and stepping
through from highest to lowest, which enforces that the combined flags
will be the ones picked. I suggested putting the responsibility onto
the class author - if you want the combined ones to be used, place
them first - which is like how aliasing works (the first one with a
given value is used in str/repr, any others are aliases). That's a
relatively minor point, and it depends on whether there'd ever be a
time when you want to provide a combined flag that _isn't_ used in
str/repr; if there is, you need my plan, but if not, go with the
simpler route.

ChrisA

From rob.cliffe at btinternet.com  Thu Mar  5 01:18:27 2015
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 05 Mar 2015 00:18:27 +0000
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAN3CYHyrfNgGyhxxebWwBdEdi7kkq9r8JCokzVq5coKKudJ1Ew@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <md79rl$ohf$1@ger.gmane.org>
 <CAN3CYHyrfNgGyhxxebWwBdEdi7kkq9r8JCokzVq5coKKudJ1Ew@mail.gmail.com>
Message-ID: <54F7A0D3.1090001@btinternet.com>

> It may require some sanity check for users only
> defining "combination constants" in the class such that the result
> could not be represented.
>
> class Stupid(IntFlags):
>      CAT = 3
>      DOG = 6
>
> in which case ~CAT would not make sense, but neither could CAT | DOG
> be represented.  I suppose in such cases an error should be raised by
> the metaclass on class definition.
>
> -Alexander
Personally, I find that CAT & DOG is almost invariably an error. :-)

From abarnert at yahoo.com  Thu Mar  5 10:25:53 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Mar 2015 01:25:53 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAPTjJmq2X_+KwEVGFtd3hqmDPOeYkRGrPW5+7nF2FCee5JZswQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org> <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
 <CAPTjJmq2X_+KwEVGFtd3hqmDPOeYkRGrPW5+7nF2FCee5JZswQ@mail.gmail.com>
Message-ID: <F8943E19-EA40-4332-9FC4-BC138973B184@yahoo.com>

On Mar 4, 2015, at 15:56, Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> On Mar 4, 2015, at 7:58, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 
>>> On 04.03.15 17:31, Chris Angelico wrote:
>>>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
>>>> <abarnert at yahoo.com.dmarc.invalid> wrote:
>>>>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
>>>> That could probably be handled by going through the flags in iteration
>>>> order. If the flag is present, emit it and move on. Something like
>>>> this:
>>> 
>>> Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value.
>> 
>> I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious.
> 
> Actually, we chose the same thing, only in slightly different ways.
> Serhiy suggested (in effect) sorting the flags by value and stepping
> through from highest to lowest, which enforces that the combined flags
> will be the ones picked. I suggested putting the responsibility onto
> the class author - if you want the combined ones to be used, place
> them first - which is like how aliasing works (the first one with a
> given value is used in str/repr, any others are aliases). That's a
> relatively minor point, and it depends on whether there'd ever be a
> time when you want to provide a combined flag that _isn't_ used in
> str/repr; if there is, you need my plan, but if not, go with the
> simpler route.

Except with yours, any time you copy the definition (whether by hand, or with an automated tool) from C, where the combined ones almost always come last, they won't be used; with his, they will. So, in the most common use case, you'll get the opposite result.

(Of course if you always pick the _last_ instead of the first or the highest, which gives you the choice for the rare case, but the nice answer for the ubiquitous case. That breaks your analogy with aliases, but it does work the same way as normal class attributes, where the last value wins...)

From mistersheik at gmail.com  Thu Mar  5 10:58:53 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 04:58:53 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <F8943E19-EA40-4332-9FC4-BC138973B184@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org> <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
 <CAPTjJmq2X_+KwEVGFtd3hqmDPOeYkRGrPW5+7nF2FCee5JZswQ@mail.gmail.com>
 <F8943E19-EA40-4332-9FC4-BC138973B184@yahoo.com>
Message-ID: <CAA68w_k+2kPpXbJtuxUUG+gGy+B0ZXiv8tFmBr=RUmD73J5Btw@mail.gmail.com>

Can I propose that instead of IntFlags, we have IntFields?

class Permissions(IntFields):
    # Some Boolean flags (the bit index)
    owner_read = 0
    owner_write = 1
    owner_exec = 2
    group_read = 3
    group_write = 4
    group_exec = 5
    user_read = 6
    user_write = 7
    user_exec = 8

    # Some named fields (can overlap)
    owner_flags = range(3)
    group_flags = range(3, 6)
    user_flags = range(6, 9)

    # Some field values (can overlap with flags and each other)
    regular_file = range(9), 0o755
    character_file = range(9), 0o664
    directory = range(9), 0o600


And then you can do the following operations:

p = Permissions(Permissions.character_file)

p.set(Permissions.regular_file)

p.owner_write = True

print(p.user_read)

p.owner_flags = 6

Another example:

class IEEE754(IntFields):
    fraction = range(23)
    exponent = range(23, 23+8)
    sign = 31

    denormalized = 'exponent', 0
    infinity = 'exponent', 255

In case it's not clear, there are two kinds of members: named fields and
field values.  Named fields are specified with a range instance or integer
represent the bit range or index.  Field values are specified with a pair,
the first member of which is either a range, an integer, or a string (which
would refer to a declared range); and a second member, which is the value.

IntFields.set accepts a field value name, and sets the bits according to
the value.
IntFields.__setattr__ accepts a named field name; it sets the corresponding
bits according to the value.
IntFields.__getattr__ accepts a named field name or field value name

There is no ~ or & operator.  Something should probably be done for __or__.

Best,

Neil

On Thu, Mar 5, 2015 at 4:25 AM, 'Andrew Barnert' via python-ideas <
python-ideas at googlegroups.com> wrote:

> On Mar 4, 2015, at 15:56, Chris Angelico <rosuav at gmail.com> wrote:
>
> > On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert
> > <abarnert at yahoo.com.dmarc.invalid> wrote:
> >> On Mar 4, 2015, at 7:58, Serhiy Storchaka <storchaka at gmail.com> wrote:
> >>
> >>> On 04.03.15 17:31, Chris Angelico wrote:
> >>>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert
> >>>> <abarnert at yahoo.com.dmarc.invalid> wrote:
> >>>>> Another issue that came up was that C flags often have "combined"
> names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you
> want a repr (in C, it's just going to print 3); does it have to be smart
> enough to show RDWR? (Or, worse, RDWR | CLOEXEC.)
> >>>> That could probably be handled by going through the flags in iteration
> >>>> order. If the flag is present, emit it and move on. Something like
> >>>> this:
> >>>
> >>> Yes, something like this, but with iterating flags in descended sorted
> order, and with special case for negative value.
> >>
> >> I think the very fact that the two of you immediately knew which order
> was obvious, but you chose the opposite one as the obvious one, proves that
> it's not obvious.
> >
> > Actually, we chose the same thing, only in slightly different ways.
> > Serhiy suggested (in effect) sorting the flags by value and stepping
> > through from highest to lowest, which enforces that the combined flags
> > will be the ones picked. I suggested putting the responsibility onto
> > the class author - if you want the combined ones to be used, place
> > them first - which is like how aliasing works (the first one with a
> > given value is used in str/repr, any others are aliases). That's a
> > relatively minor point, and it depends on whether there'd ever be a
> > time when you want to provide a combined flag that _isn't_ used in
> > str/repr; if there is, you need my plan, but if not, go with the
> > simpler route.
>
> Except with yours, any time you copy the definition (whether by hand, or
> with an automated tool) from C, where the combined ones almost always come
> last, they won't be used; with his, they will. So, in the most common use
> case, you'll get the opposite result.
>
> (Of course if you always pick the _last_ instead of the first or the
> highest, which gives you the choice for the rare case, but the nice answer
> for the ubiquitous case. That breaks your analogy with aliases, but it does
> work the same way as normal class attributes, where the last value wins...)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/99bf6cda/attachment-0001.html>

From mistersheik at gmail.com  Thu Mar  5 11:04:14 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 05:04:14 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com>
 <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com>
Message-ID: <CAA68w_kV6z8M5A48SPn1=0woXBVnhWxof4UmHfZf8T7MsWx3nA@mail.gmail.com>

This is exactly why we should not have & and ~.  It creates the possibility
to write meaningless code.  Have methods that clear fields and methods that
set them.  I proposed an interface to IntFields in another message.



Best,

Neil

On Wed, Mar 4, 2015 at 5:59 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 4, 2015, at 7:51, Neil Girdhar <mistersheik at gmail.com> wrote:
>
> For ~, I suggest either having an extra bit on the object that remembers
> negation
>
>
> Then what exactly happens when you & or | two values, one of which has the
> negation bit set? For example, using stat values, what is S_IRUSR &
> ~S_IXUSR? Or, worse, with |. You can end up with some bits set, some
> negatively set, and some set neither way. How are you going to represent
> that?
>
> In C, because the values are stored as unsigned fixed-size (say, 16-bit)
> ints and the ~ is just the usual 1's complement operator on unsigned int
> types, ~S_IXUSR is all the bits except 0o400 (that is 0o1377).
>
> There are multiple ways to solve this:
>
>  * Use signed 1's complement (as I suggested in my original message, which
> Serhily ignored and suggested the same thing), but then the value of
> ~S_IXUSR is -0o401, which isn't likely to match == to a value you got from
> C (or to be obvious to someone using/debugging the code).
>
>  * Scan the values and assume the highest bit seen is the max bit (as two
> other people suggested) and manually unsigned-1's-comp, but this is not
> only more complicated, it only provides the same values as C if you've used
> all 8/16/32/64 bits or you explicitly mask off & ALL_BITS at the end.
>
>  * Require specifying the max bit somewhere in the definition, maybe
> defaulting to 32, or maybe using C++ enum rules (round the highest value up
> to 8, 16, 32, or 64 bits).
>
>  * Store two separate ints, one for negated and one for non-negated.
>
>  * Don't allow ~ at all (as you suggest at the end of the paragraph),
> using methods instead (although you only gave mutating methods, which is
> going to make a lot of C 1-liners turn into verbose 3-liners in Python).
>

(I did give a pair of methods one of which was not mutating)


> And if you're wondering why I keep harping on having the same value as C:
> if you don't care about that, you don't need to use an int as a bitset in
> the first place; the only reason this proposal is useful in the first place
> is that functions like stat return ints that have well-known meanings from
> C, and everyone knows how to manipulate them in C, and we want to be able
> to do the same thing in Python but with readable reprs and all the other
> benefits of a real enum type.
>
> or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b with
> a.without_flags(b) or something like that.
>
> On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote:
>>
>> Enum and IntEnum classes allow constants to have nice str() and repr()
>> representations.
>>
>>  >>> socket.AF_INET
>> <AddressFamily.AF_INET: 2>
>>  >>> socket.socket()
>> <socket.socket fd=3, family=AddressFamily.AF_INET,
>> type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
>>
>> But when integer constants are flags that should be ORed, IntEnum
>> doesn't help, because the result of bitwise OR of two IntEnum instances
>> is int, and this value can't be represented as IntEnum.
>>
>> We need new type IntFlags. It is like IntEnum, but has differences:
>>
>> 1. The value of an instance should be not limited to the set of
>> predefined constants. It can be a combination of predefined constants or
>> even arbitrary integer.
>>
>> 2. The result of "|", "&" and "~" operators for IntFlags arguments
>> should be an instance of the same IntFlags subclass.
>>
>> 3. It should have nice str() and repr().
>>
>>  >>> print(stat.S_IROTH | stat.S_IWOTH)
>> stat.S_IROTH|stat.S_IWOTH
>>  >>> stat.S_IROTH | stat.S_IWOTH
>> <StatFlags.S_IROTH|S_IWOTH: 6>
>>
>> Any thoughts?
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python... at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/8068165f/attachment.html>

From mistersheik at gmail.com  Thu Mar  5 11:31:05 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 02:31:05 -0800 (PST)
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_k+2kPpXbJtuxUUG+gGy+B0ZXiv8tFmBr=RUmD73J5Btw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org> <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
 <CAPTjJmq2X_+KwEVGFtd3hqmDPOeYkRGrPW5+7nF2FCee5JZswQ@mail.gmail.com>
 <F8943E19-EA40-4332-9FC4-BC138973B184@yahoo.com>
 <CAA68w_k+2kPpXbJtuxUUG+gGy+B0ZXiv8tFmBr=RUmD73J5Btw@mail.gmail.com>
Message-ID: <6761e23a-d640-4b86-af74-0acb82d96c25@googlegroups.com>

Actually, it would be cool to support nesting too, e.g.,

class SubPermissions(IntFields):
    read = 0
    write = 1
    exec_ = 2

class Permissions(IntFields):
    owner_flags = range(3), SubPermissions
    group_flags = range(3, 6), SubPermissions
    user_flags = range(6, 9), SubPermissions

    regular_file = range(9), 0o755
    character_file = range(9), 0o664
    directory = range(9), 0o600
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/c006ac45/attachment.html>

From storchaka at gmail.com  Thu Mar  5 16:32:23 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 05 Mar 2015 17:32:23 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <md9su7$41k$1@ger.gmane.org>

Opened an issue: http://bugs.python.org/issue23591.



From random832 at fastmail.us  Thu Mar  5 16:58:44 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 05 Mar 2015 10:58:44 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
 <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>
Message-ID: <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com>

On Wed, Mar 4, 2015, at 18:19, Andrew Barnert wrote:
> Think about how that extends to the result of |.

The value is either negative (is the complement of a finite set of bits)
or it isn't. Remember, the underlying representation is an int.

 Of course the answer
> depends on how you store ~ in the first place, but for most choices, str
> is not obvious. For example, using fixed-size unsigned with automatic
> highest-bit detection, for an enum with READ, WRITE, EXEC, STICKY,
> ~(READ) is the same value as (WRITE|EXEC|STICKY),

Why would that be the case?

> Yeah, open flags are especially screwy, where flags & 3 has a special
> non-bitmapped meaning but the rest of the bits are flags. There are other
> cases where multiple ints, only some of which are bitmaps, are packed as
> separate bitfields into the same int (e.g., TCP/IP headers), but I don't
> know of any others that pretend to be a single bitmap even though they
> aren't, so that's really a unique problem, which can be ignored.

Technically file modes have the file type field (flags & 0xF000) which
has some states that don't mean anything on common platforms.

From luciano at ramalho.org  Thu Mar  5 17:15:58 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Thu, 5 Mar 2015 13:15:58 -0300
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>

I like the IntFlags concept, as long as it's clearly separated from
IntEnum -- it seems that's the consensus now.

I don't like the name IntFlags: BitFlags makes more sense to me, since
the key feature is supporting bitwise operators.

Calling it BitFlags has the additional advantage of making it very
clear that it's not closely related to IntEnum.

Best,

Luciano

On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Enum and IntEnum classes allow constants to have nice str() and repr()
> representations.
>
>>>> socket.AF_INET
> <AddressFamily.AF_INET: 2>
>>>> socket.socket()
> <socket.socket fd=3, family=AddressFamily.AF_INET,
> type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
>
> But when integer constants are flags that should be ORed, IntEnum doesn't
> help, because the result of bitwise OR of two IntEnum instances is int, and
> this value can't be represented as IntEnum.
>
> We need new type IntFlags. It is like IntEnum, but has differences:
>
> 1. The value of an instance should be not limited to the set of predefined
> constants. It can be a combination of predefined constants or even arbitrary
> integer.
>
> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be
> an instance of the same IntFlags subclass.
>
> 3. It should have nice str() and repr().
>
>>>> print(stat.S_IROTH | stat.S_IWOTH)
> stat.S_IROTH|stat.S_IWOTH
>>>> stat.S_IROTH | stat.S_IWOTH
> <StatFlags.S_IROTH|S_IWOTH: 6>
>
> Any thoughts?
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

From eltoder at gmail.com  Thu Mar  5 17:58:06 2015
From: eltoder at gmail.com (Eugene Toder)
Date: Thu, 5 Mar 2015 11:58:06 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
Message-ID: <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>

Maybe even call it BitSet, and model the interface based on frozenset,
except that every element is also a set of one. The operations will be:
X | Y -- union
X & Y -- intersection
X ^ Y -- symmetric difference
X - Y -- difference
X in Y == (X & Y) == X
len(X) -- number of set bits
bool(X) -- any bits set
isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X
(Note no negation.)

Eugene

On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho <luciano at ramalho.org>
wrote:

> I like the IntFlags concept, as long as it's clearly separated from
> IntEnum -- it seems that's the consensus now.
>
> I don't like the name IntFlags: BitFlags makes more sense to me, since
> the key feature is supporting bitwise operators.
>
> Calling it BitFlags has the additional advantage of making it very
> clear that it's not closely related to IntEnum.
>
> Best,
>
> Luciano
>
> On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> > Enum and IntEnum classes allow constants to have nice str() and repr()
> > representations.
> >
> >>>> socket.AF_INET
> > <AddressFamily.AF_INET: 2>
> >>>> socket.socket()
> > <socket.socket fd=3, family=AddressFamily.AF_INET,
> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
> >
> > But when integer constants are flags that should be ORed, IntEnum doesn't
> > help, because the result of bitwise OR of two IntEnum instances is int,
> and
> > this value can't be represented as IntEnum.
> >
> > We need new type IntFlags. It is like IntEnum, but has differences:
> >
> > 1. The value of an instance should be not limited to the set of
> predefined
> > constants. It can be a combination of predefined constants or even
> arbitrary
> > integer.
> >
> > 2. The result of "|", "&" and "~" operators for IntFlags arguments
> should be
> > an instance of the same IntFlags subclass.
> >
> > 3. It should have nice str() and repr().
> >
> >>>> print(stat.S_IROTH | stat.S_IWOTH)
> > stat.S_IROTH|stat.S_IWOTH
> >>>> stat.S_IROTH | stat.S_IWOTH
> > <StatFlags.S_IROTH|S_IWOTH: 6>
> >
> > Any thoughts?
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Luciano Ramalho
> Twitter: @ramalhoorg
>
> Professor em: http://python.pro.br
> Twitter: @pythonprobr
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/0e700618/attachment.html>

From luciano at ramalho.org  Thu Mar  5 18:03:35 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Thu, 5 Mar 2015 14:03:35 -0300
Subject: [Python-ideas] IntFlags
In-Reply-To: <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
Message-ID: <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>

The problem is that a set is by definition unordered, but the position
of the bits in BitFlags is crucial, so the name BitSet may not give
the right idea.

On Thu, Mar 5, 2015 at 1:58 PM, Eugene Toder <eltoder at gmail.com> wrote:
> Maybe even call it BitSet, and model the interface based on frozenset,
> except that every element is also a set of one. The operations will be:
> X | Y -- union
> X & Y -- intersection
> X ^ Y -- symmetric difference
> X - Y -- difference
> X in Y == (X & Y) == X
> len(X) -- number of set bits
> bool(X) -- any bits set
> isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X
> (Note no negation.)
>
> Eugene
>
> On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho <luciano at ramalho.org>
> wrote:
>>
>> I like the IntFlags concept, as long as it's clearly separated from
>> IntEnum -- it seems that's the consensus now.
>>
>> I don't like the name IntFlags: BitFlags makes more sense to me, since
>> the key feature is supporting bitwise operators.
>>
>> Calling it BitFlags has the additional advantage of making it very
>> clear that it's not closely related to IntEnum.
>>
>> Best,
>>
>> Luciano
>>
>> On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>> > Enum and IntEnum classes allow constants to have nice str() and repr()
>> > representations.
>> >
>> >>>> socket.AF_INET
>> > <AddressFamily.AF_INET: 2>
>> >>>> socket.socket()
>> > <socket.socket fd=3, family=AddressFamily.AF_INET,
>> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
>> >
>> > But when integer constants are flags that should be ORed, IntEnum
>> > doesn't
>> > help, because the result of bitwise OR of two IntEnum instances is int,
>> > and
>> > this value can't be represented as IntEnum.
>> >
>> > We need new type IntFlags. It is like IntEnum, but has differences:
>> >
>> > 1. The value of an instance should be not limited to the set of
>> > predefined
>> > constants. It can be a combination of predefined constants or even
>> > arbitrary
>> > integer.
>> >
>> > 2. The result of "|", "&" and "~" operators for IntFlags arguments
>> > should be
>> > an instance of the same IntFlags subclass.
>> >
>> > 3. It should have nice str() and repr().
>> >
>> >>>> print(stat.S_IROTH | stat.S_IWOTH)
>> > stat.S_IROTH|stat.S_IWOTH
>> >>>> stat.S_IROTH | stat.S_IWOTH
>> > <StatFlags.S_IROTH|S_IWOTH: 6>
>> >
>> > Any thoughts?
>> >
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> Luciano Ramalho
>> Twitter: @ramalhoorg
>>
>> Professor em: http://python.pro.br
>> Twitter: @pythonprobr
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>



-- 
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

From storchaka at gmail.com  Thu Mar  5 18:11:48 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 05 Mar 2015 19:11:48 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org> <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
Message-ID: <mda2ol$d97$1@ger.gmane.org>

On 05.03.15 01:42, Andrew Barnert wrote:
> I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course).

Sorry, I don't understand your argument. Why you can't compare 
complemented IntFlags with complemented int?

~(os.OpenMode.O_CLOEXEC) == int(~os.OpenMode.O_CLOEXEC) == 
~int(os.OpenMode.O_CLOEXEC) == posix.O_CLOEXEC

IntFlags is an int subclass and behaves as plain int, except that it has 
special repr and results of bitwise operations preserve a type.

> * The parenthetical brings up another issue: if you look in your platform's sys/stat.h (or whichever header actually defines these things), S_IRWXG is probably not defined as 0o700, but as S_IRGRP | S_IWGRP | S_IXGRP. Can we do that in the enum definition? If not, it may be less readable than the C. (In fact, on many platforms, S_IRGRP is itself defined as something like _S_IREAD << _S_GRP, and S_IRWXG may be defined as (_S_IREAD | _S_IWRITE | _S_IEXEC) << _S_GRP. I think Linux took this even further and defined it as S_IRWXU >> (_S_USR - _S_GRP) or something silly. But at that point, not being able to clone the C no longer looks like a loss of readability...)

Yes, of course you can define:

class Permissions(enum.IntFlags):
     S_IRGRP = 0o0040  # read by group
     S_IWGRP = 0o0020  # write by group
     S_IXGRP = 0o0010  # execute by group
     S_IRWXG = S_IRGRP | S_IWGRP | S_IXGRP
     ...

or what your like. I tried and this works.


From eltoder at gmail.com  Thu Mar  5 18:14:09 2015
From: eltoder at gmail.com (Eugene Toder)
Date: Thu, 5 Mar 2015 12:14:09 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
Message-ID: <CA+KNMz=EdOsi_ZD_Y+XQeZcUtbfm9R5ZM-rjFm0hdqdP0Fq5mA@mail.gmail.com>

On Thu, Mar 5, 2015 at 12:03 PM, Luciano Ramalho <luciano at ramalho.org>
wrote:

> The problem is that a set is by definition unordered, but the position
> of the bits in BitFlags is crucial, so the name BitSet may not give
> the right idea.

The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY |
OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC |
OpenMode.WRONLY or any other permutation. In other words, flags (aka
bitmasks) are just an optimized representation of an (unordered) set of
small integer values.

Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/1d5730b0/attachment.html>

From luciano at ramalho.org  Thu Mar  5 18:20:27 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Thu, 5 Mar 2015 14:20:27 -0300
Subject: [Python-ideas] IntFlags
In-Reply-To: <CA+KNMz=EdOsi_ZD_Y+XQeZcUtbfm9R5ZM-rjFm0hdqdP0Fq5mA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
 <CA+KNMz=EdOsi_ZD_Y+XQeZcUtbfm9R5ZM-rjFm0hdqdP0Fq5mA@mail.gmail.com>
Message-ID: <CALxg4FUmXOO8UoSCLyqXawnfPCK0tF4ChiPJBTBTG40HQVqt=w@mail.gmail.com>

On Thu, Mar 5, 2015 at 2:14 PM, Eugene Toder <eltoder at gmail.com> wrote:
> The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC | OpenMode.WRONLY or any other permutation. In other words, flags (aka bitmasks) are just an optimized representation of an (unordered) set of small integer values.

I agree that they are unordered in the common use cases. But the kind
of low-level code where such flags are used often involves building a
byte or a word out of those flags to pass to a low-level C API. How do
you make sure that the byte you will build makes sense if you don't
care about the position, or the log2 of each flag value? So I
definitely think ordering and positioning are to be preserved in a
data structure intended to manage and combine bits.

Best,

Luciano




-- 
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

From storchaka at gmail.com  Thu Mar  5 18:26:40 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 05 Mar 2015 19:26:40 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
Message-ID: <mda3l5$ttp$1@ger.gmane.org>

On 05.03.15 18:15, Luciano Ramalho wrote:
> I don't like the name IntFlags: BitFlags makes more sense to me, since
> the key feature is supporting bitwise operators.

I chose this name because the concept IntFlags is very similar to the 
concept of Flags enums in C#. The Flags decorator in C# is as close to 
IntFlags as enums in C# close to IntEnum. The Int prefix is here because 
IntFlags is just an funny int (as IntEnum) and both IntFlags and IntEnum 
are purposed to replace int constants.

> Calling it BitFlags has the additional advantage of making it very
> clear that it's not closely related to IntEnum.

But it is closely related to IntEnum. Both are int subclass and fully 
compatible with ints, both provides named constants, both have funny str 
and repr, both inherit common useful interface from Enum, both are 
purposed to replace integer constants.



From mistersheik at gmail.com  Thu Mar  5 18:29:22 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 12:29:22 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <mda3l5$ttp$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
Message-ID: <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>

Have you looked at my IntFields generalization of IntFlags?  It seems that
many of your examples (permissions, e.g.) are better expressed with fields
than with flags.

On Thu, Mar 5, 2015 at 12:26 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 05.03.15 18:15, Luciano Ramalho wrote:
>
>> I don't like the name IntFlags: BitFlags makes more sense to me, since
>> the key feature is supporting bitwise operators.
>>
>
> I chose this name because the concept IntFlags is very similar to the
> concept of Flags enums in C#. The Flags decorator in C# is as close to
> IntFlags as enums in C# close to IntEnum. The Int prefix is here because
> IntFlags is just an funny int (as IntEnum) and both IntFlags and IntEnum
> are purposed to replace int constants.
>
>  Calling it BitFlags has the additional advantage of making it very
>> clear that it's not closely related to IntEnum.
>>
>
> But it is closely related to IntEnum. Both are int subclass and fully
> compatible with ints, both provides named constants, both have funny str
> and repr, both inherit common useful interface from Enum, both are purposed
> to replace integer constants.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/9cd9cee8/attachment.html>

From eltoder at gmail.com  Thu Mar  5 18:42:08 2015
From: eltoder at gmail.com (Eugene Toder)
Date: Thu, 5 Mar 2015 12:42:08 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FUmXOO8UoSCLyqXawnfPCK0tF4ChiPJBTBTG40HQVqt=w@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
 <CA+KNMz=EdOsi_ZD_Y+XQeZcUtbfm9R5ZM-rjFm0hdqdP0Fq5mA@mail.gmail.com>
 <CALxg4FUmXOO8UoSCLyqXawnfPCK0tF4ChiPJBTBTG40HQVqt=w@mail.gmail.com>
Message-ID: <CA+KNMz=vQLJPiAxWOgxMohDD1g0UDskPmtDM9iS1cqefENjUhg@mail.gmail.com>

On Thu, Mar 5, 2015 at 12:20 PM, Luciano Ramalho <luciano at ramalho.org>
wrote:

> On Thu, Mar 5, 2015 at 2:14 PM, Eugene Toder <eltoder at gmail.com> wrote:
> > The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY |
> OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC |
> OpenMode.WRONLY or any other permutation. In other words, flags (aka
> bitmasks) are just an optimized representation of an (unordered) set of
> small integer values.
>
> I agree that they are unordered in the common use cases. But the kind
> of low-level code where such flags are used often involves building a
> byte or a word out of those flags to pass to a low-level C API. How do
> you make sure that the byte you will build makes sense if you don't
> care about the position, or the log2 of each flag value? So I
> definitely think ordering and positioning are to be preserved in a
> data structure intended to manage and combine bits.
>
I think you conflate the two separate issues: ordering of the elements in
the set, and the bit representation of the set. Flags don't preserve the
former, but guarantee the later. Normal sets don't preserve the former too,
and the latter doesn't make sense for them. BitSet would not preserve the
former, but add the later in the way compatible with flags. Specifically,
when you define a BitSet, you assign every possible element a specific bit
position (i.e. integer value). When you perform bit operations the natural
thing happens, and you get the expected bit representation.

Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/578dd76b/attachment-0001.html>

From storchaka at gmail.com  Thu Mar  5 18:57:58 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 05 Mar 2015 19:57:58 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
Message-ID: <mda5f7$u0b$1@ger.gmane.org>

On 05.03.15 19:29, Neil Girdhar wrote:
> Have you looked at my IntFields generalization of IntFlags?  It seems
> that many of your examples (permissions, e.g.) are better expressed with
> fields than with flags.

It looks too complicated for such simple case. And it has an interface 
incompatible with plain int.



From chris.barker at noaa.gov  Thu Mar  5 18:11:29 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 5 Mar 2015 09:11:29 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
Message-ID: <CALGmxEKF0fjESq=CoEFK2crmLb0YC1qiYp8SyqV1SUw7E+EQUA@mail.gmail.com>

On Thu, Mar 5, 2015 at 9:03 AM, Luciano Ramalho <luciano at ramalho.org> wrote:

> The problem is that a set is by definition unordered, but the position
> of the bits in BitFlags is crucial,


Is it? or is that an implementation detail?

What I'm getting at is that if you use an integer to store bits, then the
nth bit is, well, the nth bit in the value. But you could conceptually
think of it as the bit with the name, 'n', in which case order no longer
matters.

-Chris




> so the name BitSet may not give
> the right idea.
>
> On Thu, Mar 5, 2015 at 1:58 PM, Eugene Toder <eltoder at gmail.com> wrote:
> > Maybe even call it BitSet, and model the interface based on frozenset,
> > except that every element is also a set of one. The operations will be:
> > X | Y -- union
> > X & Y -- intersection
> > X ^ Y -- symmetric difference
> > X - Y -- difference
> > X in Y == (X & Y) == X
> > len(X) -- number of set bits
> > bool(X) -- any bits set
> > isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X
> > (Note no negation.)
> >
> > Eugene
> >
> > On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho <luciano at ramalho.org>
> > wrote:
> >>
> >> I like the IntFlags concept, as long as it's clearly separated from
> >> IntEnum -- it seems that's the consensus now.
> >>
> >> I don't like the name IntFlags: BitFlags makes more sense to me, since
> >> the key feature is supporting bitwise operators.
> >>
> >> Calling it BitFlags has the additional advantage of making it very
> >> clear that it's not closely related to IntEnum.
> >>
> >> Best,
> >>
> >> Luciano
> >>
> >> On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka <storchaka at gmail.com>
> >> wrote:
> >> > Enum and IntEnum classes allow constants to have nice str() and repr()
> >> > representations.
> >> >
> >> >>>> socket.AF_INET
> >> > <AddressFamily.AF_INET: 2>
> >> >>>> socket.socket()
> >> > <socket.socket fd=3, family=AddressFamily.AF_INET,
> >> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
> >> >
> >> > But when integer constants are flags that should be ORed, IntEnum
> >> > doesn't
> >> > help, because the result of bitwise OR of two IntEnum instances is
> int,
> >> > and
> >> > this value can't be represented as IntEnum.
> >> >
> >> > We need new type IntFlags. It is like IntEnum, but has differences:
> >> >
> >> > 1. The value of an instance should be not limited to the set of
> >> > predefined
> >> > constants. It can be a combination of predefined constants or even
> >> > arbitrary
> >> > integer.
> >> >
> >> > 2. The result of "|", "&" and "~" operators for IntFlags arguments
> >> > should be
> >> > an instance of the same IntFlags subclass.
> >> >
> >> > 3. It should have nice str() and repr().
> >> >
> >> >>>> print(stat.S_IROTH | stat.S_IWOTH)
> >> > stat.S_IROTH|stat.S_IWOTH
> >> >>>> stat.S_IROTH | stat.S_IWOTH
> >> > <StatFlags.S_IROTH|S_IWOTH: 6>
> >> >
> >> > Any thoughts?
> >> >
> >> > _______________________________________________
> >> > Python-ideas mailing list
> >> > Python-ideas at python.org
> >> > https://mail.python.org/mailman/listinfo/python-ideas
> >> > Code of Conduct: http://python.org/psf/codeofconduct/
> >>
> >>
> >>
> >> --
> >> Luciano Ramalho
> >> Twitter: @ramalhoorg
> >>
> >> Professor em: http://python.pro.br
> >> Twitter: @pythonprobr
> >> _______________________________________________
> >> Python-ideas mailing list
> >> Python-ideas at python.org
> >> https://mail.python.org/mailman/listinfo/python-ideas
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
>
>
>
> --
> Luciano Ramalho
> Twitter: @ramalhoorg
>
> Professor em: http://python.pro.br
> Twitter: @pythonprobr
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/85e41326/attachment.html>

From greg.ewing at canterbury.ac.nz  Thu Mar  5 21:30:39 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Mar 2015 09:30:39 +1300
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
 <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>
 <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com>
Message-ID: <54F8BCEF.1070806@canterbury.ac.nz>

Concerning ~, if you think of it as a set, then ~x should
be the result of exlusive-oring x with an int containing
all the valid bits, i.e. the value obtained by oring all
the defined values together.

This is different from what you would get in C, but it's
self-consistent and the repr() would make sense.

-- 
Greg

From rosuav at gmail.com  Thu Mar  5 22:22:50 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 6 Mar 2015 08:22:50 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALGmxEKF0fjESq=CoEFK2crmLb0YC1qiYp8SyqV1SUw7E+EQUA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
 <CALGmxEKF0fjESq=CoEFK2crmLb0YC1qiYp8SyqV1SUw7E+EQUA@mail.gmail.com>
Message-ID: <CAPTjJmrZNm6Yk5kDEwZmu+MvKVXjhuws87s_-rSCLHf4yS0fxw@mail.gmail.com>

On Fri, Mar 6, 2015 at 4:11 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Thu, Mar 5, 2015 at 9:03 AM, Luciano Ramalho <luciano at ramalho.org> wrote:
>>
>> The problem is that a set is by definition unordered, but the position
>> of the bits in BitFlags is crucial,
>
>
> Is it? or is that an implementation detail?
>
> What I'm getting at is that if you use an integer to store bits, then the
> nth bit is, well, the nth bit in the value. But you could conceptually think
> of it as the bit with the name, 'n', in which case order no longer matters.

The position of bits is the values of things in the set. These two are
more-or-less expressing the same concept:

flags1 = {"ReadOnly", "AllowRead", "CloseOnExec"}
flags2 = FLG_READONLY | FLG_ALLOWREAD | FLG_CLOEXEC

The order of elements in the braced set is meaningless, just as the
order of bitflags in the piped list is meaningless. The positions of
bits in the piped set corresponds to the string names in the braced
set. As the other Chris says, it's basically the bit with name "n",
and then we give it a much more convenient alias.

ChrisA

From eltoder at gmail.com  Thu Mar  5 23:07:04 2015
From: eltoder at gmail.com (Eugene Toder)
Date: Thu, 5 Mar 2015 17:07:04 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F8BCEF.1070806@canterbury.ac.nz>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
 <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>
 <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com>
 <54F8BCEF.1070806@canterbury.ac.nz>
Message-ID: <CA+KNMznYq_ner1rk3rWwRAvLWdRUvQpKVkeLp0KfLaFp8rAKxw@mail.gmail.com>

On Thu, Mar 5, 2015 at 3:30 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Concerning ~, if you think of it as a set, then ~x should
> be the result of exlusive-oring x with an int containing
> all the valid bits, i.e. the value obtained by oring all
> the defined values together.
>
Yes, that will work. Then we might want predefined constants for all set
bits and zero. (Zero can be just calling MyBitSet() with no arguments).
Then ~X is type(X).ALL_ONES - X.
The main question is whether this operation makes much sense. I think in
practice ~ is only used with flags to clear bits (X & ~Y), which can be
done in one explicit operation. It may still be worth supporting ~ to allow
painless transition from plain ints.

Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/552477d7/attachment-0001.html>

From cs at zip.com.au  Thu Mar  5 23:26:18 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 6 Mar 2015 09:26:18 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_kV6z8M5A48SPn1=0woXBVnhWxof4UmHfZf8T7MsWx3nA@mail.gmail.com>
References: <CAA68w_kV6z8M5A48SPn1=0woXBVnhWxof4UmHfZf8T7MsWx3nA@mail.gmail.com>
Message-ID: <20150305222618.GA92958@cskk.homeip.net>

On 05Mar2015 05:04, Neil Girdhar <mistersheik at gmail.com> wrote:
>This is exactly why we should not have & and ~.  It creates the possibility
>to write meaningless code.  Have methods that clear fields and methods that
>set them.  I proposed an interface to IntFields in another message.

I can see the argument against ~, unless you preagree a 1's complelment bit 
range (or equivalent, whatever): feasible, given that you might refuse to 
accept "unknown" flags. With a preagreed range, "~" is perfectly meaningful.

I do not see _any_ argument again "&". I would be very unhappy with an IntFlags 
that didn't accept "&" and "|", and immediately subclass it and never use the 
raw one again.

I'm not sure I am a fan of "remembering negation". Keep it simple. Forbid (or 
document as undefined) ~ in the absense of a range agreement at setup time.

Cheers,
Cameron Simpson <cs at zip.com.au>

Favourite proverb: Ein mann der motorrad fahrt ist fuer immer jung.

From cs at zip.com.au  Thu Mar  5 23:34:03 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 6 Mar 2015 09:34:03 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <20150305223403.GA7148@cskk.homeip.net>

On 03Mar2015 17:52, Serhiy Storchaka <storchaka at gmail.com> wrote:
>Enum and IntEnum classes allow constants to have nice str() and repr() 
>representations. [...]
>3. It should have nice str() and repr().
>>>> print(stat.S_IROTH | stat.S_IWOTH)
>stat.S_IROTH|stat.S_IWOTH
>>>> stat.S_IROTH | stat.S_IWOTH
><StatFlags.S_IROTH|S_IWOTH: 6>

Regarding this one, and regarding some of the later discussion about 
"meaningless" values from some operations, when I wrote my _Clock_Flags I made 
the repr() pull off known flags and include the remainer if not zero. So your 
example above would come out:

  <StatFlags S_IROTH|S_IWOTH 0>
and:
  <StatFlags S_IROTH|S_IWOTH 32>

if there was an extra unknown "32" bit in the mix.

On reflection, I'd like these to come out:

  <StatFlags S_IROTH|S_IWOTH>
  <StatFlags S_IROTH|S_IWOTH|32>

(with space or "." or whatever).

Cheers,
Cameron Simpson <cs at zip.com.au>

Isaac Asimov once remarked that friends had chided him for not patenting the
electronic pocket calculator, since he wrote of similar devices back in the
1940's.  His reply, "Have you ever noticed I only described what it looked
like on the *outside*?" - ijl at mediaone.net

From ckaynor at zindagigames.com  Thu Mar  5 23:34:54 2015
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Thu, 5 Mar 2015 14:34:54 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CA+KNMznYq_ner1rk3rWwRAvLWdRUvQpKVkeLp0KfLaFp8rAKxw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com>
 <BC8DEBE4-530A-4CFD-BCCC-FD8387F5096C@yahoo.com>
 <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com>
 <54F8BCEF.1070806@canterbury.ac.nz>
 <CA+KNMznYq_ner1rk3rWwRAvLWdRUvQpKVkeLp0KfLaFp8rAKxw@mail.gmail.com>
Message-ID: <CALvWhxsRXOkLB07seuvjsa=iTGVqwykT_5tpA8fC629mK2GApQ@mail.gmail.com>

On Thu, Mar 5, 2015 at 2:07 PM, Eugene Toder <eltoder at gmail.com> wrote:
> On Thu, Mar 5, 2015 at 3:30 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
>>
>> Concerning ~, if you think of it as a set, then ~x should
>> be the result of exlusive-oring x with an int containing
>> all the valid bits, i.e. the value obtained by oring all
>> the defined values together.
>
> Yes, that will work. Then we might want predefined constants for all set
> bits and zero. (Zero can be just calling MyBitSet() with no arguments). Then
> ~X is type(X).ALL_ONES - X.
> The main question is whether this operation makes much sense. I think in
> practice ~ is only used with flags to clear bits (X & ~Y), which can be done
> in one explicit operation. It may still be worth supporting ~ to allow
> painless transition from plain ints.

I wonder if it would be worth it to have an ALL value supported, which
defaults to the other values ored together, but could be overridden
for specific cases. This could be useful for compatibility with C code
where there are not publicly defined values for some bits, but they
have some meaning (possibly only for compatibility). If it were purely
computed automatically, handling of cases where there are unused bits
could get messy in some cases. Even if methods are implemented for
clearing flags (the primary case for bit-wise not on a bit field),
supporting a bit-wise not would still be extremely useful for
compatibility with other languages.

In terms of printing, I would expect that, for ease of reading, a
negated flags would print with the negation (storing it as
one-complement), however every other operation would collapse the
negation into the full int value, losing the printing value. Something
like:

>>> READ = 1; WRITE = 2; EXECUTE = 4 # Obviously, different syntax. This would default ALL = 4 | 2 | 1, however it could be overridden in the special cases where other values have meaning. I would imagine the most common override would be to 0xFFFFFFFF, though I could also see others.
>>> a = READ | EXECUTE
>>> a
(READ | EXECUTE)
>>> ~a # Stores it as a negation of the values.
~(READ | EXECUTE)
>>> b = EXECUTE | WRITE
>>> b # Reordered based on values.
(WRITE | EXECUTE)
>>> b & ~a # Collapses the negation into the full bit field, resulting in a non-negated bit field.
WRITE
>>> ~b | ~a
(READ | WRITE)

On Thu, Mar 5, 2015 at 1:22 PM, Chris Angelico <rosuav at gmail.com> wrote:
> As the other Chris says, it's basically the bit with name "n",
> and then we give it a much more convenient alias.

Now that I've joined the conversation, there are TWO other Chris's :).

From cs at zip.com.au  Thu Mar  5 23:42:11 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 6 Mar 2015 09:42:11 +1100
Subject: [Python-ideas] IntFlags
In-Reply-To: <mda3l5$ttp$1@ger.gmane.org>
References: <mda3l5$ttp$1@ger.gmane.org>
Message-ID: <20150305224211.GA35388@cskk.homeip.net>

On 05Mar2015 19:26, Serhiy Storchaka <storchaka at gmail.com> wrote:
>On 05.03.15 18:15, Luciano Ramalho wrote:
>>I don't like the name IntFlags: BitFlags makes more sense to me, since
>>the key feature is supporting bitwise operators.
>
>I chose this name because the concept IntFlags is very similar to the 
>concept of Flags enums in C#. The Flags decorator in C# is as close to 
>IntFlags as enums in C# close to IntEnum. The Int prefix is here 
>because IntFlags is just an funny int (as IntEnum) and both IntFlags 
>and IntEnum are purposed to replace int constants.
>
>>Calling it BitFlags has the additional advantage of making it very
>>clear that it's not closely related to IntEnum.
>
>But it is closely related to IntEnum. Both are int subclass and fully 
>compatible with ints, both provides named constants, both have funny 
>str and repr, both inherit common useful interface from Enum, both are 
>purposed to replace integer constants.

I also prefer IntFlags, like IntEnum.

I certainly have use cases where I do not care that there is an underlying 
numeric value, and am treating things like a set. But with fixed names (versus 
sets, which are open ended in their native form) and using an int underneath is 
both efficient an natural.

But conversely, I have use cases where I do care.

I am -1 on BitFlags.

Cheers,
Cameron Simpson <cs at zip.com.au>

You can be psychotic and still be competent.
        - John L. Young, American Academy of Psychiatry and the Law on Ted
          Kaczynski, and probably most internet users

From greg.ewing at canterbury.ac.nz  Thu Mar  5 21:27:28 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Mar 2015 09:27:28 +1300
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
Message-ID: <54F8BC30.6080307@canterbury.ac.nz>

Luciano Ramalho wrote:
> I don't like the name IntFlags: BitFlags makes more sense to me, since
> the key feature is supporting bitwise operators.

Or perhaps BitSet, emphasising set-like behaviour.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar  5 21:39:11 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Mar 2015 09:39:11 +1300
Subject: [Python-ideas] IntFlags
In-Reply-To: <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <CA+KNMznotjj3KU7kzk_PXZbXUdUKXErdS2jm=Bf45mJT9V+urg@mail.gmail.com>
 <CALxg4FX8-sUTvhimwCh-Wn89EuUpwXX6jFc_3qNote3gTpAyTw@mail.gmail.com>
Message-ID: <54F8BEEF.3070607@canterbury.ac.nz>

Luciano Ramalho wrote:
> The problem is that a set is by definition unordered, but the position
> of the bits in BitFlags is crucial, so the name BitSet may not give
> the right idea.

I think the "Bit" part of BitSet covers that. It's
a set that's represented using bits, and bits have
a position.

Also, the elements of the set are conceptually ints,
and ints are ordered.

-- 
Greg

From storchaka at gmail.com  Fri Mar  6 01:41:18 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 06 Mar 2015 02:41:18 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F8BC30.6080307@canterbury.ac.nz>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <54F8BC30.6080307@canterbury.ac.nz>
Message-ID: <mdat43$ecc$1@ger.gmane.org>

On 05.03.15 22:27, Greg Ewing wrote:
> Or perhaps BitSet, emphasising set-like behaviour.

bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They 
support such operations as changing a range of bits, but are not 
compatible with ints and sets.


From greg.ewing at canterbury.ac.nz  Fri Mar  6 02:36:14 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Mar 2015 14:36:14 +1300
Subject: [Python-ideas] IntFlags
In-Reply-To: <mdat43$ecc$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <54F8BC30.6080307@canterbury.ac.nz> <mdat43$ecc$1@ger.gmane.org>
Message-ID: <54F9048E.7050402@canterbury.ac.nz>

On 03/06/2015 01:41 PM, Serhiy Storchaka wrote:
>
> bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They
> support such operations as changing a range of bits, but are not compatible
> with ints and sets.

Java has an EnumSet, which is closer to what we're talking about. It
supports set operations and uses a bit vector internally.

-- 
Greg


From abarnert at yahoo.com  Fri Mar  6 03:13:56 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Mar 2015 18:13:56 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <mda2ol$d97$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com>
 <CAPTjJmorP-spxex9Q-3YQ0M-XEgzhjwLFSUJ6SqWxhrWvOOCkA@mail.gmail.com>
 <md7a3d$t1m$1@ger.gmane.org> <DA0FA64D-6BE4-4E8A-B8A6-A08E9179EB36@yahoo.com>
 <mda2ol$d97$1@ger.gmane.org>
Message-ID: <C2A4B6D7-4837-4F1D-A859-374FF8BF356C@yahoo.com>

On Mar 5, 2015, at 9:11, Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 05.03.15 01:42, Andrew Barnert wrote:
>> I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course).
> 
> Sorry, I don't understand your argument. Why you can't compare complemented IntFlags with complemented int?

You can, but what you can't do is compare it with the values that you'll likely get from C, because -0o200001 != 0o37775777777 in Python, whereas they are equal in C.

> ~(os.OpenMode.O_CLOEXEC) == int(~os.OpenMode.O_CLOEXEC) == ~int(os.OpenMode.O_CLOEXEC) == posix.O_CLOEXEC

I certainly hope that's not true, or ~ is a no-op... But even if you add the missing ~, none of these is going to be == the value of an unsigned int holding ~O_CLOEXEC in C.

Again, I already said I think you chose the best tradeoff, but there is a downside compared to treating the values as unsigned or to the other options people have proposed here; that's why it's a tradeoff rather than a one obvious choice.

From abarnert at yahoo.com  Fri Mar  6 03:25:15 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Mar 2015 18:25:15 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <54F9048E.7050402@canterbury.ac.nz>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <54F8BC30.6080307@canterbury.ac.nz> <mdat43$ecc$1@ger.gmane.org>
 <54F9048E.7050402@canterbury.ac.nz>
Message-ID: <D296AC26-799B-4E82-8CDC-C4C6EFB9AB31@yahoo.com>

On Mar 5, 2015, at 17:36, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> On 03/06/2015 01:41 PM, Serhiy Storchaka wrote:
>> 
>> bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They
>> support such operations as changing a range of bits, but are not compatible
>> with ints and sets.
> 
> Java has an EnumSet, which is closer to what we're talking about. It
> supports set operations and uses a bit vector internally.

If you're looking for similar names in use, PyPI has (at least) modules named bitsets, bitset, bitmap, bitstring, bitarray, and flags, which are all related to what's being discussed here, but crucially without the enum-like naming of the bits.

(I've used bitstring and bitarray for things like examining the bits of a TCP header; the others I just saw when searching for bitarray when I couldn't remember its name...)

From mistersheik at gmail.com  Fri Mar  6 05:26:42 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 23:26:42 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <mda5f7$u0b$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
Message-ID: <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>

Even if you constrain yourself to the BitFlags rather than the more general
BitFields, I strongly disagree with the interface that people are proposing
involving & and ~ operators.  In general, good interface design reflects
the way we think about objects ? not their underlying representation.  The
fact is that a BitSet's main operations are set and clear individual bits.
It is as if the BitFlags are a namedtuple with Boolean elements whose
underlying storage happens to be an integer.  Therefore, the interface that
makes the most sense is member access:

my_bit_flags.some_bit = True
my_bit_flags.some_bit = False

I don't see the justification for writing these as

my_bit_flags |= TheBitFlagsClass.some_bit
my_bit_flags &= ~TheBitFlagsClass.some_bit

The second line is particularly terrible because it exposes you to making
mistakes like:

my_bit_flags &= TheBitFlagsClass.some_bit
my_bit_flags |= ~TheBitFlagsClass.some_bit

? both of which are meaningless.

It also makes it hard to convert code between the alternate implementation
of using a namedtuple.  It should be easy to do that in my opinion.

Best,

Neil

On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 05.03.15 19:29, Neil Girdhar wrote:
>
>> Have you looked at my IntFields generalization of IntFlags?  It seems
>> that many of your examples (permissions, e.g.) are better expressed with
>> fields than with flags.
>>
>
> It looks too complicated for such simple case. And it has an interface
> incompatible with plain int.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/6053b631/attachment.html>

From mistersheik at gmail.com  Fri Mar  6 05:35:46 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Thu, 5 Mar 2015 23:35:46 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
Message-ID: <CAA68w_=TVAzJXxN2m-MJNY1q7Td2mi5Ce3SJ3iKTkvvYt2tWnQ@mail.gmail.com>

FYI:

ctypes already has a BitFlags/BitFields mechanism.

class Flags_bits(ctypes.LittleEndianStructure):
    _fields_ = [
            ("logout", c_uint8, 1),
            ("userswitch", c_uint8, 1),
            ("suspend", c_uint8, 1),
            ("idle", c_uint8, 1),
        ]


On Thu, Mar 5, 2015 at 11:26 PM, Neil Girdhar <mistersheik at gmail.com> wrote:

> Even if you constrain yourself to the BitFlags rather than the more
> general BitFields, I strongly disagree with the interface that people are
> proposing involving & and ~ operators.  In general, good interface design
> reflects the way we think about objects ? not their underlying
> representation.  The fact is that a BitSet's main operations are set and
> clear individual bits.  It is as if the BitFlags are a namedtuple with
> Boolean elements whose underlying storage happens to be an integer.
> Therefore, the interface that makes the most sense is member access:
>
> my_bit_flags.some_bit = True
> my_bit_flags.some_bit = False
>
> I don't see the justification for writing these as
>
> my_bit_flags |= TheBitFlagsClass.some_bit
> my_bit_flags &= ~TheBitFlagsClass.some_bit
>
> The second line is particularly terrible because it exposes you to making
> mistakes like:
>
> my_bit_flags &= TheBitFlagsClass.some_bit
> my_bit_flags |= ~TheBitFlagsClass.some_bit
>
> ? both of which are meaningless.
>
> It also makes it hard to convert code between the alternate implementation
> of using a namedtuple.  It should be easy to do that in my opinion.
>
> Best,
>
> Neil
>
> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
>
>> On 05.03.15 19:29, Neil Girdhar wrote:
>>
>>> Have you looked at my IntFields generalization of IntFlags?  It seems
>>> that many of your examples (permissions, e.g.) are better expressed with
>>> fields than with flags.
>>>
>>
>> It looks too complicated for such simple case. And it has an interface
>> incompatible with plain int.
>>
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>> --
>>
>> --- You received this message because you are subscribed to a topic in
>> the Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/python-ideas/L5KfCEXFaII/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150305/e596953a/attachment-0001.html>

From abarnert at yahoo.com  Fri Mar  6 10:28:29 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Mar 2015 01:28:29 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
Message-ID: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>

On Mar 5, 2015, at 20:26, Neil Girdhar <mistersheik at gmail.com> wrote:

> Even if you constrain yourself to the BitFlags rather than the more general BitFields, I strongly disagree with the interface that people are proposing involving & and ~ operators.  In general, good interface design reflects the way we think about objects ? not their underlying representation.

But sometimes the object really is "an integer used as a set of bits in some C structure/protocol field/well-known API". 

For example, if we were designing os.open or mmap or whatever as a Pythonic interface, it wouldn't have a "flags" value that or's together multiple integers. We'd probably have separate keyword-only arguments for the less common flags, etc. But they weren't designed from scratch; they were designed to closely mirror the POSIX APIs. And that doesn't just mean simpler implementation, it means people who are familiar with those APIs know how to use them. It means the vast volumes of tutorials and sample code for opening file handles or mapping memory written for C applies to Python. And so on. So, the interface makes sense.

So, one very good use for something like IntFlags is to allow people to keep using that C sample code (with trivial, easy-to-understand changes), but get better debugging, etc. when they do so--e.g., when you introspect an mmap object, it would be great if it could tell you that it was opened with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to manually convert to bits and reverse-lookup in the docs or the module dict.

Not allowing people to use C-style operations if they use named bits means that someone who wants the advantages of named bits has to rewrite their familiar C-style code. Sure, maybe the result will be more readable (although that's arguable; the suggested alternatives are pretty verbose--especially since people keep suggesting mutating-only APIs...), but it means many people will stick with plain ints rather than rewrite, and those who do rewrite will end up with code that doesn't look like the familiar code that everyone knows how to read from C.

>   The fact is that a BitSet's main operations are set and clear individual bits.  It is as if the BitFlags are a namedtuple with Boolean elements whose underlying storage happens to be an integer.

In the case where you don't really care that the underlying storage is an integer, why use an integer in the first place? Why not use a namedtuple, or a set, or whatever else is appropriate? In the very rare case where you need to store a million of these things (and can't store them even more compactly with array or NumPy or similar), you can go get a third-party lib; the vast majority of the time, there's no advantage to using an integer.

Except, of course, when the underlying representation is the whole point, because you're dealing with an API that's written in terms of integers.

>   Therefore, the interface that makes the most sense is member access:
> 
> my_bit_flags.some_bit = True
> my_bit_flags.some_bit = False
> 
> I don't see the justification for writing these as
> 
> my_bit_flags |= TheBitFlagsClass.some_bit
> my_bit_flags &= ~TheBitFlagsClass.some_bit
> 
> The second line is particularly terrible because it exposes you to making mistakes like:
> 
> my_bit_flags &= TheBitFlagsClass.some_bit
> my_bit_flags |= ~TheBitFlagsClass.some_bit
> 
> ? both of which are meaningless.

No they're not. Put some real names instead of toy names there:

    readable = m.prot
    readable &= ProtFlags.Readable

Now it's true iff m.prot includes the Readable flag.

Of course usually you'd write this in a single line without mutation:

    readable = m.prot & ProtFlags.Readable

But that just goes to show that the primary interface of bit flags is an immutable one; trying to force people to use mutating methods like set_bit and clear_bit is just getting in people's way. (And try to come up with a good name for the non-mutating operation that's obvious and reads like English and isn't approaching the ridiculous Apple level of verbosity you get in Cocoa methods like "bitSetWithBitClear:".)

> It also makes it hard to convert code between the alternate implementation of using a namedtuple.  It should be easy to do that in my opinion.
> 
> Best,
> 
> Neil
> 
> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> On 05.03.15 19:29, Neil Girdhar wrote:
>>> Have you looked at my IntFields generalization of IntFlags?  It seems
>>> that many of your examples (permissions, e.g.) are better expressed with
>>> fields than with flags.
>> 
>> It looks too complicated for such simple case. And it has an interface incompatible with plain int.
>> 
>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> -- 
>> 
>> --- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150306/a55530ea/attachment.html>

From mistersheik at gmail.com  Fri Mar  6 10:42:51 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 6 Mar 2015 04:42:51 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
Message-ID: <CAA68w_=UoxYhT33N0tQsjonZu=1QRm9gYssdZk8eypKOoi-fPw@mail.gmail.com>

On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 5, 2015, at 20:26, Neil Girdhar <mistersheik at gmail.com> wrote:
>
> Even if you constrain yourself to the BitFlags rather than the more
> general BitFields, I strongly disagree with the interface that people are
> proposing involving & and ~ operators.  In general, good interface design
> reflects the way we think about objects ? not their underlying
> representation.
>
>
> But sometimes the object really is "an integer used as a set of bits in
> some C structure/protocol field/well-known API".
>

You can always get that integer by casting to integer.


>
> For example, if we were designing os.open or mmap or whatever as a
> Pythonic interface, it wouldn't have a "flags" value that or's together
> multiple integers. We'd probably have separate keyword-only arguments for
> the less common flags, etc. But they weren't designed from scratch; they
> were designed to closely mirror the POSIX APIs. And that doesn't just mean
> simpler implementation, it means people who are familiar with those APIs
> know how to use them. It means the vast volumes of tutorials and sample
> code for opening file handles or mapping memory written for C applies to
> Python. And so on. So, the interface makes sense.
>

I disagree that there is any need to follow the style of the "vast volumes
of tutorials and sample code in C" when designing Python libraries.  The
goal is for the Python code to be as natural as possible.  Member access,
and building constants using | are natural.  Using &~ to clear a bit is not
natural;  It is a coincidence of implementation that distracts from what is
happening.


>
> So, one very good use for something like IntFlags is to allow people to
> keep using that C sample code (with trivial, easy-to-understand changes),
> but get better debugging, etc. when they do so--e.g., when you introspect
> an mmap object, it would be great if it could tell you that it was opened
> with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to
> manually convert to bits and reverse-lookup in the docs or the module dict.
>

Yes, totally agree.

>
> Not allowing people to use C-style operations if they use named bits means
> that someone who wants the advantages of named bits has to rewrite their
> familiar C-style code. Sure, maybe the result will be more readable
> (although that's arguable; the suggested alternatives are pretty
> verbose--especially since people keep suggesting mutating-only APIs...),
> but it means many people will stick with plain ints rather than rewrite,
> and those who do rewrite will end up with code that doesn't look like the
> familiar code that everyone knows how to read from C.
>

I totally agree with you that there should not only be mutating-only
functions.  I agree that | should be used for comining bit fields or
flags.  However, the people who are "familiar with C" (including me) are
frankly dying :)  Pandering to the past really gets you nowhere.  Try to be
a bit idealistic so that new Python code is natural, succinct, and
human-readable ? rather than the C values of reflecting the underlying
representation in spite of the human being.


>   The fact is that a BitSet's main operations are set and clear individual
> bits.  It is as if the BitFlags are a namedtuple with Boolean elements
> whose underlying storage happens to be an integer.
>
>
> In the case where you don't really care that the underlying storage is an
> integer, why use an integer in the first place? Why not use a namedtuple,
> or a set, or whatever else is appropriate? In the very rare case where you
> need to store a million of these things (and can't store them even more
> compactly with array or NumPy or similar), you can go get a third-party
> lib; the vast majority of the time, there's no advantage to using an
> integer.
>

The main reason is so that you can cast it to "int" and produce something
that some API requires.

>
> Except, of course, when the underlying representation is the whole point,
> because you're dealing with an API that's written in terms of integers.
>
> right.

>   Therefore, the interface that makes the most sense is member access:
>
> my_bit_flags.some_bit = True
> my_bit_flags.some_bit = False
>
> I don't see the justification for writing these as
>
> my_bit_flags |= TheBitFlagsClass.some_bit
> my_bit_flags &= ~TheBitFlagsClass.some_bit
>
> The second line is particularly terrible because it exposes you to making
> mistakes like:
>
> my_bit_flags &= TheBitFlagsClass.some_bit
> my_bit_flags |= ~TheBitFlagsClass.some_bit
>
> ? both of which are meaningless.
>
>
> No they're not. Put some real names instead of toy names there:
>
>     readable = m.prot
>     readable &= ProtFlags.Readable
>
> Now it's true iff m.prot includes the Readable flag.
>
> Of course usually you'd write this in a single line without mutation:
>
>     readable = m.prot & ProtFlags.Readable
>

We both know that the most readable version is just member access, like you
would on any object:

readable = m.prot.readable

This usage of & to filter is unnecessarily complicated.  The fact that the
machine does so is no reason for the programmer to write it so.


>
> But that just goes to show that the primary interface of bit flags is an
> immutable one; trying to force people to use mutating methods like set_bit
> and clear_bit is just getting in people's way. (And try to come up with a
> good name for the non-mutating operation that's obvious and reads like
> English and isn't approaching the ridiculous Apple level of verbosity you
> get in Cocoa methods like "bitSetWithBitClear:".)
>

I agree with you here.  I think you should also have | so that you can
build constants the way you're used to, although I'm not sure about & since
I don't see when you would use it in preference to member access.


> It also makes it hard to convert code between the alternate implementation
> of using a namedtuple.  It should be easy to do that in my opinion.
>
>

>
> Best,
>
> Neil
>
> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
>
>> On 05.03.15 19:29, Neil Girdhar wrote:
>>
>>> Have you looked at my IntFields generalization of IntFlags?  It seems
>>> that many of your examples (permissions, e.g.) are better expressed with
>>> fields than with flags.
>>>
>>
>> It looks too complicated for such simple case. And it has an interface
>> incompatible with plain int.
>>
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>> --
>>
>> --- You received this message because you are subscribed to a topic in
>> the Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/python-ideas/L5KfCEXFaII/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150306/5860385f/attachment-0001.html>

From ncoghlan at gmail.com  Fri Mar  6 14:21:36 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 6 Mar 2015 23:21:36 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <54F5EF25.4030002@stoneleaf.us>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
Message-ID: <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>

On 4 March 2015 at 03:28, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 03/02/2015 02:57 AM, Martin Teichmann wrote:
>>     class Soliton(type):
>>        def __new__(cls, name, bases, ns):
>>            self = super().__new__(name, bases, ns)
>>            return self()
>>
>>     class A(metaclass=Soliton):
>>         def f(self):
>>             print(__class__)
>
> This should be `self.__class__`, yes?

No, it's printing out the type referenced from the definition time
cell*, rather than the type of the object actually passed in to the
method.

Cheers,
Nick.

P.S. *For anyone that isn't already aware, the general gist of how that works:

* The compiler actually special cases "super()" and "__class__" in method bodies
* If it sees either of them, it implicitly adds a "__class__" cell
reference to the function
* In the "super()" case, the call is implicitly rewritten as
"super(<first param>, __class__)"
* __class__ is later filled in with a reference to the class being
defined, once that actually exists

In my view, that's never going to win any awards for "elegance in
language design", but it makes super() so much easier to use I'm happy
to tell my sense of aesthetics to be quiet :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From random832 at fastmail.us  Fri Mar  6 14:51:07 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Fri, 06 Mar 2015 08:51:07 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
Message-ID: <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com>

On Fri, Mar 6, 2015, at 04:28, Andrew Barnert wrote:
> For example, if we were designing os.open or mmap or whatever as a
> Pythonic interface, it wouldn't have a "flags" value that or's together
> multiple integers. We'd probably have separate keyword-only arguments for
> the less common flags, etc. But they weren't designed from scratch; they
> were designed to closely mirror the POSIX APIs.

It makes me wonder what the os module would look like if Windows (or
VMS, OS/2, classic Mac, or whatever other OSes have been supported by
python in the present or past) *didn't* provide close mirrors of the
POSIX APIs as part of their C runtime library.

I mean, it's not like we have opendir and readdir. And while we do have
fork and exec, there's a reason beyond convenience for spawn. Because
those are the functions that _don't_ exist, or don't work right, on
non-Unix platforms.

From marky1991 at gmail.com  Fri Mar  6 15:11:50 2015
From: marky1991 at gmail.com (Mark Young)
Date: Fri, 6 Mar 2015 09:11:50 -0500
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
Message-ID: <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>

Is that documented somewhere? I have never heard of that functionality.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150306/e034bfa6/attachment.html>

From abarnert at yahoo.com  Fri Mar  6 17:09:07 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Mar 2015 08:09:07 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
 <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com>
Message-ID: <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com>

On Mar 6, 2015, at 5:51, random832 at fastmail.us wrote:

> On Fri, Mar 6, 2015, at 04:28, Andrew Barnert wrote:
>> For example, if we were designing os.open or mmap or whatever as a
>> Pythonic interface, it wouldn't have a "flags" value that or's together
>> multiple integers. We'd probably have separate keyword-only arguments for
>> the less common flags, etc. But they weren't designed from scratch; they
>> were designed to closely mirror the POSIX APIs.
> 
> It makes me wonder what the os module would look like if Windows (or
> VMS, OS/2, classic Mac, or whatever other OSes have been supported by
> python in the present or past) *didn't* provide close mirrors of the
> POSIX APIs as part of their C runtime library.

Classic Mac didn't provide anything remotely close to the POSIX APIs. IIRC, you called FSSpecMake to create an FSSpec structure from a volume ID, directory ID, and bare filename, then called FSSpecOpenDF with it, after which you'd call various other APIs to map blocks of data to memory handles.

For that matter, Win16 and early Win9x didn't have anything quite like the POSIX APIs, but mapping CreateFile and HANDLE values to open and file descriptors wouldn't be quite as ridiculous as mapping FSSpecOpenDF and refnum values.

At any rate, I think os.open has always been only available on Unix and NT. The whole point of it is to interface with libraries that want to use file descriptors or to access platform-specific features; if you want cross-platform files, you just call open. That's why the API is so close to the POSIX API; if it tried to wrap things up at a higher level, it wouldn't be able to provide access to flags that only Solaris offers, etc.

> I mean, it's not like we have opendir and readdir. And while we do have
> fork and exec, there's a reason beyond convenience for spawn. Because
> those are the functions that _don't_ exist, or don't work right, on
> non-Unix platforms.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Fri Mar  6 17:41:00 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Mar 2015 08:41:00 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_=UoxYhT33N0tQsjonZu=1QRm9gYssdZk8eypKOoi-fPw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
 <CAA68w_=UoxYhT33N0tQsjonZu=1QRm9gYssdZk8eypKOoi-fPw@mail.gmail.com>
Message-ID: <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com>

On Mar 6, 2015, at 1:42, Neil Girdhar <mistersheik at gmail.com> wrote:

> On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> On Mar 5, 2015, at 20:26, Neil Girdhar <mistersheik at gmail.com> wrote:
>> 
>>> Even if you constrain yourself to the BitFlags rather than the more general BitFields, I strongly disagree with the interface that people are proposing involving & and ~ operators.  In general, good interface design reflects the way we think about objects ? not their underlying representation.
>> 
>> But sometimes the object really is "an integer used as a set of bits in some C structure/protocol field/well-known API". 
> 
> You can always get that integer by casting to integer. 
>  
>> 
>> For example, if we were designing os.open or mmap or whatever as a Pythonic interface, it wouldn't have a "flags" value that or's together multiple integers. We'd probably have separate keyword-only arguments for the less common flags, etc. But they weren't designed from scratch; they were designed to closely mirror the POSIX APIs. And that doesn't just mean simpler implementation, it means people who are familiar with those APIs know how to use them. It means the vast volumes of tutorials and sample code for opening file handles or mapping memory written for C applies to Python. And so on. So, the interface makes sense.
> 
> I disagree that there is any need to follow the style of the "vast volumes of tutorials and sample code in C" when designing Python libraries.  The goal is for the Python code to be as natural as possible.  Member access, and building constants using | are natural.  Using &~ to clear a bit is not natural;  It is a coincidence of implementation that distracts from what is happening.

For the vast majority of libraries, I agree. An XML parser or audio decoder has no need to follow cryptic C API standards.

But libraries that are designed for close-to-the-metal access can be an exception--again, consider os.open, which automatically gives you access to every *nix plafform's platform-specific features.

And wrapping C libraries that don't have much of a Python userbase can be another example. If enough people start using it, someone will write and document a higher-level Pythonic interface, but until that happens, having an interface which closely matches what people can find documentation, StackOverflow help, sample code, etc. for is a huge help. Consider PyGame. Much of it is still sparsely documented, but because it wraps the SDL APIs, you can almost always figure out what you need to do, which is part of the reason it's so popular while higher-level wrappers are not. (The other part of the reason is that it wraps almost all of the functionality of SDL, and nothing else can claim that, and again that's probably because it's a thin wrapper.) Or consider PyWin32: it has almost no documentation, and it's not at all Pythonic, but because you can look up a function on MSDN and directly use the C documentation, it's useful for all those areas of the Win32 API (and third-party COM libraries, etc.) that don't have higher-level wrappers.

And there are plenty of protocols, file formats, etc. for which the documentation is written for C (or is just a C implementation, as with the predecessor to RTSP that I forget the name of) as well.

In an ideal world, everything you wanted would have a high-level, Pythonic API--in fact, everything would be designed for Python in the first place. In the real world, you're better off with a C API than with no API at all.

>> So, one very good use for something like IntFlags is to allow people to keep using that C sample code (with trivial, easy-to-understand changes), but get better debugging, etc. when they do so--e.g., when you introspect an mmap object, it would be great if it could tell you that it was opened with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to manually convert to bits and reverse-lookup in the docs or the module dict.
> 
> Yes, totally agree. 
>> 
>> Not allowing people to use C-style operations if they use named bits means that someone who wants the advantages of named bits has to rewrite their familiar C-style code. Sure, maybe the result will be more readable (although that's arguable; the suggested alternatives are pretty verbose--especially since people keep suggesting mutating-only APIs...), but it means many people will stick with plain ints rather than rewrite, and those who do rewrite will end up with code that doesn't look like the familiar code that everyone knows how to read from C.
> 
> I totally agree with you that there should not only be mutating-only functions.  I agree that | should be used for comining bit fields or flags.  However, the people who are "familiar with C" (including me) are frankly dying :)

People have been saying that for a couple decades now, but there's still tons of functionality--not just system-level stuff, but APIs for high-level things like audio fingerprinting or animating sprites or streaming video or extending a Python interpreter--that only exists in C (or sometimes C++ or ObjC), or with very thin wrappers for higher-level languages. And that's still going to be true for a long time to come.

More importantly, if C really were dead and irrelevant, there would be no need for this proposal; again, the only reason you ever care about packing flags into an int in the first place is for compatibility with C or C-style code. When you don't need that, just use a namedtuple or a set or keyword arguments or whatever in the first place.

> Pandering to the past really gets you nowhere.  Try to be a bit idealistic so that new Python code is natural, succinct, and human-readable ? rather than the C values of reflecting the underlying representation in spite of the human being.
>  
>>>   The fact is that a BitSet's main operations are set and clear individual bits.  It is as if the BitFlags are a namedtuple with Boolean elements whose underlying storage happens to be an integer.
>> 
>> In the case where you don't really care that the underlying storage is an integer, why use an integer in the first place? Why not use a namedtuple, or a set, or whatever else is appropriate? In the very rare case where you need to store a million of these things (and can't store them even more compactly with array or NumPy or similar), you can go get a third-party lib; the vast majority of the time, there's no advantage to using an integer.
> 
> The main reason is so that you can cast it to "int" and produce something that some API requires. 
>> 
>> Except, of course, when the underlying representation is the whole point, because you're dealing with an API that's written in terms of integers.
> right. 
>>>   Therefore, the interface that makes the most sense is member access:
>>> 
>>> my_bit_flags.some_bit = True
>>> my_bit_flags.some_bit = False
>>> 
>>> I don't see the justification for writing these as
>>> 
>>> my_bit_flags |= TheBitFlagsClass.some_bit
>>> my_bit_flags &= ~TheBitFlagsClass.some_bit
>>> 
>>> The second line is particularly terrible because it exposes you to making mistakes like:
>>> 
>>> my_bit_flags &= TheBitFlagsClass.some_bit
>>> my_bit_flags |= ~TheBitFlagsClass.some_bit
>>> 
>>> ? both of which are meaningless.
>> 
>> No they're not. Put some real names instead of toy names there:
>> 
>>     readable = m.prot
>>     readable &= ProtFlags.Readable
>> 
>> Now it's true iff m.prot includes the Readable flag.
>> 
>> Of course usually you'd write this in a single line without mutation:
>> 
>>     readable = m.prot & ProtFlags.Readable
> 
> We both know that the most readable version is just member access, like you would on any object:
> 
> readable = m.prot.readable
> 
> This usage of & to filter is unnecessarily complicated.  The fact that the machine does so is no reason for the programmer to write it so.

Right, so someone should write a higher-level library that wraps up mmap so you don't have to use it. But no one has done so yet, and if you want to use it without waiting another couple decades until someone gets around to it, you're using the C-style API.

>> But that just goes to show that the primary interface of bit flags is an immutable one; trying to force people to use mutating methods like set_bit and clear_bit is just getting in people's way. (And try to come up with a good name for the non-mutating operation that's obvious and reads like English and isn't approaching the ridiculous Apple level of verbosity you get in Cocoa methods like "bitSetWithBitClear:".)
> 
> I agree with you here.  I think you should also have | so that you can build constants the way you're used to, although I'm not sure about & since I don't see when you would use it in preference to member access.

OK, if you have | and &, you automatically have |= and &=. There's no way to implement the former without automatically getting the latter. So if that's your suggestion, it's not possible in the first place, so you have to choose whether we get both or neither.

>>> It also makes it hard to convert code between the alternate implementation of using a namedtuple.  It should be easy to do that in my opinion.
>  
>>> 
>>> Best,
>>> 
>>> Neil
>>> 
>>> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>>> On 05.03.15 19:29, Neil Girdhar wrote:
>>>>> Have you looked at my IntFields generalization of IntFlags?  It seems
>>>>> that many of your examples (permissions, e.g.) are better expressed with
>>>>> fields than with flags.
>>>> 
>>>> It looks too complicated for such simple case. And it has an interface incompatible with plain int.
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>> 
>>>> -- 
>>>> 
>>>> --- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
>>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>> 
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150306/fd6adaeb/attachment-0001.html>

From abarnert at yahoo.com  Fri Mar  6 17:47:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Mar 2015 08:47:09 -0800
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
Message-ID: <4972E8D7-965E-4D3E-8650-9663CD88B240@yahoo.com>

On Mar 6, 2015, at 5:21, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 4 March 2015 at 03:28, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 03/02/2015 02:57 AM, Martin Teichmann wrote:
>>>    class Soliton(type):
>>>       def __new__(cls, name, bases, ns):
>>>           self = super().__new__(name, bases, ns)
>>>           return self()
>>> 
>>>    class A(metaclass=Soliton):
>>>        def f(self):
>>>            print(__class__)
>> 
>> This should be `self.__class__`, yes?
> 
> No, it's printing out the type referenced from the definition time
> cell*, rather than the type of the object actually passed in to the
> method.
> 
> Cheers,
> Nick.
> 
> P.S. *For anyone that isn't already aware, the general gist of how that works:
> 
> * The compiler actually special cases "super()" and "__class__" in method bodies
> * If it sees either of them, it implicitly adds a "__class__" cell
> reference to the function
> * In the "super()" case, the call is implicitly rewritten as
> "super(<first param>, __class__)"
> * __class__ is later filled in with a reference to the class being
> defined, once that actually exists

I think it's clearer to explain it as:

Every method has a nonlocal variable named __class__ that refers to the class it's defined in. (As an optimization, CPython doesn't create a closure cell for __class__ if the method body doesn't need it.)

The only problem with this definition is that someone implementing Python and writing type.__new__ has to know at exactly what point the __class__ cell becomes available. But I don't think an implementation that made it available too early would be considered broken, would it?

> In my view, that's never going to win any awards for "elegance in
> language design", but it makes super() so much easier to use I'm happy
> to tell my sense of aesthetics to be quiet :)
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From mistersheik at gmail.com  Fri Mar  6 18:03:58 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Fri, 6 Mar 2015 12:03:58 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
 <CAA68w_=UoxYhT33N0tQsjonZu=1QRm9gYssdZk8eypKOoi-fPw@mail.gmail.com>
 <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com>
Message-ID: <CAA68w_kS8FGAy9BqrXXGKndVMRiWvudS3FptzxruXjrNrDLxag@mail.gmail.com>

It seems to me that we probably would agree on an interface even if we have
a philosophical disagreement.  I think it's possible to have clean,
Pythonic interface that produces whatever integers you want.

In short, my preferred interface is:

__or__ (and __ior__)
__setattr__ and __getattr__
__int__

and that's it.  Is there really a use case for __and__, or __invert__?

Given that you want to follow C so closely, I'm surprised that you don't
prefer IntFields to IntFlags.  I also gave a couple motivating examples for
fields (here's a third:
http://www.tagwith.com/question_332767_rgb-color-converting-into-565-format
).


Best,

Neil

On Fri, Mar 6, 2015 at 11:41 AM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 6, 2015, at 1:42, Neil Girdhar <mistersheik at gmail.com> wrote:
>
> On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert <abarnert at yahoo.com> wrote:
>
>> On Mar 5, 2015, at 20:26, Neil Girdhar <mistersheik at gmail.com> wrote:
>>
>> Even if you constrain yourself to the BitFlags rather than the more
>> general BitFields, I strongly disagree with the interface that people are
>> proposing involving & and ~ operators.  In general, good interface design
>> reflects the way we think about objects ? not their underlying
>> representation.
>>
>>
>> But sometimes the object really is "an integer used as a set of bits in
>> some C structure/protocol field/well-known API".
>>
>
> You can always get that integer by casting to integer.
>
>
>>
>> For example, if we were designing os.open or mmap or whatever as a
>> Pythonic interface, it wouldn't have a "flags" value that or's together
>> multiple integers. We'd probably have separate keyword-only arguments for
>> the less common flags, etc. But they weren't designed from scratch; they
>> were designed to closely mirror the POSIX APIs. And that doesn't just mean
>> simpler implementation, it means people who are familiar with those APIs
>> know how to use them. It means the vast volumes of tutorials and sample
>> code for opening file handles or mapping memory written for C applies to
>> Python. And so on. So, the interface makes sense.
>>
>
> I disagree that there is any need to follow the style of the "vast volumes
> of tutorials and sample code in C" when designing Python libraries.  The
> goal is for the Python code to be as natural as possible.  Member access,
> and building constants using | are natural.  Using &~ to clear a bit is not
> natural;  It is a coincidence of implementation that distracts from what is
> happening.
>
>
> For the vast majority of libraries, I agree. An XML parser or audio
> decoder has no need to follow cryptic C API standards.
>
> But libraries that are designed for close-to-the-metal access can be an
> exception--again, consider os.open, which automatically gives you access to
> every *nix plafform's platform-specific features.
>

It's just as "close to the metal" to write things with member access.  It's
not as if it's going to be much slower!  It's just a question of how you
express setting and clearing bits.


>
> And wrapping C libraries that don't have much of a Python userbase can be
> another example. If enough people start using it, someone will write and
> document a higher-level Pythonic interface, but until that happens, having
> an interface which closely matches what people can find documentation,
> StackOverflow help, sample code, etc. for is a huge help.
>

Most people use StackOverflow and SO always adapts.


> Consider PyGame. Much of it is still sparsely documented, but because it
> wraps the SDL APIs, you can almost always figure out what you need to do,
> which is part of the reason it's so popular while higher-level wrappers are
> not. (The other part of the reason is that it wraps almost all of the
> functionality of SDL, and nothing else can claim that, and again that's
> probably because it's a thin wrapper.) Or consider PyWin32: it has almost
> no documentation, and it's not at all Pythonic, but because you can look up
> a function on MSDN and directly use the C documentation, it's useful for
> all those areas of the Win32 API (and third-party COM libraries, etc.) that
> don't have higher-level wrappers.
>
> And there are plenty of protocols, file formats, etc. for which the
> documentation is written for C (or is just a C implementation, as with the
> predecessor to RTSP that I forget the name of) as well.
>
> In an ideal world, everything you wanted would have a high-level, Pythonic
> API--in fact, everything would be designed for Python in the first place.
> In the real world, you're better off with a C API than with no API at all.
>

I don't think the above API is so "high level".  I think "x.b = False" is
just better design than "x &= ~Class.b".


>
> So, one very good use for something like IntFlags is to allow people to
>> keep using that C sample code (with trivial, easy-to-understand changes),
>> but get better debugging, etc. when they do so--e.g., when you introspect
>> an mmap object, it would be great if it could tell you that it was opened
>> with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to
>> manually convert to bits and reverse-lookup in the docs or the module dict.
>>
>
> Yes, totally agree.
>
>>
>> Not allowing people to use C-style operations if they use named bits
>> means that someone who wants the advantages of named bits has to rewrite
>> their familiar C-style code. Sure, maybe the result will be more readable
>> (although that's arguable; the suggested alternatives are pretty
>> verbose--especially since people keep suggesting mutating-only APIs...),
>> but it means many people will stick with plain ints rather than rewrite,
>> and those who do rewrite will end up with code that doesn't look like the
>> familiar code that everyone knows how to read from C.
>>
>
> I totally agree with you that there should not only be mutating-only
> functions.  I agree that | should be used for comining bit fields or
> flags.  However, the people who are "familiar with C" (including me) are
> frankly dying :)
>
>
> People have been saying that for a couple decades now, but there's still
> tons of functionality--not just system-level stuff, but APIs for high-level
> things like audio fingerprinting or animating sprites or streaming video
> or extending a Python interpreter--that only exists in C (or sometimes C++
> or ObjC), or with very thin wrappers for higher-level languages. And that's
> still going to be true for a long time to come.
>
> More importantly, if C really were dead and irrelevant, there would be no
> need for this proposal; again, the only reason you ever care about packing
> flags into an int in the first place is for compatibility with C or C-style
> code. When you don't need that, just use a namedtuple or a set or keyword
> arguments or whatever in the first place.
>

I'm not saying it's dead.  I'm saying that pandering to an audience who
knows C is a waste.  I have a feeling that the real inertia has nothing to
do with other people who might know C, and more to do with people like me
and you who want to keep writing things the same way we've been writing
things. Sometimes, we've been doing things the long way, and the next
generation can write things the short way.  It's not much more "high
level".  It's just simpler.


> Pandering to the past really gets you nowhere.  Try to be a bit idealistic
> so that new Python code is natural, succinct, and human-readable ? rather
> than the C values of reflecting the underlying representation in spite of
> the human being.
>
>
>>   The fact is that a BitSet's main operations are set and clear
>> individual bits.  It is as if the BitFlags are a namedtuple with Boolean
>> elements whose underlying storage happens to be an integer.
>>
>>
>> In the case where you don't really care that the underlying storage is an
>> integer, why use an integer in the first place? Why not use a namedtuple,
>> or a set, or whatever else is appropriate? In the very rare case where you
>> need to store a million of these things (and can't store them even more
>> compactly with array or NumPy or similar), you can go get a third-party
>> lib; the vast majority of the time, there's no advantage to using an
>> integer.
>>
>
> The main reason is so that you can cast it to "int" and produce something
> that some API requires.
>
>>
>> Except, of course, when the underlying representation is the whole point,
>> because you're dealing with an API that's written in terms of integers.
>>
>> right.
>
>>   Therefore, the interface that makes the most sense is member access:
>>
>> my_bit_flags.some_bit = True
>> my_bit_flags.some_bit = False
>>
>> I don't see the justification for writing these as
>>
>> my_bit_flags |= TheBitFlagsClass.some_bit
>> my_bit_flags &= ~TheBitFlagsClass.some_bit
>>
>> The second line is particularly terrible because it exposes you to making
>> mistakes like:
>>
>> my_bit_flags &= TheBitFlagsClass.some_bit
>> my_bit_flags |= ~TheBitFlagsClass.some_bit
>>
>> ? both of which are meaningless.
>>
>>
>> No they're not. Put some real names instead of toy names there:
>>
>>     readable = m.prot
>>     readable &= ProtFlags.Readable
>>
>> Now it's true iff m.prot includes the Readable flag.
>>
>> Of course usually you'd write this in a single line without mutation:
>>
>>     readable = m.prot & ProtFlags.Readable
>>
>
> We both know that the most readable version is just member access, like
> you would on any object:
>
> readable = m.prot.readable
>
> This usage of & to filter is unnecessarily complicated.  The fact that the
> machine does so is no reason for the programmer to write it so.
>
>
> Right, so someone should write a higher-level library that wraps up mmap
> so you don't have to use it. But no one has done so yet, and if you want to
> use it without waiting another couple decades until someone gets around to
> it, you're using the C-style API.
>
> But that just goes to show that the primary interface of bit flags is an
>> immutable one; trying to force people to use mutating methods like set_bit
>> and clear_bit is just getting in people's way. (And try to come up with a
>> good name for the non-mutating operation that's obvious and reads like
>> English and isn't approaching the ridiculous Apple level of verbosity you
>> get in Cocoa methods like "bitSetWithBitClear:".)
>>
>
> I agree with you here.  I think you should also have | so that you can
> build constants the way you're used to, although I'm not sure about & since
> I don't see when you would use it in preference to member access.
>
>
> OK, if you have | and &, you automatically have |= and &=. There's no way
> to implement the former without automatically getting the latter. So if
> that's your suggestion, it's not possible in the first place, so you have
> to choose whether we get both or neither.
>
>
It also makes it hard to convert code between the alternate implementation
>> of using a namedtuple.  It should be easy to do that in my opinion.
>>
>>
>
>>
>> Best,
>>
>> Neil
>>
>> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>
>>> On 05.03.15 19:29, Neil Girdhar wrote:
>>>
>>>> Have you looked at my IntFields generalization of IntFlags?  It seems
>>>> that many of your examples (permissions, e.g.) are better expressed with
>>>> fields than with flags.
>>>>
>>>
>>> It looks too complicated for such simple case. And it has an interface
>>> incompatible with plain int.
>>>
>>>
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>> --
>>>
>>> --- You received this message because you are subscribed to a topic in
>>> the Google Groups "python-ideas" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>> topic/python-ideas/L5KfCEXFaII/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> python-ideas+unsubscribe at googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150306/d6820c12/attachment-0001.html>

From random832 at fastmail.us  Fri Mar  6 18:19:12 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Fri, 06 Mar 2015 12:19:12 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org>
 <CALxg4FWk787q60vf=iar9a2JZqE4g1K3WEERtku0ykizWPGfAQ@mail.gmail.com>
 <mda3l5$ttp$1@ger.gmane.org>
 <CAA68w_mWWSLxnaXGP83aZHzkyb0uR4SKGYaswRyCNov4iNYKnw@mail.gmail.com>
 <mda5f7$u0b$1@ger.gmane.org>
 <CAA68w_n=Jo5=vcVtfBDZ9TcV+_xhaFAkc30U11uY5J+DSbWDDA@mail.gmail.com>
 <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com>
 <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com>
 <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com>
Message-ID: <1425662352.4091428.236549341.705B2877@webmail.messagingengine.com>

On Fri, Mar 6, 2015, at 11:09, Andrew Barnert wrote:
> Classic Mac didn't provide anything remotely close to the POSIX APIs.
> IIRC, you called FSSpecMake to create an FSSpec structure from a volume
> ID, directory ID, and bare filename, then called FSSpecOpenDF with it,
> after which you'd call various other APIs to map blocks of data to memory
> handles.
> 
> For that matter, Win16 and early Win9x didn't have anything quite like
> the POSIX APIs, but mapping CreateFile and HANDLE values to open and file
> descriptors wouldn't be quite as ridiculous as mapping FSSpecOpenDF and
> refnum values.

In both cases, open/read/write are in the libraries provided with the
commonly available C compilers on the platform (MPW or Codewarrior in
the Mac case, MSVC or Borland etc in the Windows/DOS case), and it
doesn't really matter that they're not part of the platform proper.

I don't think there's ever been a compiler for Windows or DOS that
_didn't_ provide these functions. It depends on how you define "quite
like" the POSIX APIs, I guess.

Anyway, the source code of the os module for classic mac clearly shows
that it is implemented in terms of pre-existing POSIX-like functions
provided either by the supported build platforms or by something called
"GUSI", rather than directly in terms of the native API.
https://hg.python.org/cpython/file/364638d6434d/Mac/Modules/macmodule.c
- you have to go further back to find a version of python that actually
had dosmodule.c, but it looks basically the same.

> At any rate, I think os.open has always been only available on Unix and
> NT.

Nope. List of functions on the os module on Mac in python 2.0:
https://docs.python.org/release/2.0/mac/module-mac.html

> The whole point of it is to interface with libraries that want to use
> file descriptors or to access platform-specific features; if you want
> cross-platform files, you just call open.

The os module provides the same interface on all platforms _so that_ the
higher-level functions like builtin open can be implemented in terms of
it. It's the implementation (and list of available functions) that's
different between platforms, not the interface.

> That's why the API is so close
> to the POSIX API; if it tried to wrap things up at a higher level, it
> wouldn't be able to provide access to flags that only Solaris offers,
> etc.

Your theory doesn't explain listdir and walk.

The reason the API is so close to the POSIX API is because the work of
emulating that API was already done on these platforms in order to
support ports of Unix C programs, and because no-one saw any value in
supporting platform-specific features not exposed through that layer.

From techtonik at gmail.com  Thu Mar  5 15:23:43 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Mar 2015 17:23:43 +0300
Subject: [Python-ideas] Accessible IDE
In-Reply-To: <35CE8F26-4970-438C-A8B9-E1429252A341@gmail.com>
References: <35CE8F26-4970-438C-A8B9-E1429252A341@gmail.com>
Message-ID: <CAPkN8xJpnyX6uP5+HuLtvu_Fw6GwePmRFWJOpy-_ZnguTrVnNg@mail.gmail.com>

On Tue, Feb 17, 2015 at 12:55 AM, Bryan Duarte <bryan0731 at gmail.com> wrote:
>
> I was wondering if anyone had a suggestion on an accessible IDE for Python using Voiceover on the Mac. I use a Mac with Voiceover to interact with my computer, but can also use Linux or Windows if necessary. I would really like to find a tool that will allow me to use Python with some of the features of an IDE. I have tried:
> ? PyCharm
> ? iPy
> ? iPy Notebook
> ? Idel
> ? and Eclipse
>
> So far Eclipse worked the best but still caused problems. Any suggestions and/or help would be great.

I think it is highly dependent on the reader, and reader is highly
dependent on platform. What reader do you use on Linux and Windows?
Also, I am certain that some web plumbing is very possible to add for
IPython Notebook, because it is just a web page. Do you know which
standards are actual to look into? Is it WAI-ARIA
https://en.wikipedia.org/wiki/WAI-ARIA ?

From techtonik at gmail.com  Thu Mar  5 14:40:05 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Mar 2015 16:40:05 +0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
Message-ID: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>

Hi,

While looking at the code like:

    'os': sysinfo['os'].decode('utf-8'),
    'hostname': sysinfo['hostname'].decode('utf-8'),

I can't really read if the result will be unicode or binary string in
utf-8. It would be more convenient for readability to have these
instead:

    bytes.from(encoding) -> unicode
    unicode.to(encoding) -> bytes

.encode/.decode are confusing, because it Python 2 it was:

    str.encode(encoding) -> str
    str.decode(encoding) -> str

with no encoding info attached.
-- 
anatoly t.

From ethan at stoneleaf.us  Sat Mar  7 09:07:38 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 Mar 2015 00:07:38 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <md4ldf$1a9$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org>
Message-ID: <54FAB1CA.3000105@stoneleaf.us>

On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:

> We need new type IntFlags. It is like IntEnum, but has differences:
> 
> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
> predefined constants or even arbitrary integer.
> 
> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
> 
> 3. It should have nice str() and repr().

As long as we are dreaming  :)

class Stat(IntFlag):
    RDONLY = 1
    NOSUID = 2
    NODEV = 4
    NOEXEC = 8
    SYNCHRONOUS = 16
    MANDLOCK = 64
    WRITE = 128
    APPEND = 256
    NOATIME = 1024
    NODIRATIME = 2048
    RELATIME = 4096

a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
b = Stat.RDONLY
a is b  # False
a == b  # True

c = a
a |= Stat.MANDLOCK
c.MANDLOCK  # 64
b.MANDLOCK  # 0
c is a  # True

repr(a)  # <Stat.MANDLOCK|RDONLY: 65>
repr(b)  # <Stat.RDONLY: 1>

d = b | 32  # undefined value
repr(d)  # <Stat.32|RDONLY: 33>
d.MANDLOCK = True
repr(d)  # <Stat.MANDLOCK|32|RDONLY: 97>

repr(~d)  # <Stat.RELATIME|NODIRATIME|NOATIME|512|APPEND|WRITE|SYNCHONOUS|NOEXEC|NODEV|NOSUID: 8094>

I'm not at all sure I have that last one correct.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/c99f1ce7/attachment.sig>

From storchaka at gmail.com  Sat Mar  7 11:50:46 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 07 Mar 2015 12:50:46 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FAB1CA.3000105@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
Message-ID: <mdel66$c13$1@ger.gmane.org>

On 07.03.15 10:07, Ethan Furman wrote:
> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>
>> We need new type IntFlags. It is like IntEnum, but has differences:
>>
>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
>> predefined constants or even arbitrary integer.
>>
>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
>>
>> 3. It should have nice str() and repr().
>
> As long as we are dreaming  :)
>
> class Stat(IntFlag):
>      RDONLY = 1
>      NOSUID = 2
>      NODEV = 4
>      NOEXEC = 8
>      SYNCHRONOUS = 16
>      MANDLOCK = 64
>      WRITE = 128
>      APPEND = 256
>      NOATIME = 1024
>      NODIRATIME = 2048
>      RELATIME = 4096
>
> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton

IntFlags is purposed to replace existing integer constants (as IntEnum).

globals().update(Stat.__members__)


From rosuav at gmail.com  Sat Mar  7 12:41:35 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 7 Mar 2015 22:41:35 +1100
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
Message-ID: <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>

On Fri, Mar 6, 2015 at 12:40 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> Hi,
>
> While looking at the code like:
>
>     'os': sysinfo['os'].decode('utf-8'),
>     'hostname': sysinfo['hostname'].decode('utf-8'),
>
> I can't really read if the result will be unicode or binary string in
> utf-8.

If it says "decode", the result is a Unicode string. If it says
"encode", the result is bytes. I'm not sure what is difficult here.

> .encode/.decode are confusing, because it Python 2 it was:
>
>     str.encode(encoding) -> str
>     str.decode(encoding) -> str
>
> with no encoding info attached.

$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> "test".decode("utf-8")
u'test'
>>> u"test".encode("utf-8")
'test'

Looks to me like str.decode() -> unicode, and unicode.encode() -> str,
at least for UTF-8 and other encodings that apply to Unicode. Yes,
there are some oddities in Py2:

>>> "74657374".decode("hex")
'test'

in which str.decode returns a str, but AFAIK all of those are buried
away in Python 3:

>>> codecs.decode(b"74657374","hex")
b'test'

So where's the confusion?

ChrisA

From storchaka at gmail.com  Sat Mar  7 14:27:57 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 07 Mar 2015 15:27:57 +0200
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
Message-ID: <mdeuct$l0k$1@ger.gmane.org>

On 05.03.15 15:40, anatoly techtonik wrote:
> While looking at the code like:
>
>      'os': sysinfo['os'].decode('utf-8'),
>      'hostname': sysinfo['hostname'].decode('utf-8'),
>
> I can't really read if the result will be unicode or binary string in
> utf-8. It would be more convenient for readability to have these
> instead:
>
>      bytes.from(encoding) -> unicode
>      unicode.to(encoding) -> bytes

Don't write sysinfo['os'].decode('utf-8') if it confuses you. Write 
str(sysinfo['os'], 'utf-8').



From luciano at ramalho.org  Sat Mar  7 14:42:59 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Sat, 7 Mar 2015 10:42:59 -0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
Message-ID: <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>

On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav at gmail.com> wrote:
> If it says "decode", the result is a Unicode string. If it says
> "encode", the result is bytes. I'm not sure what is difficult here.

Yep. When I teach, I use this mnemonic, which I can now quote from my
book [1] ;-)

[TIP]
====
If you need a memory aid to distinguish `.decode()` from `.encode()`,
convince yourself that a Unicode `str` contains "human" text, while
byte sequences can be cryptic machine core dumps. Therefore, it makes
sense that we *decode* `bytes` to `str` to get human readable text,
and we *encode* text to `bytes` for storage or transmission.
====

[1] http://shop.oreilly.com/product/0636920032519.do

Cheers,

Luciano




>
>> .encode/.decode are confusing, because it Python 2 it was:
>>
>>     str.encode(encoding) -> str
>>     str.decode(encoding) -> str
>>
>> with no encoding info attached.
>
> $ python
> Python 2.7.3 (default, Mar 13 2014, 11:03:55)
> [GCC 4.7.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> "test".decode("utf-8")
> u'test'
>>>> u"test".encode("utf-8")
> 'test'
>
> Looks to me like str.decode() -> unicode, and unicode.encode() -> str,
> at least for UTF-8 and other encodings that apply to Unicode. Yes,
> there are some oddities in Py2:
>
>>>> "74657374".decode("hex")
> 'test'
>
> in which str.decode returns a str, but AFAIK all of those are buried
> away in Python 3:
>
>>>> codecs.decode(b"74657374","hex")
> b'test'
>
> So where's the confusion?
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Luciano Ramalho
Twitter: @ramalhoorg

Professor em: http://python.pro.br
Twitter: @pythonprobr

From abarnert at yahoo.com  Sat Mar  7 14:41:30 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 7 Mar 2015 05:41:30 -0800
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
Message-ID: <54D70D5D-23E6-476F-86BD-23B9475FCCB9@yahoo.com>

On Mar 5, 2015, at 5:40 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> Hi,
> 
> While looking at the code like:
> 
>    'os': sysinfo['os'].decode('utf-8'),
>    'hostname': sysinfo['hostname'].decode('utf-8'),
> 
> I can't really read if the result will be unicode or binary string in
> utf-8. It would be more convenient for readability to have these
> instead:
> 
>    bytes.from(encoding) -> unicode
>    unicode.to(encoding) -> bytes
> 
> .encode/.decode are confusing, because it Python 2 it was:
> 
>    str.encode(encoding) -> str
>    str.decode(encoding) -> str

Except for a handful of confusing cases like hex and rot13, str.decode always returned unicode, and unicode.encode always returned str.

However, str.encode _did_ return str--which it did by first decoding the str with sys.getdefaultencoding() so it had something to encode, which often failed, giving those wonderful error messages about being unable to decode ASCII when you asked it to encode UTF-8. It's little wonder that so many novices resorted to throwing in extra random calls to encode and decode and the str constructor until it seemed to work for one example, and then threw their hands up in despair when it didn't work for the next example input.

Python 3 fixed that, simply by getting rid of the Unicode decode and bytes encode methods (which were occasionally useful for codecs like hex and rot13, but far more often just confusing). So now you can only encode Unicode to bytes, and decode bytes to Unicode, and you're the first person I've seen who found the Python 3 API at all confusing. I suspect that's because you didn't realize you were confused by Python 2, and carried that confusion over to 3, and somehow didn't run into any problems for the first few years?

> with no encoding info attached.
> -- 
> anatoly t.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Sat Mar  7 14:53:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 7 Mar 2015 05:53:09 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FAB1CA.3000105@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
Message-ID: <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>

On Mar 7, 2015, at 12:07 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>> 
>> We need new type IntFlags. It is like IntEnum, but has differences:
>> 
>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
>> predefined constants or even arbitrary integer.
>> 
>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
>> 
>> 3. It should have nice str() and repr().
> 
> As long as we are dreaming  :)
> 
> class Stat(IntFlag):
>    RDONLY = 1
>    NOSUID = 2
>    NODEV = 4
>    NOEXEC = 8
>    SYNCHRONOUS = 16
>    MANDLOCK = 64
>    WRITE = 128
>    APPEND = 256
>    NOATIME = 1024
>    NODIRATIME = 2048
>    RELATIME = 4096
> 
> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton

Why? Assuming the values are immutable (like int), the only difference is your is test, and I have no idea why your code would care about that. (If you're worried about the "a |=" below changing b--or, worse, changing the constant--there's no reason to worry. Just as "a = 1; a |= 2" doesn't affect any other variable holding the int 1, the same would be true here.)

> b = Stat.RDONLY
> a is b  # False
> a == b  # True
> 
> c = a
> a |= Stat.MANDLOCK
> c.MANDLOCK  # 64
> b.MANDLOCK  # 0
> c is a  # True
> 
> repr(a)  # <Stat.MANDLOCK|RDONLY: 65>
> repr(b)  # <Stat.RDONLY: 1>
> 
> d = b | 32  # undefined value
> repr(d)  # <Stat.32|RDONLY: 33>
> d.MANDLOCK = True
> repr(d)  # <Stat.MANDLOCK|32|RDONLY: 97>
> 
> repr(~d)  # <Stat.RELATIME|NODIRATIME|NOATIME|512|APPEND|WRITE|SYNCHONOUS|NOEXEC|NODEV|NOSUID: 8094>
> 
> I'm not at all sure I have that last one correct.

That last one is the big question. In C, ~d is going to have the 8192, 16384, ... 2b bits set, not just the bits you defined and the gaps in between. Are you sure you want a different result in Python? (Especially if you're on a platform that does something with those extra bits, so you can get values back that actually have them set. Although I don't think that's an issue with stat, it is with lots of other flags from POSIX-land.)

Look at all of the alternatives that have been suggested (round up to power of 256, guess what C would do on your platform with the most likely equivalent definitions, use signed instead of unsigned so the issue moves somewhere else, handle complementing with a special bool, maybe others); why is this one better? Why is it OK to set the undefined 512 bit but not the equally-undefined 8192 bit when someone does ~ on a value of 32?

From ethan at stoneleaf.us  Sat Mar  7 15:50:38 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 Mar 2015 06:50:38 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <mdel66$c13$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <mdel66$c13$1@ger.gmane.org>
Message-ID: <54FB103E.4060107@stoneleaf.us>

On 03/07/2015 02:50 AM, Serhiy Storchaka wrote:
> On 07.03.15 10:07, Ethan Furman wrote:
>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>>
>>> We need new type IntFlags. It is like IntEnum, but has differences:
>>>
>>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
>>> predefined constants or even arbitrary integer.
>>>
>>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
>>>
>>> 3. It should have nice str() and repr().
>>
>> As long as we are dreaming  :)
>>
>> class Stat(IntFlag):
>>      RDONLY = 1
>>      NOSUID = 2
>>      NODEV = 4
>>      NOEXEC = 8
>>      SYNCHRONOUS = 16
>>      MANDLOCK = 64
>>      WRITE = 128
>>      APPEND = 256
>>      NOATIME = 1024
>>      NODIRATIME = 2048
>>      RELATIME = 4096
>>
>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
> 
> IntFlags is purposed to replace existing integer constants (as IntEnum).
> 
> globals().update(Stat.__members__)

And that can still work -- if the flag is accessed from the class (Stat.RDONLY) it will always be the value assigned
(1); if it is accessed from a member, it will be the value assigned /if set/, otherwise 0.

This also has the advantage of supporting both C style operations (x = Stat.WRITE | Stat.APPEND), or the more customary
Python operations (x = Stat.WRITE; x.APPEND = True).

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/0927f716/attachment.sig>

From ethan at stoneleaf.us  Sat Mar  7 16:05:05 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 Mar 2015 07:05:05 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
Message-ID: <54FB13A1.5020008@stoneleaf.us>

On 03/07/2015 05:53 AM, Andrew Barnert wrote:
> On Mar 7, 2015, at 12:07 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
>>>
>>> We need new type IntFlags. It is like IntEnum, but has differences:
>>>
>>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of
>>> predefined constants or even arbitrary integer.
>>>
>>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass.
>>>
>>> 3. It should have nice str() and repr().
>>
>> As long as we are dreaming  :)
>>
>> class Stat(IntFlag):
>>    RDONLY = 1
>>    NOSUID = 2
>>    NODEV = 4
>>    NOEXEC = 8
>>    SYNCHRONOUS = 16
>>    MANDLOCK = 64
>>    WRITE = 128
>>    APPEND = 256
>>    NOATIME = 1024
>>    NODIRATIME = 2048
>>    RELATIME = 4096
>>
>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
> 
> Why?

Because by having mutable instances of Stat we can have more Python operations:

instead of:

  x = Stat(some_Stat_value_from_somewhere)
  x = x | Stat.NOEXEC  # to set the bit

we can say:

  x = Stat(some_Stat_value_from_somewhere)
  x.NOEXEC = True

or, even more illustratively, instead of:

  x = Stat(some_Stat_value_from_somewhere)
  x &= ~Stat.NOEXEC  # to clear the bit

we can have:

  x = Stat(some_Stat_value_from_somewhere)
  x.NOEXEC = False  # much more readable


> Assuming the values are immutable (like int), [...]

Drat.  Instances of Stat would not be immutuable, but that means they can't be thin wrappers around `int`, doesn't it?
Which also means more work around C call sites.  Drat and double-drat.


>> repr(~d)  # <Stat.RELATIME|NODIRATIME|NOATIME|512|APPEND|WRITE|SYNCHONOUS|NOEXEC|NODEV|NOSUID: 8094>
>>
>> I'm not at all sure I have that last one correct.
> 
> That last one is the big question. In C, ~d is going to have the 8192, 16384, ... 2b
> bits set, not just the bits you defined and the gaps in between. Are you sure you want
> a different result in Python? (Especially if you're on a platform that does something
> with those extra bits, so you can get values back that actually have them set.
> Although I don't think that's an issue with stat, it is with lots of other flags from
> POSIX-land.)

A `bytes` (or `byte_size`) would need to be set to control how many bits got flipped.  If not set, then only defined
bits get flipped.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/b5a8c36f/attachment.sig>

From mistersheik at gmail.com  Sat Mar  7 19:21:44 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 7 Mar 2015 13:21:44 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FB13A1.5020008@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
Message-ID: <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>

On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/07/2015 05:53 AM, Andrew Barnert wrote:
> > On Mar 7, 2015, at 12:07 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> >>
> >>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote:
> >>>
> >>> We need new type IntFlags. It is like IntEnum, but has differences:
> >>>
> >>> 1. The value of an instance should be not limited to the set of
> predefined constants. It can be a combination of
> >>> predefined constants or even arbitrary integer.
> >>>
> >>> 2. The result of "|", "&" and "~" operators for IntFlags arguments
> should be an instance of the same IntFlags subclass.
> >>>
> >>> 3. It should have nice str() and repr().
> >>
> >> As long as we are dreaming  :)
> >>
> >> class Stat(IntFlag):
> >>    RDONLY = 1
> >>    NOSUID = 2
> >>    NODEV = 4
> >>    NOEXEC = 8
> >>    SYNCHRONOUS = 16
> >>    MANDLOCK = 64
> >>    WRITE = 128
> >>    APPEND = 256
> >>    NOATIME = 1024
> >>    NODIRATIME = 2048
> >>    RELATIME = 4096
> >>
> >> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
> >
> > Why?
>
> Because by having mutable instances of Stat we can have more Python
> operations:
>
> instead of:
>
>   x = Stat(some_Stat_value_from_somewhere)
>   x = x | Stat.NOEXEC  # to set the bit
>
> we can say:
>
>   x = Stat(some_Stat_value_from_somewhere)
>   x.NOEXEC = True
>
> or, even more illustratively, instead of:
>
>   x = Stat(some_Stat_value_from_somewhere)
>   x &= ~Stat.NOEXEC  # to clear the bit
>
> we can have:
>
>   x = Stat(some_Stat_value_from_somewhere)
>   x.NOEXEC = False  # much more readable
>

Yes, +1

>
>
> > Assuming the values are immutable (like int), [...]
>
> Drat.  Instances of Stat would not be immutuable, but that means they
> can't be thin wrappers around `int`, doesn't it?
> Which also means more work around C call sites.  Drat and double-drat.
>

Can't your C call sites cast the object to int?


>
>
> >> repr(~d)  #
> <Stat.RELATIME|NODIRATIME|NOATIME|512|APPEND|WRITE|SYNCHONOUS|NOEXEC|NODEV|NOSUID:
> 8094>
> >>
> >> I'm not at all sure I have that last one correct.
> >
> > That last one is the big question. In C, ~d is going to have the 8192,
> 16384, ... 2b
> > bits set, not just the bits you defined and the gaps in between. Are you
> sure you want
> > a different result in Python? (Especially if you're on a platform that
> does something
> > with those extra bits, so you can get values back that actually have
> them set.
> > Although I don't think that's an issue with stat, it is with lots of
> other flags from
> > POSIX-land.)
>
> A `bytes` (or `byte_size`) would need to be set to control how many bits
> got flipped.  If not set, then only defined
> bits get flipped.
>

Why do you need ~ at all?  Do any API calls that you want to make want the
inverted flags?  Isn't the only point of inverting the bits in order to
clear a field?


>
> --
> ~Ethan~
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/579993c1/attachment.html>

From ethan at stoneleaf.us  Sat Mar  7 19:51:18 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 Mar 2015 10:51:18 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
Message-ID: <54FB48A6.5080405@stoneleaf.us>

On 03/07/2015 10:21 AM, Neil Girdhar wrote:
> On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote:

>> or, even more illustratively, instead of:
>> 
>>   x = Stat(some_Stat_value_from_somewhere)
>>   x &= ~Stat.NOEXEC  # to clear the bit
>> 
>> we can have:
>> 
>>   x = Stat(some_Stat_value_from_somewhere)
>>   x.NOEXEC = False  # much more readable
> 
> Yes, +1 

>> Drat.  Instances of Stat would not be immutuable, but that means they can't be thin wrappers around `int`, doesn't it?
>> Which also means more work around C call sites.  Drat and double-drat.
> 
> Can't your C call sites cast the object to int?

They could, but then you no longer have a drop-in replacement, which is what IntEnum is.  Plus it's a hassle.  So
IntFlag (or whatever it's called) would to be immutable, which means no neat tricks like `obj.bit_name = False`.

> Why do you need ~ at all?  Do any API calls that you want to make want the inverted
> flags?  Isn't the only point of inverting the bits in order to clear a field?

Hopefully somebody else can address this point.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/0fdd0942/attachment-0001.sig>

From storchaka at gmail.com  Sat Mar  7 19:58:33 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 07 Mar 2015 20:58:33 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
Message-ID: <mdfhop$7rg$1@ger.gmane.org>

On 07.03.15 20:21, Neil Girdhar wrote:
> Why do you need ~ at all?  Do any API calls that you want to make want
> the inverted flags?  Isn't the only point of inverting the bits in order
> to clear a field?

Because existing code uses it.

mode = os.stat(path).st_mode
os.chmod(path, mode & ~(stat.S_IRUSR|stat.S_IRGRP|stat.S_IROTH))



From mistersheik at gmail.com  Sat Mar  7 20:02:13 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 7 Mar 2015 14:02:13 -0500
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FB48A6.5080405@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
 <54FB48A6.5080405@stoneleaf.us>
Message-ID: <CAA68w_kFoHkrV-TVnvH1ythYWjoL-mRBHJ_0XZ1+1oeLHvNxeg@mail.gmail.com>

On Sat, Mar 7, 2015 at 1:51 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/07/2015 10:21 AM, Neil Girdhar wrote:
> > On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote:
>
> >> or, even more illustratively, instead of:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x &= ~Stat.NOEXEC  # to clear the bit
> >>
> >> we can have:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x.NOEXEC = False  # much more readable
> >
> > Yes, +1
>
> >> Drat.  Instances of Stat would not be immutuable, but that means they
> can't be thin wrappers around `int`, doesn't it?
> >> Which also means more work around C call sites.  Drat and double-drat.
> >
> > Can't your C call sites cast the object to int?
>
> They could, but then you no longer have a drop-in replacement, which is
> what IntEnum is.  Plus it's a hassle.  So
> IntFlag (or whatever it's called) would to be immutable, which means no
> neat tricks like `obj.bit_name = False`.
>

A drop-in replacement for what?  Is there a lot of Python code that
manipulates flags that you expect people to convert to IntFlags from int?
Isn't the point of IntFlags for writing new code?


> > Why do you need ~ at all?  Do any API calls that you want to make want
> the inverted
> > flags?  Isn't the only point of inverting the bits in order to clear a
> field?
>
> Hopefully somebody else can address this point.
>
> --
> ~Ethan~
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/7bc651df/attachment.html>

From storchaka at gmail.com  Sat Mar  7 20:39:40 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 07 Mar 2015 21:39:40 +0200
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_kFoHkrV-TVnvH1ythYWjoL-mRBHJ_0XZ1+1oeLHvNxeg@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
 <54FB48A6.5080405@stoneleaf.us>
 <CAA68w_kFoHkrV-TVnvH1ythYWjoL-mRBHJ_0XZ1+1oeLHvNxeg@mail.gmail.com>
Message-ID: <mdfk5s$ch1$1@ger.gmane.org>

On 07.03.15 21:02, Neil Girdhar wrote:
> A drop-in replacement for what?  Is there a lot of Python code that
> manipulates flags that you expect people to convert to IntFlags from
> int?  Isn't the point of IntFlags for writing new code?

No, the point of IntFlags to be drop-in replacement for integer 
constants used as flags. See my patch in issue23591.


From g.brandl at gmx.net  Sat Mar  7 21:28:14 2015
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 07 Mar 2015 21:28:14 +0100
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FB13A1.5020008@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
Message-ID: <mdfn0v$o48$1@ger.gmane.org>

On 03/07/2015 04:05 PM, Ethan Furman wrote:

>>> As long as we are dreaming  :)
>>>
>>> class Stat(IntFlag):
>>>    RDONLY = 1
>>>    NOSUID = 2
>>>    NODEV = 4
>>>    NOEXEC = 8
>>>    SYNCHRONOUS = 16
>>>    MANDLOCK = 64
>>>    WRITE = 128
>>>    APPEND = 256
>>>    NOATIME = 1024
>>>    NODIRATIME = 2048
>>>    RELATIME = 4096
>>>
>>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
>>
>> Why?
> 
> Because by having mutable instances of Stat we can have more Python operations:
> 
> instead of:
> 
>   x = Stat(some_Stat_value_from_somewhere)
>   x = x | Stat.NOEXEC  # to set the bit
> 
> we can say:
> 
>   x = Stat(some_Stat_value_from_somewhere)
>   x.NOEXEC = True

Please no.  You're making a mutable type out of something that is conceptually
(and in people's minds) an integer.  Remember how long it can take to understand
that

a = 1
a = 2

does not change the integer "1" to now be "2".

Georg


From ethan at stoneleaf.us  Sat Mar  7 21:38:20 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 07 Mar 2015 12:38:20 -0800
Subject: [Python-ideas] IntFlags
In-Reply-To: <mdfn0v$o48$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us> <mdfn0v$o48$1@ger.gmane.org>
Message-ID: <54FB61BC.8090107@stoneleaf.us>

On 03/07/2015 12:28 PM, Georg Brandl wrote:
> On 03/07/2015 04:05 PM, Ethan Furman wrote:
> 
>>>> As long as we are dreaming  :)
>>>>
>>>> class Stat(IntFlag):
>>>>    RDONLY = 1
>>>>    NOSUID = 2
>>>>    NODEV = 4
>>>>    NOEXEC = 8
>>>>    SYNCHRONOUS = 16
>>>>    MANDLOCK = 64
>>>>    WRITE = 128
>>>>    APPEND = 256
>>>>    NOATIME = 1024
>>>>    NODIRATIME = 2048
>>>>    RELATIME = 4096
>>>>
>>>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
>>>
>>> Why?
>>
>> Because by having mutable instances of Stat we can have more Python operations:
>>
>> instead of:
>>
>>   x = Stat(some_Stat_value_from_somewhere)
>>   x = x | Stat.NOEXEC  # to set the bit
>>
>> we can say:
>>
>>   x = Stat(some_Stat_value_from_somewhere)
>>   x.NOEXEC = True
> 
> Please no.  You're making a mutable type out of something that is conceptually
> (and in people's minds) an integer.  Remember how long it can take to understand
> that
> 
> a = 1
> a = 2
> 
> does not change the integer "1" to now be "2".

Good point.  To do something like that the name would have be BitFlags or something no so solidly tied to "immutable".
At any rate, for this to work would require `int(x)` around every call to a lower-level API, and that's a non-starter.

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150307/028ffce2/attachment.sig>

From mistersheik at gmail.com  Sun Mar  8 10:36:10 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sun, 8 Mar 2015 05:36:10 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FB48A6.5080405@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
 <54FB48A6.5080405@stoneleaf.us>
Message-ID: <CAA68w_nuzc-JcLKQsw4A5zWD=i1Rz8Pf_iGuHqzKnj0r=k06mg@mail.gmail.com>

On Sat, Mar 7, 2015 at 1:51 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/07/2015 10:21 AM, Neil Girdhar wrote:
> > On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote:
>
> >> or, even more illustratively, instead of:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x &= ~Stat.NOEXEC  # to clear the bit
> >>
> >> we can have:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x.NOEXEC = False  # much more readable
> >
> > Yes, +1
>
> >> Drat.  Instances of Stat would not be immutuable, but that means they
> can't be thin wrappers around `int`, doesn't it?
> >> Which also means more work around C call sites.  Drat and double-drat.
> >
> > Can't your C call sites cast the object to int?
>
> They could, but then you no longer have a drop-in replacement, which is
> what IntEnum is.  Plus it's a hassle.  So
> IntFlag (or whatever it's called) would to be immutable, which means no
> neat tricks like `obj.bit_name = False`.
>

Looks like they already do cast to int:

In [3]: class X:
   ...:     def __int__(self): return 0
   ...:

In [4]: os.chmod('a', X())

I really don't see the problem.



>
> > Why do you need ~ at all?  Do any API calls that you want to make want
> the inverted
> > flags?  Isn't the only point of inverting the bits in order to clear a
> field?
>
> Hopefully somebody else can address this point.
>
> --
> ~Ethan~
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150308/91e5959f/attachment-0001.html>

From mistersheik at gmail.com  Sun Mar  8 10:45:30 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sun, 8 Mar 2015 05:45:30 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <54FB61BC.8090107@stoneleaf.us>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <mdfn0v$o48$1@ger.gmane.org> <54FB61BC.8090107@stoneleaf.us>
Message-ID: <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>

On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/07/2015 12:28 PM, Georg Brandl wrote:
> > On 03/07/2015 04:05 PM, Ethan Furman wrote:
> >
> >>>> As long as we are dreaming  :)
> >>>>
> >>>> class Stat(IntFlag):
> >>>>    RDONLY = 1
> >>>>    NOSUID = 2
> >>>>    NODEV = 4
> >>>>    NOEXEC = 8
> >>>>    SYNCHRONOUS = 16
> >>>>    MANDLOCK = 64
> >>>>    WRITE = 128
> >>>>    APPEND = 256
> >>>>    NOATIME = 1024
> >>>>    NODIRATIME = 2048
> >>>>    RELATIME = 4096
> >>>>
> >>>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
> >>>
> >>> Why?
> >>
> >> Because by having mutable instances of Stat we can have more Python
> operations:
> >>
> >> instead of:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x = x | Stat.NOEXEC  # to set the bit
> >>
> >> we can say:
> >>
> >>   x = Stat(some_Stat_value_from_somewhere)
> >>   x.NOEXEC = True
> >
> > Please no.  You're making a mutable type out of something that is
> conceptually
> > (and in people's minds) an integer.  Remember how long it can take to
> understand
> > that
> >
> > a = 1
> > a = 2
> >
> > does not change the integer "1" to now be "2".
>

The fact that the flags are stored in an int ? that is, the implementation
has nothing at all to do with the conceptual nature of flags.  Flags are
properties.  Your knowledge of the underlying implementation is misleading
you.  It would be just as easy and intuitive to implement those flags as
keys and values in a dict.

If flags were conceptually subtypes of int, then you should be able to do
things like:

flags ** 7

or

flags // 91

Do you agree that this is totally meaningless?  There is no "is a"
relationship between Flags and int.  There is a conversion between the
conceptual mapping that is a Flags object to int for the sole purpose of
calling into APIs.


> Good point.  To do something like that the name would have be BitFlags or
> something no so solidly tied to "immutable".
> At any rate, for this to work would require `int(x)` around every call to
> a lower-level API, and that's a non-starter.
>

Correct me if I'm wrong, but doesn't Boost.Python when generating Python
methods that accept ints, automatically call __int__ on the arguments.
Doesn't SWIG do the same?  Can you give me some examples of methods that
accept ints but don't call __int__?  It seems to me to be a bug in those
methods than in the caller.  A method that wants an int should call __int__
on its argument ? not expect that isinstance(X, int) etc.


>
> --
> ~Ethan~
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150308/b9ffe14f/attachment.html>

From ncoghlan at gmail.com  Sun Mar  8 13:28:57 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Mar 2015 22:28:57 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
Message-ID: <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>

On 7 March 2015 at 00:11, Mark Young <marky1991 at gmail.com> wrote:
> Is that documented somewhere? I have never heard of that functionality.

The implicit __class__ cell? PEP 3135 introduced the ability to call
super() without any arguments in Python 3 to ask the interpreter to
just "do the right thing": https://www.python.org/dev/peps/pep-3135/

The actual language reference docs are here:
https://docs.python.org/3/reference/datamodel.html#creating-the-class-object

Cheers,
Nick.

P.S. Those language reference class creation docs weren't fully
updated for the original Python 3.0 release. They were updated in 3.3
when the new dynamic type creation APIs were added to the types module
to simplify PEP 3115 compliant dynamic type creation:
https://docs.python.org/3/library/types.html#dynamic-type-creation


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From g.rodola at gmail.com  Sun Mar  8 14:07:10 2015
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 8 Mar 2015 14:07:10 +0100
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
Message-ID: <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>

On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho <luciano at ramalho.org> wrote:

> On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav at gmail.com> wrote:
> > If it says "decode", the result is a Unicode string. If it says
> > "encode", the result is bytes. I'm not sure what is difficult here.
>
> Yep. When I teach, I use this mnemonic, which I can now quote from my
> book [1] ;-)
>
> [TIP]
> ====
> If you need a memory aid to distinguish `.decode()` from `.encode()`,
> convince yourself that a Unicode `str` contains "human" text, while
> byte sequences can be cryptic machine core dumps. Therefore, it makes
> sense that we *decode* `bytes` to `str` to get human readable text,
> and we *encode* text to `bytes` for storage or transmission.
> ====


This is a great advice (and yes, I also often get confused by the two).


-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150308/ea0727f4/attachment.html>

From mistersheik at gmail.com  Sun Mar  8 16:50:12 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sun, 8 Mar 2015 08:50:12 -0700 (PDT)
Subject: [Python-ideas] IntFlags
In-Reply-To: <mdfk5s$ch1$1@ger.gmane.org>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <CAA68w_nDOQCJS1rcEgXNWOkD9KmdnTh1BpwzA9yADnDjj8LfrA@mail.gmail.com>
 <54FB48A6.5080405@stoneleaf.us>
 <CAA68w_kFoHkrV-TVnvH1ythYWjoL-mRBHJ_0XZ1+1oeLHvNxeg@mail.gmail.com>
 <mdfk5s$ch1$1@ger.gmane.org>
Message-ID: <8ed6ef09-8203-4389-9378-e0dce0190382@googlegroups.com>

For what it's worth, I really like your patch.  My suggestion is to change 
a couple dozen lines so that IntFlags uses composition rather than 
inheritance, and provides setattr and getattr.  As far as I can tell, this 
would still be a "drop-in replacement", although I'm still not sure in 
which contexts this has to work.

Best,

Neil

On Saturday, March 7, 2015 at 2:40:29 PM UTC-5, Serhiy Storchaka wrote:
>
> On 07.03.15 21:02, Neil Girdhar wrote: 
> > A drop-in replacement for what?  Is there a lot of Python code that 
> > manipulates flags that you expect people to convert to IntFlags from 
> > int?  Isn't the point of IntFlags for writing new code? 
>
> No, the point of IntFlags to be drop-in replacement for integer 
> constants used as flags. See my patch in issue23591. 
>
> _______________________________________________ 
> Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150308/e510edb9/attachment.html>

From marky1991 at gmail.com  Sun Mar  8 18:17:41 2015
From: marky1991 at gmail.com (Mark Young)
Date: Sun, 8 Mar 2015 13:17:41 -0400
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
 <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
Message-ID: <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>

Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess
not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150308/c3c5ed84/attachment.html>

From tjreedy at udel.edu  Sun Mar  8 19:31:49 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 08 Mar 2015 14:31:49 -0400
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
Message-ID: <mdi4il$408$1@ger.gmane.org>

On 3/8/2015 9:07 AM, Giampaolo Rodola' wrote:

> On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho

>     [TIP]
>     ====
>     If you need a memory aid to distinguish `.decode()` from `.encode()`,
>     convince yourself that a Unicode `str` contains "human" text, while
>     byte sequences can be cryptic machine core dumps. Therefore, it makes
>     sense that we *decode* `bytes` to `str` to get human readable text,
>     and we *encode* text to `bytes` for storage or transmission.
>     ====
>
>
> This is a great advice (and yes, I also often get confused by the two).

Unless a bytes object simply represents a sequence of counts in 
range(256), which in unusual, then the bytes are an encoding of some 
other information, which in Python is embodied in an object.

Struct.pack (en)codes python ints to bytes.  Here the coding issue is 
big versus little endian.  Struct.unpack decodes bytes back to int 
objects.  PIL can decode (unpack and deserialize) bytes in various image 
encodings to an image object and can likewise encode an image object to 
bytes in various formats.  The common theme is that non-bytes objects 
are encoded to bytes for transmission and storage and decoded back to 
non-bytes objects.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Mon Mar  9 01:12:38 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Mar 2015 10:12:38 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
 <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
 <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>
Message-ID: <CADiSq7d0DohC8scm4C1j_+uOUWqMr_BC-NrQFg-7JKDR+reG3w@mail.gmail.com>

On 9 Mar 2015 03:18, "Mark Young" <marky1991 at gmail.com> wrote:
>
> Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess
not.

Every other reference on that page is to the class attribute rather than
the implicit method nonlocal, so it took a moment for me to relocate it
myself. I should have searched for "super" instead :)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/fe59858f/attachment-0001.html>

From ncoghlan at gmail.com  Mon Mar  9 01:22:25 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Mar 2015 10:22:25 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7d0DohC8scm4C1j_+uOUWqMr_BC-NrQFg-7JKDR+reG3w@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
 <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
 <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>
 <CADiSq7d0DohC8scm4C1j_+uOUWqMr_BC-NrQFg-7JKDR+reG3w@mail.gmail.com>
Message-ID: <CADiSq7e8Ds1XaeiQaxc9d_kpB-+wEEqXLDKoO+w6XACT_=Jyjg@mail.gmail.com>

On 9 Mar 2015 10:12, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>
> On 9 Mar 2015 03:18, "Mark Young" <marky1991 at gmail.com> wrote:
> >
> > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I
guess not.
>
> Every other reference on that page is to the class attribute rather than
the implicit method nonlocal, so it took a moment for me to relocate it
myself. I should have searched for "super" instead :)

With the benefit of hindsight, if I had the chance to do this over, I'd
suggest we call the implicit method nonlocal "__classdef__" instead,
specifically to avoid the name collision with the runtime type information
on object instances. My general impression is that the current name
triggers too many incorrect assumptions for most readers, and the name
collision makes it harder than it needs to be to build a more correct
mental model.

It would actually be possible to still make that transition, with a view to
deprecating the current "__class__" spelling at some point in the distant
future. I'm too close to the problem to judge whether or not it would
actually help reduce confusion, though.

Regards,
Nick.

>
> Cheers,
> Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/575fe5c2/attachment.html>

From abarnert at yahoo.com  Mon Mar  9 13:29:21 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 9 Mar 2015 05:29:21 -0700
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us> <mdfn0v$o48$1@ger.gmane.org>
 <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
Message-ID: <BC89FFBA-E0A3-476C-93B3-E2F724ABABE3@yahoo.com>

On Mar 8, 2015, at 1:45 AM, Neil Girdhar <mistersheik at gmail.com> wrote:
> 
> 
> 
>> On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 03/07/2015 12:28 PM, Georg Brandl wrote:
>> > On 03/07/2015 04:05 PM, Ethan Furman wrote:
>> >
>> >>>> As long as we are dreaming  :)
>> >>>>
>> >>>> class Stat(IntFlag):
>> >>>>    RDONLY = 1
>> >>>>    NOSUID = 2
>> >>>>    NODEV = 4
>> >>>>    NOEXEC = 8
>> >>>>    SYNCHRONOUS = 16
>> >>>>    MANDLOCK = 64
>> >>>>    WRITE = 128
>> >>>>    APPEND = 256
>> >>>>    NOATIME = 1024
>> >>>>    NODIRATIME = 2048
>> >>>>    RELATIME = 4096
>> >>>>
>> >>>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
>> >>>
>> >>> Why?
>> >>
>> >> Because by having mutable instances of Stat we can have more Python operations:
>> >>
>> >> instead of:
>> >>
>> >>   x = Stat(some_Stat_value_from_somewhere)
>> >>   x = x | Stat.NOEXEC  # to set the bit
>> >>
>> >> we can say:
>> >>
>> >>   x = Stat(some_Stat_value_from_somewhere)
>> >>   x.NOEXEC = True
>> >
>> > Please no.  You're making a mutable type out of something that is conceptually
>> > (and in people's minds) an integer.  Remember how long it can take to understand
>> > that
>> >
>> > a = 1
>> > a = 2
>> >
>> > does not change the integer "1" to now be "2".
> 
> The fact that the flags are stored in an int ? that is, the implementation has nothing at all to do with the conceptual nature of flags.  Flags are properties.  Your knowledge of the underlying implementation is misleading you.  It would be just as easy and intuitive to implement those flags as keys and values in a dict.

Usually that's true. And when it's true, you use a dict (or a set or a namedtuple or whatever's appropriate), so you have no need for IntFlags in the first place. The only reason you'd ever want this class is when the fact that the flags are stored in an int is important.

Sure, an IntFlags that isn't a subclass on int can define an __int__ method. But so can some class that doesn't use an int for storage in the first place. If you want to convert back and forth, that's perfectly fine, but why are you using an int for storage?

> If flags were conceptually subtypes of int, then you should be able to do things like:
> 
> flags ** 7
> 
> or
> 
> flags // 91
> 
> Do you agree that this is totally meaningless?  There is no "is a" relationship between Flags and int.  There is a conversion between the conceptual mapping that is a Flags object to int for the sole purpose of calling into APIs.
> 
>> 
>> Good point.  To do something like that the name would have be BitFlags or something no so solidly tied to "immutable".
>> At any rate, for this to work would require `int(x)` around every call to a lower-level API, and that's a non-starter.
> 
> Correct me if I'm wrong, but doesn't Boost.Python when generating Python methods that accept ints, automatically call __int__ on the arguments.  Doesn't SWIG do the same?  Can you give me some examples of methods that accept ints but don't call __int__?  It seems to me to be a bug in those methods than in the caller.  A method that wants an int should call __int__ on its argument ? not expect that isinstance(X, int) etc.
>    
>> 
>> --
>> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/37657e12/attachment.html>

From abarnert at yahoo.com  Mon Mar  9 13:40:03 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 9 Mar 2015 05:40:03 -0700
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <CADiSq7e8Ds1XaeiQaxc9d_kpB-+wEEqXLDKoO+w6XACT_=Jyjg@mail.gmail.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
 <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
 <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>
 <CADiSq7d0DohC8scm4C1j_+uOUWqMr_BC-NrQFg-7JKDR+reG3w@mail.gmail.com>
 <CADiSq7e8Ds1XaeiQaxc9d_kpB-+wEEqXLDKoO+w6XACT_=Jyjg@mail.gmail.com>
Message-ID: <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com>

On Mar 8, 2015, at 5:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 9 Mar 2015 10:12, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> >
> >
> > On 9 Mar 2015 03:18, "Mark Young" <marky1991 at gmail.com> wrote:
> > >
> > > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess not.
> >
> > Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :)
> 
> With the benefit of hindsight, if I had the chance to do this over, I'd suggest we call the implicit method nonlocal "__classdef__" instead, specifically to avoid the name collision with the runtime type information on object instances. My general impression is that the current name triggers too many incorrect assumptions for most readers, and the name collision makes it harder than it needs to be to build a more correct mental model.
> 
I can tell you that every single time I've mentioned the __class__ nonlocal in a StackOverflow answer, someone mistakenly "corrected" me to tell me either "you need to write self.__class__" or "that won't work because it's the runtime type of self, which could be a subclass".

I think a better solution, if you were doing it from scratch, might be to get rid of the __class__ member and force people to use type(self) to get it, so TOOTDI. (Of course that implies that code that needs to change __class__ would have to call some function to do so, but I'm not sure that's a bad thing. Dynamically changing your type is a rare thing to do in Python, and unusual or impossible in most other OO languages, except maybe during initialization in the Smalltalk family, so why shouldn't it be signaled by something scary?)

On the other hand, is __class__ the only thing that's spelled with double underscored that isn't an attribute? Maybe that's the part that's confusing people...
> It would actually be possible to still make that transition, with a view to deprecating the current "__class__" spelling at some point in the distant future. I'm too close to the problem to judge whether or not it would actually help reduce confusion, though.
> 
> Regards,
> Nick.
> 
> >
> > Cheers,
> > Nick.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/1f77dbb8/attachment.html>

From ncoghlan at gmail.com  Mon Mar  9 14:36:59 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Mar 2015 23:36:59 +1000
Subject: [Python-ideas] An even simpler customization of class creation
In-Reply-To: <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com>
References: <CAK9R32Q5NrohxG_Et_023pYrTLA8XJfqFq_6VKbO-6q61BhEnQ@mail.gmail.com>
 <CADiSq7eOgAbR1TWDaEzxRgSoqX6=Ztzs62YPLCuV-Y5uya1isQ@mail.gmail.com>
 <CAK9R32SNm6dvU4ZiiLzyo8ksSo1Xzo0Rh+QQm4BE7RH0EWYaKA@mail.gmail.com>
 <CADiSq7cpB2rNWHRQyiZg1E=VXoVroji3vpLL87gK=n1mZVm=ww@mail.gmail.com>
 <CAK9R32T3KBqX5kysvV5_cTv0gNnrvmsFypRv4kVvfy7S3CyxOQ@mail.gmail.com>
 <54F5EF25.4030002@stoneleaf.us>
 <CADiSq7czWQN402J5hun5rYVt4Fx-1nAvwyv03XsZX1ygroEnLw@mail.gmail.com>
 <CAG3cHabM8wtDqwpmjXNKko3R36+=+mxu1n2FEXajVW3L-2xWPg@mail.gmail.com>
 <CADiSq7c5PbWJKKP5Xq4Pb9QT+q4r4waq_Oe_A85Wam6DkQDZFQ@mail.gmail.com>
 <CAG3cHaZPZEkv=tnVH1FZ3PV9BwMJN8frznDKGLMkdJm7pk_b6A@mail.gmail.com>
 <CADiSq7d0DohC8scm4C1j_+uOUWqMr_BC-NrQFg-7JKDR+reG3w@mail.gmail.com>
 <CADiSq7e8Ds1XaeiQaxc9d_kpB-+wEEqXLDKoO+w6XACT_=Jyjg@mail.gmail.com>
 <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com>
Message-ID: <CADiSq7f6By2z-=qe0xGHu88GN-srV2Rkz=7A-KwsfHgU14d=Cg@mail.gmail.com>

On 9 March 2015 at 22:40, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Mar 8, 2015, at 5:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>
> On 9 Mar 2015 10:12, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>>
>>
>> On 9 Mar 2015 03:18, "Mark Young" <marky1991 at gmail.com> wrote:
>> >
>> > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess
>> > not.
>>
>> Every other reference on that page is to the class attribute rather than
>> the implicit method nonlocal, so it took a moment for me to relocate it
>> myself. I should have searched for "super" instead :)
>
> With the benefit of hindsight, if I had the chance to do this over, I'd
> suggest we call the implicit method nonlocal "__classdef__" instead,
> specifically to avoid the name collision with the runtime type information
> on object instances. My general impression is that the current name triggers
> too many incorrect assumptions for most readers, and the name collision
> makes it harder than it needs to be to build a more correct mental model.
>
> I can tell you that every single time I've mentioned the __class__ nonlocal
> in a StackOverflow answer, someone mistakenly "corrected" me to tell me
> either "you need to write self.__class__" or "that won't work because it's
> the runtime type of self, which could be a subclass".
>
> I think a better solution, if you were doing it from scratch, might be to
> get rid of the __class__ member and force people to use type(self) to get
> it, so TOOTDI.

They're not substitutes for each other, as __class__ lets you lie
about your type to the consumer, while type() doesn't, and this is by
design:

>>> import weakref
>>> class Example():
...     pass
...
>>> a = Example()
>>> b = weakref.proxy(a)
>>> b.__class__
<class '__main__.Example'>
>>> type(b)
<class 'weakproxy'>

For many use cases, you actually want obj.__class__ in order to handle
proxy objects correctly.

> (Of course that implies that code that needs to change
> __class__ would have to call some function to do so, but I'm not sure that's
> a bad thing. Dynamically changing your type is a rare thing to do in Python,
> and unusual or impossible in most other OO languages, except maybe during
> initialization in the Smalltalk family, so why shouldn't it be signaled by
> something scary?)

It's not (just) about being able to dynamically change your type, it's
about the fact that __class__ goes through the descriptor and class
attribute lookup machinery, while type() doesn't.

>
> On the other hand, is __class__ the only thing that's spelled with double
> underscored that isn't an attribute? Maybe that's the part that's confusing
> people...

It's not the only one - implicit module attributes like __name__ work
that way, as do the __debug__ and __import__ builtins.

As far as I can tell, it's specifically the fact that "__class__"
looks like a plausible typo for "self.__class__" that causes problems,
as this also creates the problem that if you search for "__class__"
not only on the data model page, but anywhere, you're far more likely
to get a hit on the attribute (which has been around for as long as
Python has had classes) than you will on the implicit method nonlocal
introduced in Python 3.

Hence why I suspect changing/aliasing the preferred name for
referencing the nonlocal to be __classdef__ may help improve things,
as it better hints at the lexical scoping, where you're referring to
the class where the method is being defined at compile time, rather
than the one where it was retrieved at run time.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mistersheik at gmail.com  Mon Mar  9 14:41:57 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 09:41:57 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <BC89FFBA-E0A3-476C-93B3-E2F724ABABE3@yahoo.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <mdfn0v$o48$1@ger.gmane.org> <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
 <BC89FFBA-E0A3-476C-93B3-E2F724ABABE3@yahoo.com>
Message-ID: <CAA68w_m3bZJiwamb0PetomLj9ssxkxVuzSrtydD8AbGF6DgXfA@mail.gmail.com>

On Mon, Mar 9, 2015 at 8:29 AM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 8, 2015, at 1:45 AM, Neil Girdhar <mistersheik at gmail.com> wrote:
>
>
>
> On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 03/07/2015 12:28 PM, Georg Brandl wrote:
>> > On 03/07/2015 04:05 PM, Ethan Furman wrote:
>> >
>> >>>> As long as we are dreaming  :)
>> >>>>
>> >>>> class Stat(IntFlag):
>> >>>>    RDONLY = 1
>> >>>>    NOSUID = 2
>> >>>>    NODEV = 4
>> >>>>    NOEXEC = 8
>> >>>>    SYNCHRONOUS = 16
>> >>>>    MANDLOCK = 64
>> >>>>    WRITE = 128
>> >>>>    APPEND = 256
>> >>>>    NOATIME = 1024
>> >>>>    NODIRATIME = 2048
>> >>>>    RELATIME = 4096
>> >>>>
>> >>>> a = Stat.RDONLY  # creates a new instance of Stat, not a singleton
>> >>>
>> >>> Why?
>> >>
>> >> Because by having mutable instances of Stat we can have more Python
>> operations:
>> >>
>> >> instead of:
>> >>
>> >>   x = Stat(some_Stat_value_from_somewhere)
>> >>   x = x | Stat.NOEXEC  # to set the bit
>> >>
>> >> we can say:
>> >>
>> >>   x = Stat(some_Stat_value_from_somewhere)
>> >>   x.NOEXEC = True
>> >
>> > Please no.  You're making a mutable type out of something that is
>> conceptually
>> > (and in people's minds) an integer.  Remember how long it can take to
>> understand
>> > that
>> >
>> > a = 1
>> > a = 2
>> >
>> > does not change the integer "1" to now be "2".
>>
>
> The fact that the flags are stored in an int ? that is, the implementation
> has nothing at all to do with the conceptual nature of flags.  Flags are
> properties.  Your knowledge of the underlying implementation is misleading
> you.  It would be just as easy and intuitive to implement those flags as
> keys and values in a dict.
>
>
> Usually that's true. And when it's true, you use a dict (or a set or a
> namedtuple or whatever's appropriate), so you have no need for IntFlags in
> the first place. The only reason you'd ever want this class is when the
> fact that the flags are stored in an int is important.
>

> Sure, an IntFlags that isn't a subclass on int can define an __int__
> method. But so can some class that doesn't use an int for storage in the
> first place. If you want to convert back and forth, that's perfectly fine,
> but why are you using an int for storage?
>

I would word it this way: the only time you use IntFlags is when you need
access to an integer representation.  The implementation is free to do what
it likes provided it meets the interface guarantees.  You're not "using an
int for storage" and interfaces should never make implementation promises.
They should only make interface guarantees.


>
> If flags were conceptually subtypes of int, then you should be able to do
> things like:
>
> flags ** 7
>
> or
>
> flags // 91
>
> Do you agree that this is totally meaningless?  There is no "is a"
> relationship between Flags and int.  There is a conversion between the
> conceptual mapping that is a Flags object to int for the sole purpose of
> calling into APIs.
>
>
In my opinion, this is the most important point when deciding on
inheritance.


>
>
>> Good point.  To do something like that the name would have be BitFlags or
>> something no so solidly tied to "immutable".
>> At any rate, for this to work would require `int(x)` around every call to
>> a lower-level API, and that's a non-starter.
>>
>
> Correct me if I'm wrong, but doesn't Boost.Python when generating Python
> methods that accept ints, automatically call __int__ on the arguments.
> Doesn't SWIG do the same?  Can you give me some examples of methods that
> accept ints but don't call __int__?  It seems to me to be a bug in those
> methods than in the caller.  A method that wants an int should call __int__
> on its argument ? not expect that isinstance(X, int) etc.
>
>
>>
>> --
>> ~Ethan~
>>
>> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/d7f97b1b/attachment.html>

From random832 at fastmail.us  Mon Mar  9 14:49:35 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 09 Mar 2015 09:49:35 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us> <mdfn0v$o48$1@ger.gmane.org>
 <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
Message-ID: <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>

On Sun, Mar 8, 2015, at 05:45, Neil Girdhar wrote:
> If flags were conceptually subtypes of int, then you should be able to do
> things like:
> 
> flags ** 7
> 
> or
> 
> flags // 91

*cough* bool *cough*

> 
> Do you agree that this is totally meaningless?  There is no "is a"
> relationship between Flags and int.  There is a conversion between the
> conceptual mapping that is a Flags object to int for the sole purpose of
> calling into APIs.

From mistersheik at gmail.com  Mon Mar  9 15:00:20 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 10:00:20 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <mdfn0v$o48$1@ger.gmane.org> <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
 <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>
Message-ID: <CAA68w_n==8tFXP5SR85-afE8aLVWLC1NR9y_TjYw+9-AwX8BDw@mail.gmail.com>

People have been discussing bool since it was introduced.  Many people have
proposed that bool should not subclass int exactly for this reason (the
Liskov substitution principle), but unfortunately bool evolved from int in
Python 2.2.1 (I think?) when there was already a lot of code using int to
implement Boolean variables with constants like False, True = 0, 1.  In
order not to break too much code, it was decided to allow bool to inherit
from int.  The inheritance of bool from int is not a good precedent when
making inheritance decisions in the future.

On Mon, Mar 9, 2015 at 9:49 AM, <random832 at fastmail.us> wrote:

> On Sun, Mar 8, 2015, at 05:45, Neil Girdhar wrote:
> > If flags were conceptually subtypes of int, then you should be able to do
> > things like:
> >
> > flags ** 7
> >
> > or
> >
> > flags // 91
>
> *cough* bool *cough*
>
> >
> > Do you agree that this is totally meaningless?  There is no "is a"
> > relationship between Flags and int.  There is a conversion between the
> > conceptual mapping that is a Flags object to int for the sole purpose of
> > calling into APIs.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/11a5a4d5/attachment-0001.html>

From random832 at fastmail.us  Mon Mar  9 16:01:12 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 09 Mar 2015 11:01:12 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_n==8tFXP5SR85-afE8aLVWLC1NR9y_TjYw+9-AwX8BDw@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us> <mdfn0v$o48$1@ger.gmane.org>
 <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
 <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>
 <CAA68w_n==8tFXP5SR85-afE8aLVWLC1NR9y_TjYw+9-AwX8BDw@mail.gmail.com>
Message-ID: <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com>

On Mon, Mar 9, 2015, at 10:00, Neil Girdhar wrote:
> People have been discussing bool since it was introduced.  Many people
> have
> proposed that bool should not subclass int exactly for this reason (the
> Liskov substitution principle), but unfortunately bool evolved from int
> in
> Python 2.2.1 (I think?) when there was already a lot of code using int to
> implement Boolean variables with constants like False, True = 0, 1.  In
> order not to break too much code, it was decided to allow bool to inherit
> from int.  The inheritance of bool from int is not a good precedent when
> making inheritance decisions in the future.

And we're already using int in all these places people are talking about
dropping this new IntFlags class into. The situations seem exactly
analogous to me.

From mistersheik at gmail.com  Mon Mar  9 16:54:07 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Mon, 9 Mar 2015 11:54:07 -0400
Subject: [Python-ideas] IntFlags
In-Reply-To: <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us>
 <mdfn0v$o48$1@ger.gmane.org> <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
 <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>
 <CAA68w_n==8tFXP5SR85-afE8aLVWLC1NR9y_TjYw+9-AwX8BDw@mail.gmail.com>
 <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com>
Message-ID: <CAA68w_kmgUFBcenovJJFfK6u=gfQfs6JSKUHW2Qbm7nVK4GtQg@mail.gmail.com>

The differences are that the flags are used in a tiny portion of Python
code, and of that tiny amount of code, 99% of it probably uses nothing more
than the minimum "flags interface" that the composition solution has.  bool
was not like that.  Changing bool to be its own type would led to many more
problems.

On Mon, Mar 9, 2015 at 11:01 AM, <random832 at fastmail.us> wrote:

> On Mon, Mar 9, 2015, at 10:00, Neil Girdhar wrote:
> > People have been discussing bool since it was introduced.  Many people
> > have
> > proposed that bool should not subclass int exactly for this reason (the
> > Liskov substitution principle), but unfortunately bool evolved from int
> > in
> > Python 2.2.1 (I think?) when there was already a lot of code using int to
> > implement Boolean variables with constants like False, True = 0, 1.  In
> > order not to break too much code, it was decided to allow bool to inherit
> > from int.  The inheritance of bool from int is not a good precedent when
> > making inheritance decisions in the future.
>
> And we're already using int in all these places people are talking about
> dropping this new IntFlags class into. The situations seem exactly
> analogous to me.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/c1d88e14/attachment.html>

From rymg19 at gmail.com  Mon Mar  9 18:40:11 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 9 Mar 2015 12:40:11 -0500
Subject: [Python-ideas] CPython needs to revamp/replace it's build system &&
 Cross-compiling Python sucks
Message-ID: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>

I've been trying to work on Issue 23496. And want to tear my hair out. I've
noticed a few problems with Python's current build system:

- There is no separation of the host C compiler and the target C compiler.
When you give configure a C compiler for cross-compiling, make will happily
try to build Parser/pgen with the cross-compiler. And try to run the
generated executable. And fail. I know you can use `make touch` to get
around this, but it still sucks.

- For some reason, autoconf decided to re-configure everything about 2
minutes into make. But it didn't pass any of the flags I needed, resulting
in everything failing.

- I do not like having to use this for configuring:

   PYTHON_FOR_BUILD=`realpath ../cpython/python`
CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6
ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
ac_cv_have_long_long_format=yes

At the same time, I'm not quite sure how many people would want to sift
through the 5000 lines in configure.ac or the 1650 lines in Makefile.pre.in.

I just get the feeling that, whenever I build Python, there are too many
holes/hacks to get around make/autoconf things.

-- 
Ryan
If I were in a 10-story building glass-sided building and forced to write
either Go or autconf scripts, I?d jump out a window.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150309/c9614dee/attachment.html>

From russell at keith-magee.com  Tue Mar 10 01:57:54 2015
From: russell at keith-magee.com (Russell Keith-Magee)
Date: Tue, 10 Mar 2015 08:57:54 +0800
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
Message-ID: <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>

Hi Ryan,

First off I'm no fan of autoconf, but...

On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> I've been trying to work on Issue 23496. And want to tear my hair out.
> I've noticed a few problems with Python's current build system:
>
> - There is no separation of the host C compiler and the target C compiler.
> When you give configure a C compiler for cross-compiling, make will happily
> try to build Parser/pgen with the cross-compiler. And try to run the
> generated executable. And fail. I know you can use `make touch` to get
> around this, but it still sucks.
>

True there is no separation of host/target compilers, but are you sure this
matters? In my cross-compiling for iOS, the pgen steps are skipped --
Parser/pgen is compiled, but not ever used. Are you sure that this failed
execution isn't an artefact of identifying the Android build a "linux"?
I've had to make a bunch of small modifications to configure.ac to make the
iOS build process not build and run things that won't work on the build
platform.


> - For some reason, autoconf decided to re-configure everything about 2
> minutes into make. But it didn't pass any of the flags I needed, resulting
> in everything failing.
>

That's the step compiling libffi/ctypes. libffi is vendored into the Python
source, and the mid-build configure step is the configuration of that
vendored source code. You'll see the mid-build configure execute regardless
of whether you're cross-compiling or not -- unless you're for OS/X, because
the build instructions for OS/X libffi are hard-coded into setup.py.


> - I do not like having to use this for configuring:
>
>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
> ./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6
> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
> ac_cv_have_long_long_format=yes
>
>
The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
compiling Python 3.X, as long as the build platform has a python3.X
executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
appropriate python in your path.

ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually specified
flags because they can't be auto detected on a cross-compile (at least, not
in any way I can think of). They're checking whether the *target* device
has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could probably
hard-code the responses on the basis that the target platform is a known
quantity, but in the general case (which is the current case), you can't
auto detect the properties of the target platform.

I can't comment on ac_cv_have_long_long_format, but I'm guessing the cause
is the same - you can't compile and run the configure test program.

--disable-ipv6 is a feature flag - I don't know why it's necessary on
Android.

At the same time, I'm not quite sure how many people would want to sift
> through the 5000 lines in configure.ac or the 1650 lines in
> Makefile.pre.in.
>
> I just get the feeling that, whenever I build Python, there are too many
> holes/hacks to get around make/autoconf things.
>

Sure - configure sucks. What's the alternative?

Sometimes things are complex because... they're complex. This is a build
system that will support building everything from AiX to Android, and in
*most* cases, with little more than a ./configure && make && make install.
Android (and other cross compiling cases) are a little more complex because
you have to *tell* configure what the target looks like, because it can't
detect anything about the platform.

Yours,
Russ Magee %-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/b08c26c5/attachment-0001.html>

From rymg19 at gmail.com  Tue Mar 10 03:43:46 2015
From: rymg19 at gmail.com (Ryan)
Date: Mon, 09 Mar 2015 21:43:46 -0500
Subject: [Python-ideas] CPython needs to revamp/replace it's build
	system && Cross-compiling Python sucks
In-Reply-To: <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
Message-ID: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>



Russell Keith-Magee <russell at keith-magee.com> wrote:
>Hi Ryan,
>
>First off I'm no fan of autoconf, but...
>

Who is other than the one who designed it? :)

>On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com>
>wrote:
>
>> I've been trying to work on Issue 23496. And want to tear my hair
>out.
>> I've noticed a few problems with Python's current build system:
>>
>> - There is no separation of the host C compiler and the target C
>compiler.
>> When you give configure a C compiler for cross-compiling, make will
>happily
>> try to build Parser/pgen with the cross-compiler. And try to run the
>> generated executable. And fail. I know you can use `make touch` to
>get
>> around this, but it still sucks.
>>
>
>True there is no separation of host/target compilers, but are you sure
>this
>matters? In my cross-compiling for iOS, the pgen steps are skipped --
>Parser/pgen is compiled, but not ever used. Are you sure that this
>failed
>execution isn't an artefact of identifying the Android build a "linux"?
>I've had to make a bunch of small modifications to configure.ac to make
>the
>iOS build process not build and run things that won't work on the build
>platform.
>

Hmmm...

But you gave an example. You shouldn't need to edit configure.ac to not try to run what can't be run because the host != the builder.

>
>> - For some reason, autoconf decided to re-configure everything about
>2
>> minutes into make. But it didn't pass any of the flags I needed,
>resulting
>> in everything failing.
>>
>
>That's the step compiling libffi/ctypes. libffi is vendored into the
>Python
>source, and the mid-build configure step is the configuration of that
>vendored source code. You'll see the mid-build configure execute
>regardless
>of whether you're cross-compiling or not -- unless you're for OS/X,
>because
>the build instructions for OS/X libffi are hard-coded into setup.py.
>

Still sucks.

>
>> - I do not like having to use this for configuring:
>>
>>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
>> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
>> ./configure --host=arm-linux-androideabi --build=x86_64
>--disable-ipv6
>> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
>> ac_cv_have_long_long_format=yes
>>
>>
>The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
>compiling Python 3.X, as long as the build platform has a python3.X
>executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
>necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
>appropriate python in your path.

The build was giving errors because apparently 3.4's _sre module doesn't have some constant that the build script needs (MAXGROUPS).

>
>ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually
>specified
>flags because they can't be auto detected on a cross-compile (at least,
>not
>in any way I can think of). They're checking whether the *target*
>device
>has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could
>probably
>hard-code the responses on the basis that the target platform is a
>known
>quantity, but in the general case (which is the current case), you
>can't
>auto detect the properties of the target platform.

It took me a while to figure out that that was the option I needed to pass, though.

>
>I can't comment on ac_cv_have_long_long_format, but I'm guessing the
>cause
>is the same - you can't compile and run the configure test program.
>
>--disable-ipv6 is a feature flag - I don't know why it's necessary on
>Android.

I don't really know why, either. Something to do with a getaddrline bug.

>
>At the same time, I'm not quite sure how many people would want to sift
>> through the 5000 lines in configure.ac or the 1650 lines in
>> Makefile.pre.in.
>>
>> I just get the feeling that, whenever I build Python, there are too
>many
>> holes/hacks to get around make/autoconf things.
>>
>
>Sure - configure sucks. What's the alternative?

All 1000 other build systems out there?

>
>Sometimes things are complex because... they're complex. This is a
>build
>system that will support building everything from AiX to Android, and
>in
>*most* cases, with little more than a ./configure && make && make
>install.
>Android (and other cross compiling cases) are a little more complex
>because
>you have to *tell* configure what the target looks like, because it
>can't
>detect anything about the platform.

But autotools tends to make things more complex than needed. When Qt (or was it KDE?) moved to CMake from autotools, the build script size dropped a large amount.

Just take a look at the code in configure.ac that manually detects the compiler on OSX systems because autoconf SOMEHOW manages to get it wrong.

>
>Yours,
>Russ Magee %-)

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Check out my website: http://kirbyfan64.github.io/

From russell at keith-magee.com  Tue Mar 10 12:20:42 2015
From: russell at keith-magee.com (Russell Keith-Magee)
Date: Tue, 10 Mar 2015 19:20:42 +0800
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
Message-ID: <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>

On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:

>
>
> Russell Keith-Magee <russell at keith-magee.com> wrote:
> >Hi Ryan,
> >
> >First off I'm no fan of autoconf, but...
> >
>
> Who is other than the one who designed it? :)
>

Let me be clear - there's a difference between "not a fan" and "calling for
it's head". Autoconf is crufty, annoying, and sometimes aggravating - but
it *works*, and has had some pretty seriously battle testing - both in the
abstract "autoconf as a product", and in the specific "autoconf as used in
Python". This isn't something that can be said of most of the alternatives
in the abstract case, or *any* of the alternatives in the Python case.

>On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com>
> >wrote:
> >
> >> I've been trying to work on Issue 23496. And want to tear my hair
> >out.
> >> I've noticed a few problems with Python's current build system:
> >>
> >> - There is no separation of the host C compiler and the target C
> >compiler.
> >> When you give configure a C compiler for cross-compiling, make will
> >happily
> >> try to build Parser/pgen with the cross-compiler. And try to run the
> >> generated executable. And fail. I know you can use `make touch` to
> >get
> >> around this, but it still sucks.
> >>
> >
> >True there is no separation of host/target compilers, but are you sure
> >this
> >matters? In my cross-compiling for iOS, the pgen steps are skipped --
> >Parser/pgen is compiled, but not ever used. Are you sure that this
> >failed
> >execution isn't an artefact of identifying the Android build a "linux"?
> >I've had to make a bunch of small modifications to configure.ac to make
> >the
> >iOS build process not build and run things that won't work on the build
> >platform.
> >
>
> Hmmm...
>
> But you gave an example. You shouldn't need to edit configure.ac to not
> try to run what can't be run because the host != the builder.
>

Agreed. My claim is that the behaviour you're seeing is a bug in the
existing configure script, probably born of the fact that using the "linux"
system type for Android confuses the build system into thinking that the
host executable will run on the build machine.

I suspect that the fix will be quite simple, and will reveal itself almost
immediately if you build with --host=armv7-google-android instead of
--host=armv7-linux-gnu or --host=armv7-linux-android

The existence of bugs isn't a reason to replace the entire build system,
unless you've got good reason to believe that a new build system won't be
subject to the same class of bugs.

>> - For some reason, autoconf decided to re-configure everything about
> >2
> >> minutes into make. But it didn't pass any of the flags I needed,
> >resulting
> >> in everything failing.
> >>
> >
> >That's the step compiling libffi/ctypes. libffi is vendored into the
> >Python
> >source, and the mid-build configure step is the configuration of that
> >vendored source code. You'll see the mid-build configure execute
> >regardless
> >of whether you're cross-compiling or not -- unless you're for OS/X,
> >because
> >the build instructions for OS/X libffi are hard-coded into setup.py.
> >
>
> Still sucks.
>

I don't see why. libffi is a complex piece of machinery. Every platform/CPU
architecture needs a different set of files to be compiled and included.
libffi's sources includes those definitions.

The alternative is hard-coding every possible target platform. Using
configure (the same configure script that libffi ships, I might add) means
this isn't required.

I'd also add that the other alternative is to configure using
--with-system-ffi, and ensure there is an appropriate libffi.so in your
path, etc. No internal configure step is required in this case.

>> - I do not like having to use this for configuring:
> >>
> >>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
> >> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
> >> ./configure --host=arm-linux-androideabi --build=x86_64
> >--disable-ipv6
> >> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
> >> ac_cv_have_long_long_format=yes
> >>
> >>
> >The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
> >compiling Python 3.X, as long as the build platform has a python3.X
> >executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
> >necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
> >appropriate python in your path.
>
> The build was giving errors because apparently 3.4's _sre module doesn't
> have some constant that the build script needs (MAXGROUPS).
>
> >
> >ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually
> >specified
> >flags because they can't be auto detected on a cross-compile (at least,
> >not
> >in any way I can think of). They're checking whether the *target*
> >device
> >has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could
> >probably
> >hard-code the responses on the basis that the target platform is a
> >known
> >quantity, but in the general case (which is the current case), you
> >can't
> >auto detect the properties of the target platform.
>
> It took me a while to figure out that that was the option I needed to
> pass, though.
>

Granted it isn't obvious. However, I'd suggest the logical extension of
that argument is "Lets document the existing build system", not "Lets
replace the build system with something new".

>I can't comment on ac_cv_have_long_long_format, but I'm guessing the
> >cause
> >is the same - you can't compile and run the configure test program.
> >
> >--disable-ipv6 is a feature flag - I don't know why it's necessary on
> >Android.
>
> I don't really know why, either. Something to do with a getaddrline bug.
>

Sure. And if it turns out that there's a good systemic reason why ipv6
won't ever work on Android, it's a couple of line patch to configure.ac
(around line 2666) to fix the problem permanently, removing the need for
the manual flag to be specified.

>At the same time, I'm not quite sure how many people would want to sift
> >> through the 5000 lines in configure.ac or the 1650 lines in
> >> Makefile.pre.in.
> >>
> >> I just get the feeling that, whenever I build Python, there are too
> >many
> >> holes/hacks to get around make/autoconf things.
> >>
> >
> >Sure - configure sucks. What's the alternative?
>
> All 1000 other build systems out there?
>

Care to state a specific preference, rather than engage in hyperbole?

>Sometimes things are complex because... they're complex. This is a
> >build
> >system that will support building everything from AiX to Android, and
> >in
> >*most* cases, with little more than a ./configure && make && make
> >install.
> >Android (and other cross compiling cases) are a little more complex
> >because
> >you have to *tell* configure what the target looks like, because it
> >can't
> >detect anything about the platform.
>
> But autotools tends to make things more complex than needed. When Qt (or
> was it KDE?) moved to CMake from autotools, the build script size dropped a
> large amount.
>
> Just take a look at the code in configure.ac that manually detects the
> compiler on OSX systems because autoconf SOMEHOW manages to get it wrong.
>

<sarcasm>
Wow - who would have thought that a complex set of build requirements that
have to work on a wide range of platforms and has been developed over 20+
years would occasionally yield edge cases?
</sarcasm>

Seriously, if your claim is that moving to a new build system will remove
the existence of edge cases, I want some of what you are smoking.

To address the specific example you gave: OSX is a particularly complex
platform to support because it spans 2 completely different CPU
architectures, has the concept of Framework builds that other platforms
don't have, and has, at various time, used clang, llvm-gcc and gcc as the
default compiler. Net result: AC_PROG_CC autoconf tool doesn't handle this
complexity well. But then, most build systems don't. However, Autoconf is
rich enough that it is possible to work around this problem.

I can't speak for the core team, but from my perspective, this sounds like
a proposal where the only argument that is going to work is a working (or
mostly working) port of the existing build system that proves it is
possible to support all the existing edge cases in a much simpler fashion.

For my money - of all the problems I've hit trying to port Python to iOS,
autoconf is the least of them. No, it isn't my favourite tool - but for all
it's faults, it works. If you want to convince me that there is a better
alternative, you're going to need to do more than just repeating that
"autoconf sucks".

Yours,
Russ Magee %-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/b4c36536/attachment.html>

From ncoghlan at gmail.com  Tue Mar 10 13:51:30 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Mar 2015 22:51:30 +1000
Subject: [Python-ideas] IntFlags
In-Reply-To: <CAA68w_kmgUFBcenovJJFfK6u=gfQfs6JSKUHW2Qbm7nVK4GtQg@mail.gmail.com>
References: <md4ldf$1a9$1@ger.gmane.org> <54FAB1CA.3000105@stoneleaf.us>
 <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com>
 <54FB13A1.5020008@stoneleaf.us> <mdfn0v$o48$1@ger.gmane.org>
 <54FB61BC.8090107@stoneleaf.us>
 <CAA68w_m-YBhYqAAgsHQLnX-mJc2JLgh6v8vNX6UMQG25PF=ZzQ@mail.gmail.com>
 <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com>
 <CAA68w_n==8tFXP5SR85-afE8aLVWLC1NR9y_TjYw+9-AwX8BDw@mail.gmail.com>
 <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com>
 <CAA68w_kmgUFBcenovJJFfK6u=gfQfs6JSKUHW2Qbm7nVK4GtQg@mail.gmail.com>
Message-ID: <CADiSq7f4pyH==AMbWnTgPkcDb96dpi5EpsfxtLN2tj1DGkuzLA@mail.gmail.com>

On 10 March 2015 at 01:54, Neil Girdhar <mistersheik at gmail.com> wrote:
> The differences are that the flags are used in a tiny portion of Python
> code, and of that tiny amount of code, 99% of it probably uses nothing more
> than the minimum "flags interface" that the composition solution has.  bool
> was not like that.  Changing bool to be its own type would led to many more
> problems.

The flag values in modules like "stat" and "socket" are not niche use
cases where we can be cavalier with backwards compatibility concerns,
as they get passed to operating system level APIs both inside and
outside the standard library. Most application level code will never
need to touch those modules, but system boundary code does it all the
time. It's the exact same rationale as was used for adopting IntEnum
in the socket and errno modules rather than the base Enum class.

There's likely no need to get especially creative with the overall
design here, as the 3.4 enum module should provide a good general
architecture to follow, it's just the runtime behaviour that will be
changing to be "ordered collection of combinatorial flags" rather than
"ordered collection of named values".

The main question in my mind is whether you might want named bitmask
support as well, but that's probably not worth the extra complexity,
since you don't need it for debugging purposes.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Mar 10 14:43:14 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Mar 2015 23:43:14 +1000
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
Message-ID: <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>

On 10 March 2015 at 21:20, Russell Keith-Magee <russell at keith-magee.com> wrote:
>
> On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
>>
>> Russell Keith-Magee <russell at keith-magee.com> wrote:
>> >Hi Ryan,
>> >
>> >First off I'm no fan of autoconf, but...
>> >
>>
>> Who is other than the one who designed it? :)
>
> Let me be clear - there's a difference between "not a fan" and "calling for
> it's head". Autoconf is crufty, annoying, and sometimes aggravating - but it
> *works*, and has had some pretty seriously battle testing - both in the
> abstract "autoconf as a product", and in the specific "autoconf as used in
> Python". This isn't something that can be said of most of the alternatives
> in the abstract case, or *any* of the alternatives in the Python case.

Building architecture dependent software is a genuinely hard problem,
which is why it's full of "80%" solutions like autotools and the
setuptools/distutils pairing. Yes, they're quirky and annoying, but
they also basically do their job, so folks tend to develop a high
tolerance level for their quirks.

In our particular case, we also have the fun problem of "How do you
build CPython locally when you don't have a working Python interpreter
yet?". A number of the more interesting alternative C/C++ build
systems (like Meson, Scons, Waf and the Google "gyp" project that
powers the node-gyp native extension build system for Node.js) require
an already working Python interpreter to get things started (when
starting from a completely blank slate, even our own build sequence
has to build a deliberately broken partial interpreter just to freeze
importlib._bootstrap before going on to build a properly configured
one that can actually import modules).

CMake seems like a plausible alternative that would avoid the
bootstrapping problem (as even building it from source only requires a
working C++ compiler for the platform, either native or
cross-compiled): http://www.cmake.org/

KDE switched over to CMake from autotools ~9 years ago
(http://lwn.net/Articles/188693/) and plenty of other projects have
either followed suit or simply never used autotools in the first
place: http://www.cmake.org/Wiki/CMake_Projects

It's far too late in the 3.5 dev cycle to contemplate a change like
that, but if anyone was looking for a potentially interesting side
project, seeing exactly what was involved in getting CPython to build
with CMake instead of autotools could be a worthwhile investigation:
https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rymg19 at gmail.com  Tue Mar 10 16:39:50 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 10 Mar 2015 10:39:50 -0500
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
Message-ID: <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>

On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 10 March 2015 at 21:20, Russell Keith-Magee <russell at keith-magee.com>
> wrote:
> >
> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
> >>
> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
> >> >Hi Ryan,
> >> >
> >> >First off I'm no fan of autoconf, but...
> >> >
> >>
> >> Who is other than the one who designed it? :)
> >
> > Let me be clear - there's a difference between "not a fan" and "calling
> for
> > it's head". Autoconf is crufty, annoying, and sometimes aggravating -
> but it
> > *works*, and has had some pretty seriously battle testing - both in the
> > abstract "autoconf as a product", and in the specific "autoconf as used
> in
> > Python". This isn't something that can be said of most of the
> alternatives
> > in the abstract case, or *any* of the alternatives in the Python case.
>
> Building architecture dependent software is a genuinely hard problem,
> which is why it's full of "80%" solutions like autotools and the
> setuptools/distutils pairing. Yes, they're quirky and annoying, but
> they also basically do their job, so folks tend to develop a high
> tolerance level for their quirks.
>
> In our particular case, we also have the fun problem of "How do you
> build CPython locally when you don't have a working Python interpreter
> yet?". A number of the more interesting alternative C/C++ build
> systems (like Meson, Scons, Waf and the Google "gyp" project that
> powers the node-gyp native extension build system for Node.js) require
> an already working Python interpreter to get things started (when
> starting from a completely blank slate, even our own build sequence
> has to build a deliberately broken partial interpreter just to freeze
> importlib._bootstrap before going on to build a properly configured
> one that can actually import modules).
>

What about something like Google's new GN? It's designed to replace Gyp in
building Chromium and is written in C++.

I'm pretty sure using a build system written in Lua to build Python is kind
of weird.

With all that in mind...what exactly would be requirements that Python
would need in an autotools replacement?


>
> CMake seems like a plausible alternative that would avoid the
> bootstrapping problem (as even building it from source only requires a
> working C++ compiler for the platform, either native or
> cross-compiled): http://www.cmake.org/
>
> KDE switched over to CMake from autotools ~9 years ago
> (http://lwn.net/Articles/188693/) and plenty of other projects have
> either followed suit or simply never used autotools in the first
> place: http://www.cmake.org/Wiki/CMake_Projects
>
>
That was what I was referring to before.


> It's far too late in the 3.5 dev cycle to contemplate a change like
> that, but if anyone was looking for a potentially interesting side
> project, seeing exactly what was involved in getting CPython to build
> with CMake instead of autotools could be a worthwhile investigation:
>
> https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake
>
>
Someone is maintaining a set of CMake build files for Python 2.7 somewhere.


> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>



-- 
Ryan
If I were in a 10-story building glass-sided building and forced to write
either Go or autotools scripts, I?d jump out a window.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/0a04bf57/attachment.html>

From dmalinovsky at thumbtack.net  Tue Mar 10 16:43:17 2015
From: dmalinovsky at thumbtack.net (Dmitry Malinovsky)
Date: Tue, 10 Mar 2015 21:43:17 +0600
Subject: [Python-ideas] "Regular" namespace packages
Message-ID: <CAOAz==LhiqCgHQUzFH=eNbt5GXzKO63Exmrc8goqqWyMJNKFag@mail.gmail.com>

Hello,

I'm implementing an extensible library. Along with the library itself I
want to implement various extensions (and some of these extensions will be
also pluggable).

I found that creating a namespace package will be user-friendly (to
distinguish default implementations from third-party ones).
However, current namespace packages implementation does not allow me to
have a dedicated regular package with the same name as the namespace.
E.g. if I have to namespaced packages: top.packageA and top.packageB, I
can't have dedicated top package.
(Actually, I can do that: py module is the example, but this way is not
easy to implement for the library developers and to understand
by external contributors).

So my idea is to have a standard (and easy) way to declare a package which
can act both as a namespace package and a regular package,
like pep-420 (implicit namespace packages) standardized way to create
namespace packages.

Here is an example:

first/first/__init__.py:
X = 1

second/first/second/__init__.py:
Y = 2

import sys
sys.path += ['first', 'second']

import first
first.X  # 1
import first.second  # I expect to get 2,
first.second.Y         # but currently ImportError is raised

If I understand this correctly, this change will require to modify the
default submodule lookup algorithm,
so when the import machinery finds the cached first module, and there is no
second submodule defined,
the machinery launches the standard namespace packages finder algorithm (as
described in [1]),
excluding first package directory from the search path.

>From a user perspective, assuming we have a mature library with lots of
extensions (for example, flask),
and a pluggable extension (like flask-flatpages), the following imports

from flask import Flask
from flask_flatpages import FlatPages
from flask_flatpages_knitr import FlatPagesKnitr

can be rewritten as follows:

from flask import Flask
from flask.flatpages import FlatPages
from flask.flatpages.knitr import FlatPagesKnitr

which I find more readable than the first snippet.

What do you think about it? Is flat better than nested, or namespaces are
really great?

Thank you for any feedback!

[1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation

-- 
Dmitry Malinovsky
Senior DevOps
Thumbtack Technology
Email: dmalinovsky at thumbtack.net <pnegoduyko at thumbtack.net>
Skype: d_malinoff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/89aac046/attachment-0001.html>

From guido at python.org  Tue Mar 10 17:14:59 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Mar 2015 09:14:59 -0700
Subject: [Python-ideas] "Regular" namespace packages
In-Reply-To: <CAOAz==LhiqCgHQUzFH=eNbt5GXzKO63Exmrc8goqqWyMJNKFag@mail.gmail.com>
References: <CAOAz==LhiqCgHQUzFH=eNbt5GXzKO63Exmrc8goqqWyMJNKFag@mail.gmail.com>
Message-ID: <CAP7+vJJ1mhkVRJO4kBhGBsUHOqA-jxouR1WqdbNHRgUN5W3xtA@mail.gmail.com>

Your example code can be corrected by updating first.__path__:

first.__path__.append('second/first')

Now "import first.second" will succeed.

On Tue, Mar 10, 2015 at 8:43 AM, Dmitry Malinovsky <
dmalinovsky at thumbtack.net> wrote:

> Hello,
>
> I'm implementing an extensible library. Along with the library itself I
> want to implement various extensions (and some of these extensions will be
> also pluggable).
>
> I found that creating a namespace package will be user-friendly (to
> distinguish default implementations from third-party ones).
> However, current namespace packages implementation does not allow me to
> have a dedicated regular package with the same name as the namespace.
> E.g. if I have to namespaced packages: top.packageA and top.packageB, I
> can't have dedicated top package.
> (Actually, I can do that: py module is the example, but this way is not
> easy to implement for the library developers and to understand
> by external contributors).
>
> So my idea is to have a standard (and easy) way to declare a package which
> can act both as a namespace package and a regular package,
> like pep-420 (implicit namespace packages) standardized way to create
> namespace packages.
>
> Here is an example:
>
> first/first/__init__.py:
> X = 1
>
> second/first/second/__init__.py:
> Y = 2
>
> import sys
> sys.path += ['first', 'second']
>
> import first
> first.X  # 1
> import first.second  # I expect to get 2,
> first.second.Y         # but currently ImportError is raised
>
> If I understand this correctly, this change will require to modify the
> default submodule lookup algorithm,
> so when the import machinery finds the cached first module, and there is
> no second submodule defined,
> the machinery launches the standard namespace packages finder algorithm
> (as described in [1]),
> excluding first package directory from the search path.
>
> From a user perspective, assuming we have a mature library with lots of
> extensions (for example, flask),
> and a pluggable extension (like flask-flatpages), the following imports
>
> from flask import Flask
> from flask_flatpages import FlatPages
> from flask_flatpages_knitr import FlatPagesKnitr
>
> can be rewritten as follows:
>
> from flask import Flask
> from flask.flatpages import FlatPages
> from flask.flatpages.knitr import FlatPagesKnitr
>
> which I find more readable than the first snippet.
>
> What do you think about it? Is flat better than nested, or namespaces are
> really great?
>
> Thank you for any feedback!
>
> [1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation
>
> --
> Dmitry Malinovsky
> Senior DevOps
> Thumbtack Technology
> Email: dmalinovsky at thumbtack.net <pnegoduyko at thumbtack.net>
> Skype: d_malinoff
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/d22de36d/attachment.html>

From dmalinovsky at thumbtack.net  Tue Mar 10 18:20:38 2015
From: dmalinovsky at thumbtack.net (Dmitry Malinovsky)
Date: Tue, 10 Mar 2015 23:20:38 +0600
Subject: [Python-ideas] "Regular" namespace packages
In-Reply-To: <CAP7+vJJ1mhkVRJO4kBhGBsUHOqA-jxouR1WqdbNHRgUN5W3xtA@mail.gmail.com>
References: <CAOAz==LhiqCgHQUzFH=eNbt5GXzKO63Exmrc8goqqWyMJNKFag@mail.gmail.com>
 <CAP7+vJJ1mhkVRJO4kBhGBsUHOqA-jxouR1WqdbNHRgUN5W3xtA@mail.gmail.com>
Message-ID: <CAOAz==JL-qcAJUKszDBjvVPL0e5DiMxExVR+mSqHMBvBFaRdag@mail.gmail.com>

Thank you, Guido, however the whole idea is to avoid modifying "first"
package, so the "first" package author can freely leave his "first"
namespace for extensions developers without even knowing that such
extensions exist.

On Tue, Mar 10, 2015 at 10:14 PM, Guido van Rossum <guido at python.org> wrote:

> Your example code can be corrected by updating first.__path__:
>
> first.__path__.append('second/first')
>
> Now "import first.second" will succeed.
>
> On Tue, Mar 10, 2015 at 8:43 AM, Dmitry Malinovsky <
> dmalinovsky at thumbtack.net> wrote:
>
>> Hello,
>>
>> I'm implementing an extensible library. Along with the library itself I
>> want to implement various extensions (and some of these extensions will be
>> also pluggable).
>>
>> I found that creating a namespace package will be user-friendly (to
>> distinguish default implementations from third-party ones).
>> However, current namespace packages implementation does not allow me to
>> have a dedicated regular package with the same name as the namespace.
>> E.g. if I have to namespaced packages: top.packageA and top.packageB, I
>> can't have dedicated top package.
>> (Actually, I can do that: py module is the example, but this way is not
>> easy to implement for the library developers and to understand
>> by external contributors).
>>
>> So my idea is to have a standard (and easy) way to declare a package
>> which can act both as a namespace package and a regular package,
>> like pep-420 (implicit namespace packages) standardized way to create
>> namespace packages.
>>
>> Here is an example:
>>
>> first/first/__init__.py:
>> X = 1
>>
>> second/first/second/__init__.py:
>> Y = 2
>>
>> import sys
>> sys.path += ['first', 'second']
>>
>> import first
>> first.X  # 1
>> import first.second  # I expect to get 2,
>> first.second.Y         # but currently ImportError is raised
>>
>> If I understand this correctly, this change will require to modify the
>> default submodule lookup algorithm,
>> so when the import machinery finds the cached first module, and there is
>> no second submodule defined,
>> the machinery launches the standard namespace packages finder algorithm
>> (as described in [1]),
>> excluding first package directory from the search path.
>>
>> From a user perspective, assuming we have a mature library with lots of
>> extensions (for example, flask),
>> and a pluggable extension (like flask-flatpages), the following imports
>>
>> from flask import Flask
>> from flask_flatpages import FlatPages
>> from flask_flatpages_knitr import FlatPagesKnitr
>>
>> can be rewritten as follows:
>>
>> from flask import Flask
>> from flask.flatpages import FlatPages
>> from flask.flatpages.knitr import FlatPagesKnitr
>>
>> which I find more readable than the first snippet.
>>
>> What do you think about it? Is flat better than nested, or namespaces are
>> really great?
>>
>> Thank you for any feedback!
>>
>> [1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation
>>
>> --
>> Dmitry Malinovsky
>> Senior DevOps
>> Thumbtack Technology
>> Email: dmalinovsky at thumbtack.net <pnegoduyko at thumbtack.net>
>> Skype: d_malinoff
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
Dmitry Malinovsky
Support Engineer
Thumbtack Technology
Email: dmalinovsky at thumbtack.net <pnegoduyko at thumbtack.net>
Skype: d_malinoff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/7d7d631a/attachment.html>

From eltoder at gmail.com  Tue Mar 10 19:00:23 2015
From: eltoder at gmail.com (Eugene Toder)
Date: Tue, 10 Mar 2015 14:00:23 -0400
Subject: [Python-ideas] "Regular" namespace packages
In-Reply-To: <CAOAz==JL-qcAJUKszDBjvVPL0e5DiMxExVR+mSqHMBvBFaRdag@mail.gmail.com>
References: <CAOAz==LhiqCgHQUzFH=eNbt5GXzKO63Exmrc8goqqWyMJNKFag@mail.gmail.com>
 <CAP7+vJJ1mhkVRJO4kBhGBsUHOqA-jxouR1WqdbNHRgUN5W3xtA@mail.gmail.com>
 <CAOAz==JL-qcAJUKszDBjvVPL0e5DiMxExVR+mSqHMBvBFaRdag@mail.gmail.com>
Message-ID: <CA+KNMz=n+kx+baXetEN1x1qkvbH8PGvNCpXmUF-X7u_o=ygOAQ@mail.gmail.com>

On Tue, Mar 10, 2015 at 1:20 PM, Dmitry Malinovsky <
dmalinovsky at thumbtack.net> wrote:

> Thank you, Guido, however the whole idea is to avoid modifying "first"
> package, so the "first" package author can freely leave his "first"
> namespace for extensions developers without even knowing that such
> extensions exist.
>
The simplest solution is probably to make the installation process of
package "second" to install itself in the directory of "first", or make a
symlink, e.g.:

first/first/__init__.py
first/first/second --> second/first/second # added by package second
second/first/second/__init__.py

Alternatively, you can add "second" to the package __path__ as Guido
pointed above via .pth file[1], to avoid changing anything in the "first"
package.

[1] https://docs.python.org/3.4/library/site.html, especially the part
about lines starting with "import"

Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/86a45480/attachment-0001.html>

From julien at tayon.net  Wed Mar 11 13:28:49 2015
From: julien at tayon.net (julien tayon)
Date: Wed, 11 Mar 2015 08:28:49 -0400
Subject: [Python-ideas] Implementation of adding dicts
Message-ID: <CAFpLVkxGy=Km5T6B1Wdf=-BXcYpcFUa0F9=C_SgV5YgQvJF4qw@mail.gmail.com>

Hello,

Just in case I have to implementation of dict/sub/mult/div for dict.

I have chosen only one rule: + - / * should behave with the least surprise
principle which is so far linear algebrae when dealing with stuff that
looks like mathematics

Implementation 2 (trying trait and mixin)
http://archery.readthedocs.org/en/latest/bow.html
Implementation 1 (experimenting weired stuff with dict (like addition and
transforming dict with dict in a ?matrix like? fashion)
http://vectordict.readthedocs.org/en/latest/algebrae.html

And the linear algebrae test:
http://vectordict.readthedocs.org/en/latest/consistent.html
And the results
https://github.com/jul/ADictAdd_iction/blob/master/docs/source/result.txt

I just noticed the dicussion, I have been using the + - / * for
aggregations for 2 years now and it is really helpful. I can give any
feedback.

I also had to overload + - for Sets ^^ and I have the cosine similarities
that works nicely out of the box with my definition
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/ce497787/attachment.html>

From brett at python.org  Wed Mar 11 16:07:19 2015
From: brett at python.org (Brett Cannon)
Date: Wed, 11 Mar 2015 15:07:19 +0000
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
Message-ID: <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>

On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez <rymg19 at gmail.com> wrote:

> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 10 March 2015 at 21:20, Russell Keith-Magee <russell at keith-magee.com>
>> wrote:
>> >
>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
>> >>
>> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
>> >> >Hi Ryan,
>> >> >
>> >> >First off I'm no fan of autoconf, but...
>> >> >
>> >>
>> >> Who is other than the one who designed it? :)
>> >
>> > Let me be clear - there's a difference between "not a fan" and "calling
>> for
>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating -
>> but it
>> > *works*, and has had some pretty seriously battle testing - both in the
>> > abstract "autoconf as a product", and in the specific "autoconf as used
>> in
>> > Python". This isn't something that can be said of most of the
>> alternatives
>> > in the abstract case, or *any* of the alternatives in the Python case.
>>
>> Building architecture dependent software is a genuinely hard problem,
>> which is why it's full of "80%" solutions like autotools and the
>> setuptools/distutils pairing. Yes, they're quirky and annoying, but
>> they also basically do their job, so folks tend to develop a high
>> tolerance level for their quirks.
>>
>> In our particular case, we also have the fun problem of "How do you
>> build CPython locally when you don't have a working Python interpreter
>> yet?". A number of the more interesting alternative C/C++ build
>> systems (like Meson, Scons, Waf and the Google "gyp" project that
>> powers the node-gyp native extension build system for Node.js) require
>> an already working Python interpreter to get things started (when
>> starting from a completely blank slate, even our own build sequence
>> has to build a deliberately broken partial interpreter just to freeze
>> importlib._bootstrap before going on to build a properly configured
>> one that can actually import modules).
>>
>
> What about something like Google's new GN? It's designed to replace Gyp in
> building Chromium and is written in C++.
>
> I'm pretty sure using a build system written in Lua to build Python is
> kind of weird.
>


>
> With all that in mind...what exactly would be requirements that Python
> would need in an autotools replacement?
>

Can't require anything more than an ANSI C compiler basically. People run
Python in places where they don't have much more than gcc.


>
>
>>
>> CMake seems like a plausible alternative that would avoid the
>> bootstrapping problem (as even building it from source only requires a
>> working C++ compiler for the platform, either native or
>> cross-compiled): http://www.cmake.org/
>>
>> KDE switched over to CMake from autotools ~9 years ago
>> (http://lwn.net/Articles/188693/) and plenty of other projects have
>> either followed suit or simply never used autotools in the first
>> place: http://www.cmake.org/Wiki/CMake_Projects
>>
>>
> That was what I was referring to before.
>

I casually discussed the idea back at PyCon 2008 about using CMake and was
told "no" because of potential build tool dependencies for CMake itself. So
you have to make sure that you can prove CMake is highly portable.


>
>
>> It's far too late in the 3.5 dev cycle to contemplate a change like
>> that, but if anyone was looking for a potentially interesting side
>> project, seeing exactly what was involved in getting CPython to build
>> with CMake instead of autotools could be a worthwhile investigation:
>>
>> https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake
>>
>>
> Someone is maintaining a set of CMake build files for Python 2.7 somewhere.
>

That's fine, but obviously things have been tweaked since then and this
could only go into 3.x.

-Brett


>
>
>> Regards,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>
>
>
>
> --
> Ryan
> If I were in a 10-story building glass-sided building and forced to write
> either Go or autotools scripts, I?d jump out a window.
> http://kirbyfan64.github.io/
>
>  _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/69737fe0/attachment.html>

From rymg19 at gmail.com  Wed Mar 11 19:30:12 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 11 Mar 2015 13:30:12 -0500
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
 <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
Message-ID: <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>

On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon <brett at python.org> wrote:

>
>
> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 10 March 2015 at 21:20, Russell Keith-Magee <russell at keith-magee.com>
>>> wrote:
>>> >
>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
>>> >>
>>> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
>>> >> >Hi Ryan,
>>> >> >
>>> >> >First off I'm no fan of autoconf, but...
>>> >> >
>>> >>
>>> >> Who is other than the one who designed it? :)
>>> >
>>> > Let me be clear - there's a difference between "not a fan" and
>>> "calling for
>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating -
>>> but it
>>> > *works*, and has had some pretty seriously battle testing - both in the
>>> > abstract "autoconf as a product", and in the specific "autoconf as
>>> used in
>>> > Python". This isn't something that can be said of most of the
>>> alternatives
>>> > in the abstract case, or *any* of the alternatives in the Python case.
>>>
>>> Building architecture dependent software is a genuinely hard problem,
>>> which is why it's full of "80%" solutions like autotools and the
>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but
>>> they also basically do their job, so folks tend to develop a high
>>> tolerance level for their quirks.
>>>
>>> In our particular case, we also have the fun problem of "How do you
>>> build CPython locally when you don't have a working Python interpreter
>>> yet?". A number of the more interesting alternative C/C++ build
>>> systems (like Meson, Scons, Waf and the Google "gyp" project that
>>> powers the node-gyp native extension build system for Node.js) require
>>> an already working Python interpreter to get things started (when
>>> starting from a completely blank slate, even our own build sequence
>>> has to build a deliberately broken partial interpreter just to freeze
>>> importlib._bootstrap before going on to build a properly configured
>>> one that can actually import modules).
>>>
>>
>> What about something like Google's new GN? It's designed to replace Gyp
>> in building Chromium and is written in C++.
>>
>> I'm pretty sure using a build system written in Lua to build Python is
>> kind of weird.
>>
>
>
>>
>> With all that in mind...what exactly would be requirements that Python
>> would need in an autotools replacement?
>>
>
> Can't require anything more than an ANSI C compiler basically. People run
> Python in places where they don't have much more than gcc.
>

Not even C99? Darn it...


>
>
>>
>>
>>>
>>> CMake seems like a plausible alternative that would avoid the
>>> bootstrapping problem (as even building it from source only requires a
>>> working C++ compiler for the platform, either native or
>>> cross-compiled): http://www.cmake.org/
>>>
>>> KDE switched over to CMake from autotools ~9 years ago
>>> (http://lwn.net/Articles/188693/) and plenty of other projects have
>>> either followed suit or simply never used autotools in the first
>>> place: http://www.cmake.org/Wiki/CMake_Projects
>>>
>>>
>> That was what I was referring to before.
>>
>
> I casually discussed the idea back at PyCon 2008 about using CMake and was
> told "no" because of potential build tool dependencies for CMake itself. So
> you have to make sure that you can prove CMake is highly portable.
>

I'm guessing that's because CMake is written in C++, which is a large
dependency.

I *really* wish someone on this earth still had a C++-to-C99 compiler like
cfront was.


>
>
>>
>>
>>> It's far too late in the 3.5 dev cycle to contemplate a change like
>>> that, but if anyone was looking for a potentially interesting side
>>> project, seeing exactly what was involved in getting CPython to build
>>> with CMake instead of autotools could be a worthwhile investigation:
>>>
>>> https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake
>>>
>>>
>> Someone is maintaining a set of CMake build files for Python 2.7
>> somewhere.
>>
>
> That's fine, but obviously things have been tweaked since then and this
> could only go into 3.x.
>
>
I know. Just an example to show it's possible.

So, here's what I've gathered thus far:

- Needs to be written in ANSI C89 and *only* ANSI C89
- Needs to support autotools-style
try-to-build-this-and-do-something-if-it-fails
- Needs to be nicer than autotools, otherwise this entire thread is 100%
pointless.



> -Brett
>
>
>>
>>
>>> Regards,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>>
>>
>>
>>
>> --
>> Ryan
>> If I were in a 10-story building glass-sided building and forced to write
>> either Go or autotools scripts, I?d jump out a window.
>> http://kirbyfan64.github.io/
>>
>>  _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/26b7eafb/attachment-0001.html>

From nas-python at arctrix.com  Wed Mar 11 20:43:25 2015
From: nas-python at arctrix.com (Neil Schemenauer)
Date: Wed, 11 Mar 2015 13:43:25 -0600
Subject: [Python-ideas] Migration of /usr/bin/python to python3
Message-ID: <20150311194325.GA15390@python.ca>

This has been brought up elsewhere, I think this is a better forum
to discuss it.  Changing /usr/bin/python directly to python3.x is
bad idea, in my option.  It's going to cause users trouble and the
benefit to Python 3.x users is not worth it.  Instead, let's try to
find a smooth migration path.

One idea is to install Python 3 as /usr/bin/python but have it
always execute scripts with Python 2 unless they are explicitly
marked as Python 3 compatible.  

The interpreter could parse the first line of the script to look for
options.  Traditionally the OS only passes the first option to the
interpreter so anything after that could be used to pass 3.x
specific options.

For example, a script starting with 

    #!/usr/bin/python -x -3

could signify that the script is compatible with a Python 3
interpreter.

An alternative idea is to allow something like the 'encoding'
declaration.  It wouldn't have to be in the first line and would
allow more flexibility.  Many years in the future when all scripts
are marked as Python 3 compatible, Python 2 can be removed from the
system and /usr/bin/python will start Python 3 by default.

I suppose running '/usr/bin/python' interactively could start Python
3 by default.  However, I strongly feel that existing Python 2
scripts should not get broken in a flag day like way.


  Neil

From mertz at gnosis.cx  Wed Mar 11 21:03:36 2015
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 11 Mar 2015 13:03:36 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311194325.GA15390@python.ca>
References: <20150311194325.GA15390@python.ca>
Message-ID: <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>

https://www.python.org/dev/peps/pep-0394/
On Mar 11, 2015 12:44 PM, "Neil Schemenauer" <nas-python at arctrix.com> wrote:

> This has been brought up elsewhere, I think this is a better forum
> to discuss it.  Changing /usr/bin/python directly to python3.x is
> bad idea, in my option.  It's going to cause users trouble and the
> benefit to Python 3.x users is not worth it.  Instead, let's try to
> find a smooth migration path.
>
> One idea is to install Python 3 as /usr/bin/python but have it
> always execute scripts with Python 2 unless they are explicitly
> marked as Python 3 compatible.
>
> The interpreter could parse the first line of the script to look for
> options.  Traditionally the OS only passes the first option to the
> interpreter so anything after that could be used to pass 3.x
> specific options.
>
> For example, a script starting with
>
>     #!/usr/bin/python -x -3
>
> could signify that the script is compatible with a Python 3
> interpreter.
>
> An alternative idea is to allow something like the 'encoding'
> declaration.  It wouldn't have to be in the first line and would
> allow more flexibility.  Many years in the future when all scripts
> are marked as Python 3 compatible, Python 2 can be removed from the
> system and /usr/bin/python will start Python 3 by default.
>
> I suppose running '/usr/bin/python' interactively could start Python
> 3 by default.  However, I strongly feel that existing Python 2
> scripts should not get broken in a flag day like way.
>
>
>   Neil
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/47ae6dbc/attachment.html>

From rymg19 at gmail.com  Wed Mar 11 21:10:24 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 11 Mar 2015 15:10:24 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311194325.GA15390@python.ca>
References: <20150311194325.GA15390@python.ca>
Message-ID: <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>

On Wed, Mar 11, 2015 at 2:43 PM, Neil Schemenauer <nas-python at arctrix.com>
wrote:

> This has been brought up elsewhere, I think this is a better forum
> to discuss it.  Changing /usr/bin/python directly to python3.x is
> bad idea, in my option.  It's going to cause users trouble and the
> benefit to Python 3.x users is not worth it.  Instead, let's try to
> find a smooth migration path.
>
> One idea is to install Python 3 as /usr/bin/python but have it
> always execute scripts with Python 2 unless they are explicitly
> marked as Python 3 compatible.
>
>
So Python 2 needs to be there anyway?


> The interpreter could parse the first line of the script to look for
> options.  Traditionally the OS only passes the first option to the
> interpreter so anything after that could be used to pass 3.x
> specific options.
>
> For example, a script starting with
>
>     #!/usr/bin/python -x -3
>
> could signify that the script is compatible with a Python 3
> interpreter.
>

What about using */usr/bin/env*? That wouldn't work; on my Linux system,
*/usr/bin/env* doesn't like any command like arguments.

Whenever I download a Python program, I usually check the hashbang to see
if it works with Python 2 or 3. This is bound to cause confusion, since a
quick glance would say "hey, this looks like it works under Python 2",
followed by the error-fest.


>
> An alternative idea is to allow something like the 'encoding'
> declaration.  It wouldn't have to be in the first line and would
> allow more flexibility.  Many years in the future when all scripts
> are marked as Python 3 compatible, Python 2 can be removed from the
> system and /usr/bin/python will start Python 3 by default.
>
>
I guess that's better.


> I suppose running '/usr/bin/python' interactively could start Python
> 3 by default.  However, I strongly feel that existing Python 2
> scripts should not get broken in a flag day like way.
>
>
But parsing hashbangs really isn't the answer. It's just really
Perl-magic-ish confusing.


>
>   Neil
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/8c6c44f4/attachment.html>

From skip.montanaro at gmail.com  Wed Mar 11 21:13:18 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 11 Mar 2015 15:13:18 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
Message-ID: <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>

On Wed, Mar 11, 2015 at 3:10 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> What about using */usr/bin/env*? That wouldn't work; on my Linux system,
> */usr/bin/env* doesn't like any command like arguments.


Not sure what planet your Linux system is from, but I use

#!/usr/bin/env python

in all my scripts. It likes the argument ("python" in this case) just
fine...

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/bb1a70a7/attachment.html>

From donald at stufft.io  Wed Mar 11 21:14:14 2015
From: donald at stufft.io (Donald Stufft)
Date: Wed, 11 Mar 2015 16:14:14 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311194325.GA15390@python.ca>
References: <20150311194325.GA15390@python.ca>
Message-ID: <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io>

What's worse. Breaking everything that uses an unversioned shebang because /usr/bin/python doesn't exist or breaking some things that doesn't work with Python 3 because /usr/bin/python is now 3?

Generally discussions along this line talk about Python 2 not being installed by default at all (whereas Python 3 would be). So really it's two different broken by default.


> On Mar 11, 2015, at 3:43 PM, Neil Schemenauer <nas-python at arctrix.com> wrote:
> 
> This has been brought up elsewhere, I think this is a better forum
> to discuss it.  Changing /usr/bin/python directly to python3.x is
> bad idea, in my option.  It's going to cause users trouble and the
> benefit to Python 3.x users is not worth it.  Instead, let's try to
> find a smooth migration path.
> 
> One idea is to install Python 3 as /usr/bin/python but have it
> always execute scripts with Python 2 unless they are explicitly
> marked as Python 3 compatible.  
> 
> The interpreter could parse the first line of the script to look for
> options.  Traditionally the OS only passes the first option to the
> interpreter so anything after that could be used to pass 3.x
> specific options.
> 
> For example, a script starting with 
> 
>    #!/usr/bin/python -x -3
> 
> could signify that the script is compatible with a Python 3
> interpreter.
> 
> An alternative idea is to allow something like the 'encoding'
> declaration.  It wouldn't have to be in the first line and would
> allow more flexibility.  Many years in the future when all scripts
> are marked as Python 3 compatible, Python 2 can be removed from the
> system and /usr/bin/python will start Python 3 by default.
> 
> I suppose running '/usr/bin/python' interactively could start Python
> 3 by default.  However, I strongly feel that existing Python 2
> scripts should not get broken in a flag day like way.
> 
> 
>  Neil
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From phd at phdru.name  Wed Mar 11 21:22:35 2015
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 11 Mar 2015 21:22:35 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311194325.GA15390@python.ca>
References: <20150311194325.GA15390@python.ca>
Message-ID: <20150311202235.GA30739@phdru.name>

Hi!

On Wed, Mar 11, 2015 at 01:43:25PM -0600, Neil Schemenauer <nas-python at arctrix.com> wrote:
[skip]
> a script starting with 
> 
>     #!/usr/bin/python -x -3
[skip]

   That wouldn't work. Shebang line accepts one and only one parameter.
There are operating systems that allow more but they are exceptions.

$ cat test1
#! /usr/local/bin/python -x3
print "Ok"

$ ./test1
Ok

$ cat test2
#! /usr/local/bin/python -x -3
print "Ok"

$ ./test2
Unknown option: - 
usage: /usr/local/bin/python [option] ... [-c cmd | -m mod | file | -]
[arg] ...
Try `python -h' for more information.

$ cat test3
#! /usr/bin/env python -x3
print "Ok"

$ ./test3
env: python -x3: No such file or directory

$ cat test4
#! /usr/bin/env python -x -3
print "Ok"

$ ./test4
env: python -x -3: No such file or directory

$ uname -rs
FreeBSD 9.3-RELEASE

>   Neil

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From skip.montanaro at gmail.com  Wed Mar 11 21:26:07 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 11 Mar 2015 15:26:07 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311202235.GA30739@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <20150311202235.GA30739@phdru.name>
Message-ID: <CANc-5Uycym+T0PUMCAk68Rve6wVRHxS9DTxaL3ov8scr21OPZA@mail.gmail.com>

On Wed, Mar 11, 2015 at 3:22 PM, Oleg Broytman <phd at phdru.name> wrote:

>    That wouldn't work. Shebang line accepts one and only one parameter.
> There are operating systems that allow more but they are exceptions.
>

I think Neal was suggesting that you rely on that property (in which case
the exceptions might be the problem). If I understood correctly,
/usr/bin/python would be started by the kernel, and it would be passed the
-x flag. It would then inspect the #! line again, paying attention to the
second (and following) arguments. Seeing the -3, it would continue
executing. If it saw nothing (or saw -2?) it would exec /usr/bin/python2
(or similar).

I hope I'm not putting words in Neal's mouth.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/6c1d1c79/attachment.html>

From phd at phdru.name  Wed Mar 11 21:32:45 2015
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 11 Mar 2015 21:32:45 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CANc-5Uycym+T0PUMCAk68Rve6wVRHxS9DTxaL3ov8scr21OPZA@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <20150311202235.GA30739@phdru.name>
 <CANc-5Uycym+T0PUMCAk68Rve6wVRHxS9DTxaL3ov8scr21OPZA@mail.gmail.com>
Message-ID: <20150311203245.GB30739@phdru.name>

On Wed, Mar 11, 2015 at 03:26:07PM -0500, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> On Wed, Mar 11, 2015 at 3:22 PM, Oleg Broytman <phd at phdru.name> wrote:
> 
> >    That wouldn't work. Shebang line accepts one and only one parameter.
> > There are operating systems that allow more but they are exceptions.
> 
> If I understood correctly,
> /usr/bin/python would be started by the kernel, and it would be passed the
> -x flag.

   But it (the kernel) doesn't:

$ cat test2
#! /usr/local/bin/python -x -3
print "Ok"

$ ./test2
Unknown option: - 
usage: /usr/local/bin/python [option] ... [-c cmd | -m mod | file | -]
[arg] ...
Try `python -h' for more information.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From mal at egenix.com  Wed Mar 11 21:38:06 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 11 Mar 2015 21:38:06 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
Message-ID: <5500A7AE.9020400@egenix.com>

On 11.03.2015 21:03, David Mertz wrote:
> https://www.python.org/dev/peps/pep-0394/

I think the migration should start with modifying scripts to use

#!/usr/bin/env python2

when they are Python 2 only (*) and

#!/usr/bin/env python3

when they are Python 3 only and

#!/usr/bin/env python

only when they support both Python 2 and 3.

"Explicit is better than implicit" and all that Zen :-)

Once that's done, switching the symlink is really a no-brainer.

The recipe for this is easy too:

1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
2. migrate your scripts one by one to either Python 3.x or
   to Python 2.7 + 3.4+
3. after migration replace "#!/usr/bin/env python2" with
   "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.

(*) Some OSes may require to use python2.7, if they don't come
with a symlink from python2 -> python2.7.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From rymg19 at gmail.com  Wed Mar 11 21:50:31 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 11 Mar 2015 15:50:31 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
Message-ID: <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>

No; this won't work:

#!/usr/bin/env python -x -3

As Oleg said, anything in the hashbang can only be passed one argument.
It's not the kernel that invokes that; it's the shell.

http://mail-index.netbsd.org/netbsd-users/2008/11/09/msg002388.html

On Wed, Mar 11, 2015 at 3:13 PM, Skip Montanaro <skip.montanaro at gmail.com>
wrote:

>
> On Wed, Mar 11, 2015 at 3:10 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> What about using */usr/bin/env*? That wouldn't work; on my Linux system,
>> */usr/bin/env* doesn't like any command like arguments.
>
>
> Not sure what planet your Linux system is from, but I use
>
> #!/usr/bin/env python
>
> in all my scripts. It likes the argument ("python" in this case) just
> fine...
>
> Skip
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/4cb69adc/attachment.html>

From dwblas at gmail.com  Wed Mar 11 22:08:20 2015
From: dwblas at gmail.com (David Blaschke)
Date: Wed, 11 Mar 2015 14:08:20 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
Message-ID: <CAFx9zoefqLM==LQCvNaCa9bhSqKEq6SmUteDB0z+MT4LjWaS5g@mail.gmail.com>

Sorry if this is a double post-didn't reply to all

Checking for default version is trivial unless I am missing something
 .import sys
 .if sys.version_info[0] < 3:
 .   ## /usr/bin/python2 via subprocess.call or whatever
 .else:
 .    ##/usr/bin/python3
-- 
Only when you actually get to the state where there is neither
delusion nor enlightenment are you finally comfortable...Foyan

From ethan at stoneleaf.us  Wed Mar 11 22:19:06 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Mar 2015 14:19:06 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500A7AE.9020400@egenix.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
Message-ID: <5500B14A.6090705@stoneleaf.us>

On 03/11/2015 01:38 PM, M.-A. Lemburg wrote:
> On 11.03.2015 21:03, David Mertz wrote:

> I think the migration should start with modifying scripts to use
> 
>   #!/usr/bin/env python2
> 
> when they are Python 2 only (*) and
> 
>   #!/usr/bin/env python3
> 
> when they are Python 3 only and
> 
>   #!/usr/bin/env python
> 
> only when they support both Python 2 and 3.

+1

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/c70b9ddb/attachment.sig>

From ianlee1521 at gmail.com  Wed Mar 11 22:52:06 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Wed, 11 Mar 2015 14:52:06 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500A7AE.9020400@egenix.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
Message-ID: <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>

+1

An issue that I've noticed recently (I haven't had the time to fully track
this one down yet, but I suspect it's a setuptools issue) takes a "last one
wins" approach to scripts that get written into the same bin directory
(e.g. --user installs).

$ python2.7 -m pip install --user ipython
>
> $ head -n1 ~/.local/bin/ipython*
> ==> /home/ianlee1521/.local/bin/ipython <==
> #!/usr/bin/python2.7
>
> ==> /home/ianlee1521/.local/bin/ipython2 <==
> #!/usr/bin/python2.7
>
> $ python3.4 -m pip install --user ipython
>
> $ head -n1 ~/.local/bin/ipython*
> ==> /home/ianlee1521/.local/bin/ipython <==
> #!/usr/bin/python3.4
>
> ==> /home/ianlee1521/.local/bin/ipython2 <==
> #!/usr/bin/python2.7
>
> ==> /home/ianlee1521/.local/bin/ipython3 <==
> #!/usr/bin/python3.4


So the packages end up in the separate, versioned directories
(~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in
the same bin directory, the implicit script "~/.local/bin/ipython" ends up
changing to use the hashbang of the Python version which last installed the
package (see highlighted above).

I would expect that the implicit script ("~/.local/bin/ipython") should use
the implicit python version, e.g. "#!/usr/bin/python".

Alternatively, maybe I just shouldn't ever be installing user versions of
both Python 2 and Python 3 versions of a package.



~ Ian Lee

On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 11.03.2015 21:03, David Mertz wrote:
> > https://www.python.org/dev/peps/pep-0394/
>
> I think the migration should start with modifying scripts to use
>
> #!/usr/bin/env python2
>
> when they are Python 2 only (*) and
>
> #!/usr/bin/env python3
>
> when they are Python 3 only and
>
> #!/usr/bin/env python
>
> only when they support both Python 2 and 3.
>
> "Explicit is better than implicit" and all that Zen :-)
>
> Once that's done, switching the symlink is really a no-brainer.
>
> The recipe for this is easy too:
>
> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
> 2. migrate your scripts one by one to either Python 3.x or
>    to Python 2.7 + 3.4+
> 3. after migration replace "#!/usr/bin/env python2" with
>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>
> (*) Some OSes may require to use python2.7, if they don't come
> with a symlink from python2 -> python2.7.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Mar 11 2015)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/c1c09bfd/attachment.html>

From abarnert at yahoo.com  Wed Mar 11 23:29:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 15:29:20 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
Message-ID: <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>

On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
> 
> +1
> 
> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).

Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?

> 
>> $ python2.7 -m pip install --user ipython
>>  
>> $ head -n1 ~/.local/bin/ipython*
>> ==> /home/ianlee1521/.local/bin/ipython <==
>> #!/usr/bin/python2.7
>>  
>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>> #!/usr/bin/python2.7
>>  
>> $ python3.4 -m pip install --user ipython
>>  
>> $ head -n1 ~/.local/bin/ipython*
>> ==> /home/ianlee1521/.local/bin/ipython <==
>> #!/usr/bin/python3.4
>>  
>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>> #!/usr/bin/python2.7
>>  
>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>> #!/usr/bin/python3.4
> 
> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).
> 
> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".
> 
> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.
> 
> 
> ~ Ian Lee
> 
>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 11.03.2015 21:03, David Mertz wrote:
>> > https://www.python.org/dev/peps/pep-0394/
>> 
>> I think the migration should start with modifying scripts to use
>> 
>> #!/usr/bin/env python2
>> 
>> when they are Python 2 only (*) and
>> 
>> #!/usr/bin/env python3
>> 
>> when they are Python 3 only and
>> 
>> #!/usr/bin/env python
>> 
>> only when they support both Python 2 and 3.
>> 
>> "Explicit is better than implicit" and all that Zen :-)
>> 
>> Once that's done, switching the symlink is really a no-brainer.
>> 
>> The recipe for this is easy too:
>> 
>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>> 2. migrate your scripts one by one to either Python 3.x or
>>    to Python 2.7 + 3.4+
>> 3. after migration replace "#!/usr/bin/env python2" with
>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>> 
>> (*) Some OSes may require to use python2.7, if they don't come
>> with a symlink from python2 -> python2.7.
>> 
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>> 
>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>> ________________________________________________________________________
>> 
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>> 
>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>                http://www.egenix.com/company/contact/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/6af1566e/attachment-0001.html>

From greg at krypto.org  Wed Mar 11 23:41:47 2015
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 11 Mar 2015 22:41:47 +0000
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500A7AE.9020400@egenix.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
Message-ID: <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>

I think /usr/bin/env is a bad idea to promote. It changes from an explicit
designation of what you need in order to execute to something being derived
from the users $PATH.  It is *less specific* than /usr/bin/python. For some
people that is the goal, but I discourage it.

I do agree with Donald (and Nick from the other conversation elsewhere)
that once this is framed as "the OS won't have Python 2.7 installed at all"
it is a much better idea: Yes, we will eventually need to update PEP 394 to
recommend under what circumstances /usr/bin/python -> python3.6 is
acceptable. (it should never change meaning on a given OS release based on
the packages installed and 2.7 won't be required for the default install
images so 3.x makes sense)

The harsh reality is that anyone who does not explicitly specify a specific
X.Y version of Python in their #! line via has effectively declared that
they do not care, consequences be damned.  If your code supports multiple
versions, fixing up the #! line to use the locally appropriate one is an
install time task.

-gps

On Wed, Mar 11, 2015 at 1:38 PM M.-A. Lemburg <mal at egenix.com> wrote:

> On 11.03.2015 21:03, David Mertz wrote:
> > https://www.python.org/dev/peps/pep-0394/
>
> I think the migration should start with modifying scripts to use
>
> #!/usr/bin/env python2
>
> when they are Python 2 only (*) and
>
> #!/usr/bin/env python3
>
> when they are Python 3 only and
>
> #!/usr/bin/env python
>
> only when they support both Python 2 and 3.
>
> "Explicit is better than implicit" and all that Zen :-)
>
> Once that's done, switching the symlink is really a no-brainer.
>
> The recipe for this is easy too:
>
> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
> 2. migrate your scripts one by one to either Python 3.x or
>    to Python 2.7 + 3.4+
> 3. after migration replace "#!/usr/bin/env python2" with
>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>
> (*) Some OSes may require to use python2.7, if they don't come
> with a symlink from python2 -> python2.7.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Mar 11 2015)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/6c3a4ca1/attachment.html>

From ianlee1521 at gmail.com  Wed Mar 11 23:45:15 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Wed, 11 Mar 2015 15:45:15 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
Message-ID: <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>

Sure, and I agree that called "ipython2" / "ipython3" explicitly is better.
I just expected it would default to use whatever "python" is implicitly on
my path. E.g. that "python" and "ipython" would use the same interpreter.

~ Ian Lee
On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:

> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>
> +1
>
> An issue that I've noticed recently (I haven't had the time to fully track
> this one down yet, but I suspect it's a setuptools issue) takes a "last one
> wins" approach to scripts that get written into the same bin directory
> (e.g. --user installs).
>
>
> Many packages will install spamX and spamX.Y scripts as well as plain
> spam, so you can deal with this the same way you deal with Python
> instead--run ipython3 or ipython3.4 instead of just ipython. (Not to
> mention ipython_pypy3!) Also, sometimes python2 and python3 will install to
> different directories (/usr for the one that's part of the system,
> /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...),
> so "last one wins" isn't even predictable cross-platform; you have to be
> able to deal with either "last one wins" or "controlled by PATH" when you
> want to just run "ipython". But really, since these scripts don't need to
> be run by a shbang or other automated mechanism, and the user who
> intentionally installs IPython for both 2.7 and 3.4 is going to need some
> way to manually select the one they want to run each time anyway. So I
> think this solution isn't too bad: 90% of the time, a script is only
> installed for one Python version so "last one wins" works trivially; most
> of the rest of the time, the user has two or more and needs to use
> "ipython2" vs. "ipython3" rather than just "ipython" because how else could
> be use both as desired, so "last one wins" is irrelevant.The thing is, I
> don't think you get this script versioning automatically from setuptools.
> If you don't, maybe you should?
>
>
> $ python2.7 -m pip install --user ipython
>>
>> $ head -n1 ~/.local/bin/ipython*
>> ==> /home/ianlee1521/.local/bin/ipython <==
>> #!/usr/bin/python2.7
>>
>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>> #!/usr/bin/python2.7
>>
>> $ python3.4 -m pip install --user ipython
>>
>> $ head -n1 ~/.local/bin/ipython*
>> ==> /home/ianlee1521/.local/bin/ipython <==
>> #!/usr/bin/python3.4
>>
>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>> #!/usr/bin/python2.7
>>
>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>> #!/usr/bin/python3.4
>
>
> So the packages end up in the separate, versioned directories
> (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in
> the same bin directory, the implicit script "~/.local/bin/ipython" ends up
> changing to use the hashbang of the Python version which last installed the
> package (see highlighted above).
>
> I would expect that the implicit script ("~/.local/bin/ipython") should
> use the implicit python version, e.g. "#!/usr/bin/python".
>
> Alternatively, maybe I just shouldn't ever be installing user versions of
> both Python 2 and Python 3 versions of a package.
>
>
>
> ~ Ian Lee
>
> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>
>> On 11.03.2015 21:03, David Mertz wrote:
>> > https://www.python.org/dev/peps/pep-0394/
>>
>> I think the migration should start with modifying scripts to use
>>
>> #!/usr/bin/env python2
>>
>> when they are Python 2 only (*) and
>>
>> #!/usr/bin/env python3
>>
>> when they are Python 3 only and
>>
>> #!/usr/bin/env python
>>
>> only when they support both Python 2 and 3.
>>
>> "Explicit is better than implicit" and all that Zen :-)
>>
>> Once that's done, switching the symlink is really a no-brainer.
>>
>> The recipe for this is easy too:
>>
>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>> 2. migrate your scripts one by one to either Python 3.x or
>>    to Python 2.7 + 3.4+
>> 3. after migration replace "#!/usr/bin/env python2" with
>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>
>> (*) Some OSes may require to use python2.7, if they don't come
>> with a symlink from python2 -> python2.7.
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>> ________________________________________________________________________
>>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>
>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>                http://www.egenix.com/company/contact/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/bef420ff/attachment-0001.html>

From abarnert at yahoo.com  Wed Mar 11 23:45:19 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 15:45:19 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500A7AE.9020400@egenix.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
Message-ID: <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com>

On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 11.03.2015 21:03, David Mertz wrote:
>> https://www.python.org/dev/peps/pep-0394/
> 
> I think the migration should start with modifying scripts to use
> 
> #!/usr/bin/env python2
> 
> when they are Python 2 only (*) and
> 
> #!/usr/bin/env python3
> 
> when they are Python 3 only and
> 
> #!/usr/bin/env python
> 
> only when they support both Python 2 and 3.

Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".

So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.

In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)

> "Explicit is better than implicit" and all that Zen :-)
> 
> Once that's done, switching the symlink is really a no-brainer.
> 
> The recipe for this is easy too:
> 
> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
> 2. migrate your scripts one by one to either Python 3.x or
>   to Python 2.7 + 3.4+
> 3. after migration replace "#!/usr/bin/env python2" with
>   "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
> 
> (*) Some OSes may require to use python2.7, if they don't come
> with a symlink from python2 -> python2.7.
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From chris.barker at noaa.gov  Wed Mar 11 23:54:04 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Mar 2015 15:54:04 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
Message-ID: <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>

On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg at krypto.org> wrote:

> I think /usr/bin/env is a bad idea to promote. It changes from an explicit
> designation of what you need in order to execute to something being derived
> from the users $PATH.  It is *less specific* than /usr/bin/python. For some
> people that is the goal, but I discourage it.
>

I don't think the goal is specificity, or lack there of, but a shift in
control:

Who controls where python is installed?

If you put an explicit full path on the #! line, then the script author
controls which python is used, AND where it must be installed.

If you put /usr/bin env on the #! line, then the script author is saying
'use python", and the script user can have python installed anywhere they
want, and can use PATH to tell everything what the default it.

I think that _should_ be encouraged -- but ideally with some versioning:

#!/usr/bin/env python2

or even:

#!/usr/bin/env python3.4

If any of you remember what a pain it was when RedHat installed system
tools with (I think):

!#/usr/bin/env python

You'll know what a nightmare that was -- you could not upgrade the
'default" python without breaking stuff.

So system tools should probably use the full specific path -- the script is
tied closely to the environment. But if they had at least put a version in
there, it would have worked fine:

#!/usr/bin/env python1.5

These days, I try to use distutils or setuptools to install scripts, and
they can do the right thing to use the correct python to start up anyway...

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/8a57da62/attachment.html>

From abarnert at yahoo.com  Wed Mar 11 23:52:12 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 15:52:12 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
Message-ID: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>

On Mar 11, 2015, at 3:45 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
> Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter.
> 
That's actually not a bad idea; I didn't even think about it because Unix stuff doesn't "naturally" work that way, but it's probably not too hard to do it "unnaturally".

Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython".
> ~ Ian Lee
> 
>> On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>>> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>>> 
>>> +1
>>> 
>>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).
>> 
>> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?
>> 
>>> 
>>>> $ python2.7 -m pip install --user ipython
>>>>  
>>>> $ head -n1 ~/.local/bin/ipython*
>>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>>> #!/usr/bin/python2.7
>>>>  
>>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>>> #!/usr/bin/python2.7
>>>>  
>>>> $ python3.4 -m pip install --user ipython
>>>>  
>>>> $ head -n1 ~/.local/bin/ipython*
>>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>>> #!/usr/bin/python3.4
>>>>  
>>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>>> #!/usr/bin/python2.7
>>>>  
>>>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>>>> #!/usr/bin/python3.4
>>> 
>>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).
>>> 
>>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".
>>> 
>>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.
>>> 
>>> 
>>> ~ Ian Lee
>>> 
>>>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> On 11.03.2015 21:03, David Mertz wrote:
>>>> > https://www.python.org/dev/peps/pep-0394/
>>>> 
>>>> I think the migration should start with modifying scripts to use
>>>> 
>>>> #!/usr/bin/env python2
>>>> 
>>>> when they are Python 2 only (*) and
>>>> 
>>>> #!/usr/bin/env python3
>>>> 
>>>> when they are Python 3 only and
>>>> 
>>>> #!/usr/bin/env python
>>>> 
>>>> only when they support both Python 2 and 3.
>>>> 
>>>> "Explicit is better than implicit" and all that Zen :-)
>>>> 
>>>> Once that's done, switching the symlink is really a no-brainer.
>>>> 
>>>> The recipe for this is easy too:
>>>> 
>>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>>> 2. migrate your scripts one by one to either Python 3.x or
>>>>    to Python 2.7 + 3.4+
>>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>>> 
>>>> (*) Some OSes may require to use python2.7, if they don't come
>>>> with a symlink from python2 -> python2.7.
>>>> 
>>>> --
>>>> Marc-Andre Lemburg
>>>> eGenix.com
>>>> 
>>>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>>>> ________________________________________________________________________
>>>> 
>>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>>> 
>>>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>>>                http://www.egenix.com/company/contact/
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>> 
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/49886a9d/attachment-0001.html>

From rymg19 at gmail.com  Thu Mar 12 00:02:49 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 11 Mar 2015 18:02:49 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
Message-ID: <CAO41-mOypBaYPcmcm2+bdU1t0s4Nx00-Fh7eV9Fk1tFTeQbwuw@mail.gmail.com>

Bad idea? What about when you're running simple scripts you've downloaded
and Python 3 is on an external hard drive, not /usr/bin?

That's my usage case, anyway.

On Wed, Mar 11, 2015 at 5:41 PM, Gregory P. Smith <greg at krypto.org> wrote:

> I think /usr/bin/env is a bad idea to promote. It changes from an explicit
> designation of what you need in order to execute to something being derived
> from the users $PATH.  It is *less specific* than /usr/bin/python. For some
> people that is the goal, but I discourage it.
>
> I do agree with Donald (and Nick from the other conversation elsewhere)
> that once this is framed as "the OS won't have Python 2.7 installed at all"
> it is a much better idea: Yes, we will eventually need to update PEP 394 to
> recommend under what circumstances /usr/bin/python -> python3.6 is
> acceptable. (it should never change meaning on a given OS release based on
> the packages installed and 2.7 won't be required for the default install
> images so 3.x makes sense)
>
> The harsh reality is that anyone who does not explicitly specify a
> specific X.Y version of Python in their #! line via has effectively
> declared that they do not care, consequences be damned.  If your code
> supports multiple versions, fixing up the #! line to use the locally
> appropriate one is an install time task.
>
> -gps
>
> On Wed, Mar 11, 2015 at 1:38 PM M.-A. Lemburg <mal at egenix.com> wrote:
>
>> On 11.03.2015 21:03, David Mertz wrote:
>> > https://www.python.org/dev/peps/pep-0394/
>>
>> I think the migration should start with modifying scripts to use
>>
>> #!/usr/bin/env python2
>>
>> when they are Python 2 only (*) and
>>
>> #!/usr/bin/env python3
>>
>> when they are Python 3 only and
>>
>> #!/usr/bin/env python
>>
>> only when they support both Python 2 and 3.
>>
>> "Explicit is better than implicit" and all that Zen :-)
>>
>> Once that's done, switching the symlink is really a no-brainer.
>>
>> The recipe for this is easy too:
>>
>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>> 2. migrate your scripts one by one to either Python 3.x or
>>    to Python 2.7 + 3.4+
>> 3. after migration replace "#!/usr/bin/env python2" with
>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>
>> (*) Some OSes may require to use python2.7, if they don't come
>> with a symlink from python2 -> python2.7.
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>> ________________________________________________________________________
>>
>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>
>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>                http://www.egenix.com/company/contact/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/b47760b3/attachment.html>

From donald at stufft.io  Thu Mar 12 00:03:33 2015
From: donald at stufft.io (Donald Stufft)
Date: Wed, 11 Mar 2015 19:03:33 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
 <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
Message-ID: <8E9600FA-2CE4-4A01-BBC2-B1F950418288@stufft.io>


> On Mar 11, 2015, at 6:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
> On Mar 11, 2015, at 3:45 PM, Ian Lee <ianlee1521 at gmail.com <mailto:ianlee1521 at gmail.com>> wrote:
>> Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter.
>> 
> That's actually not a bad idea; I didn't even think about it because Unix stuff doesn't "naturally" work that way, but it's probably not too hard to do it "unnaturally".
> 
> Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython?.


This is a problem that we?ve yet to tackle really (besides special cases for pip itself). You can?t really do it reasonably without not making Wheels, or making wheels that are python version specific. This is because the build process runs on the author?s machine and ?bakes? in the names of the commands at that time.

It?s likely going to require figuring out which trade off makes the most sense, adding explicit fooX.Y is one option, but that leaves into question what we do with alternate interpreters like PyPy or Jython (including versioning them, because you might have multiple versions of them installed). It also can cause issues when you have multiple things called ?python2? or ?python2.7? on your path, which version of ?python2? does ?foo2? call? Obviously it?s possible to just define something and say ?this is what it is? but we need to figure out what that thing should be, and if the trade offs make sense.

On the other side, executing versioned commands with -m is not quite as nice (python2.7 -m pip is more letters than pip2.7 and doesn?t autocomplete) however it is really explicit under which interpreter it?s going to run and support for things like alternate interpreters and multiple versions is built in.

>> ~ Ian Lee
>> 
>> On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com <mailto:abarnert at yahoo.com>> wrote:
>> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com <mailto:ianlee1521 at gmail.com>> wrote:
>> 
>>> +1
>>> 
>>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).
>> 
>> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?
>> 
>>> 
>>> $ python2.7 -m pip install --user ipython
>>> 
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python2.7
>>> 
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>> 
>>> $ python3.4 -m pip install --user ipython
>>> 
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python3.4
>>> 
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>> 
>>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>>> #!/usr/bin/python3.4
>>> 
>>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).
>>> 
>>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".
>>> 
>>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.
>>> 
>>> 
>>> ~ Ian Lee
>>> 
>>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com <mailto:mal at egenix.com>> wrote:
>>> On 11.03.2015 21:03, David Mertz wrote:
>>> > https://www.python.org/dev/peps/pep-0394/ <https://www.python.org/dev/peps/pep-0394/>
>>> 
>>> I think the migration should start with modifying scripts to use
>>> 
>>> #!/usr/bin/env python2
>>> 
>>> when they are Python 2 only (*) and
>>> 
>>> #!/usr/bin/env python3
>>> 
>>> when they are Python 3 only and
>>> 
>>> #!/usr/bin/env python
>>> 
>>> only when they support both Python 2 and 3.
>>> 
>>> "Explicit is better than implicit" and all that Zen :-)
>>> 
>>> Once that's done, switching the symlink is really a no-brainer.
>>> 
>>> The recipe for this is easy too:
>>> 
>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>> 2. migrate your scripts one by one to either Python 3.x or
>>>    to Python 2.7 + 3.4+
>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>> 
>>> (*) Some OSes may require to use python2.7, if they don't come
>>> with a symlink from python2 -> python2.7.
>>> 
>>> --
>>> Marc-Andre Lemburg
>>> eGenix.com <http://egenix.com/>
>>> 
>>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/ <http://www.egenix.com/>
>>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/ <http://zope.egenix.com/>
>>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/ <http://python.egenix.com/>
>>> ________________________________________________________________________
>>> 
>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>> 
>>>    eGenix.com <http://egenix.com/> Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>>                http://www.egenix.com/company/contact/ <http://www.egenix.com/company/contact/>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org <mailto:Python-ideas at python.org>
>>> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
>>> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
>>> 
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org <mailto:Python-ideas at python.org>
>>> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas>
>>> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>_______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/29326724/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/29326724/attachment-0001.sig>

From ianlee1521 at gmail.com  Thu Mar 12 00:03:35 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Wed, 11 Mar 2015 16:03:35 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
 <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
Message-ID: <CA+GyjM=tFNPT2LaTebhwfYccDGJUzQ8aesdTODF3vBTLXRiWbQ@mail.gmail.com>

~ Ian Lee

On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 11, 2015, at 3:45 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>
> Sure, and I agree that called "ipython2" / "ipython3" explicitly is
> better. I just expected it would default to use whatever "python" is
> implicitly on my path. E.g. that "python" and "ipython" would use the same
> interpreter.
>
> That's actually not a bad idea; I didn't even think about it because Unix
> stuff doesn't "naturally" work that way, but it's probably not too hard to
> do it "unnaturally".
>
> Maybe setuptools could check "there's something on the path called python
> that doesn't run the same version of Python that I'm running right now, so
> only install ipython3.4 and ipython3, not ipython".
>

I'm not sure this should be a decision at install time to set the hashbang.
Instead I'd propose the hashbangs should end up:

ipython  ==> "#! /usr/bin/env python"
ipython2  ==> "#! /usr/bin/env python2"
ipython3  ==> "#! /usr/bin/env python3"


> ~ Ian Lee
> On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>
>> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>>
>> +1
>>
>> An issue that I've noticed recently (I haven't had the time to fully
>> track this one down yet, but I suspect it's a setuptools issue) takes a
>> "last one wins" approach to scripts that get written into the same bin
>> directory (e.g. --user installs).
>>
>>
>> Many packages will install spamX and spamX.Y scripts as well as plain
>> spam, so you can deal with this the same way you deal with Python
>> instead--run ipython3 or ipython3.4 instead of just ipython. (Not to
>> mention ipython_pypy3!) Also, sometimes python2 and python3 will install to
>> different directories (/usr for the one that's part of the system,
>> /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...),
>> so "last one wins" isn't even predictable cross-platform; you have to be
>> able to deal with either "last one wins" or "controlled by PATH" when you
>> want to just run "ipython". But really, since these scripts don't need to
>> be run by a shbang or other automated mechanism, and the user who
>> intentionally installs IPython for both 2.7 and 3.4 is going to need some
>> way to manually select the one they want to run each time anyway. So I
>> think this solution isn't too bad: 90% of the time, a script is only
>> installed for one Python version so "last one wins" works trivially; most
>> of the rest of the time, the user has two or more and needs to use
>> "ipython2" vs. "ipython3" rather than just "ipython" because how else could
>> be use both as desired, so "last one wins" is irrelevant.The thing is, I
>> don't think you get this script versioning automatically from setuptools.
>> If you don't, maybe you should?
>>
>>
>> $ python2.7 -m pip install --user ipython
>>>
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python2.7
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>>
>>> $ python3.4 -m pip install --user ipython
>>>
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python3.4
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>>> #!/usr/bin/python3.4
>>
>>
>> So the packages end up in the separate, versioned directories
>> (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in
>> the same bin directory, the implicit script "~/.local/bin/ipython" ends up
>> changing to use the hashbang of the Python version which last installed the
>> package (see highlighted above).
>>
>> I would expect that the implicit script ("~/.local/bin/ipython") should
>> use the implicit python version, e.g. "#!/usr/bin/python".
>>
>> Alternatively, maybe I just shouldn't ever be installing user versions of
>> both Python 2 and Python 3 versions of a package.
>>
>>
>>
>> ~ Ian Lee
>>
>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> On 11.03.2015 21:03, David Mertz wrote:
>>> > https://www.python.org/dev/peps/pep-0394/
>>>
>>> I think the migration should start with modifying scripts to use
>>>
>>> #!/usr/bin/env python2
>>>
>>> when they are Python 2 only (*) and
>>>
>>> #!/usr/bin/env python3
>>>
>>> when they are Python 3 only and
>>>
>>> #!/usr/bin/env python
>>>
>>> only when they support both Python 2 and 3.
>>>
>>> "Explicit is better than implicit" and all that Zen :-)
>>>
>>> Once that's done, switching the symlink is really a no-brainer.
>>>
>>> The recipe for this is easy too:
>>>
>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>> 2. migrate your scripts one by one to either Python 3.x or
>>>    to Python 2.7 + 3.4+
>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>>
>>> (*) Some OSes may require to use python2.7, if they don't come
>>> with a symlink from python2 -> python2.7.
>>>
>>> --
>>> Marc-Andre Lemburg
>>> eGenix.com
>>>
>>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>>> ________________________________________________________________________
>>>
>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>>
>>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>>                http://www.egenix.com/company/contact/
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/668cc108/attachment.html>

From ianlee1521 at gmail.com  Thu Mar 12 00:03:42 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Wed, 11 Mar 2015 16:03:42 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
Message-ID: <CA+GyjMnSVNEZEjmyBVU6LBnkEKRQ_xB+DUatSt5Nc_e7AxaUvA@mail.gmail.com>

~ Ian Lee

On Wed, Mar 11, 2015 at 3:54 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
>> I think /usr/bin/env is a bad idea to promote. It changes from an
>> explicit designation of what you need in order to execute to something
>> being derived from the users $PATH.  It is *less specific* than
>> /usr/bin/python. For some people that is the goal, but I discourage it.
>>
>
> I don't think the goal is specificity, or lack there of, but a shift in
> control:
>
> Who controls where python is installed?
>
> If you put an explicit full path on the #! line, then the script author
> controls which python is used, AND where it must be installed.
>
> If you put /usr/bin env on the #! line, then the script author is saying
> 'use python", and the script user can have python installed anywhere they
> want, and can use PATH to tell everything what the default it.
>
> I think that _should_ be encouraged -- but ideally with some versioning:
>
> #!/usr/bin/env python2
>
> or even:
>
> #!/usr/bin/env python3.4
>

+1 to encouraging this.


>
> If any of you remember what a pain it was when RedHat installed system
> tools with (I think):
>
> !#/usr/bin/env python
>
> You'll know what a nightmare that was -- you could not upgrade the
> 'default" python without breaking stuff.
>
> So system tools should probably use the full specific path -- the script
> is tied closely to the environment. But if they had at least put a version
> in there, it would have worked fine:
>
> #!/usr/bin/env python1.5
>
> These days, I try to use distutils or setuptools to install scripts, and
> they can do the right thing to use the correct python to start up anyway...
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/3b2b85de/attachment-0001.html>

From rymg19 at gmail.com  Thu Mar 12 00:12:00 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 11 Mar 2015 18:12:00 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
Message-ID: <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>

On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
>> I think /usr/bin/env is a bad idea to promote. It changes from an
>> explicit designation of what you need in order to execute to something
>> being derived from the users $PATH.  It is *less specific* than
>> /usr/bin/python. For some people that is the goal, but I discourage it.
>>
>
> I don't think the goal is specificity, or lack there of, but a shift in
> control:
>
> Who controls where python is installed?
>
> If you put an explicit full path on the #! line, then the script author
> controls which python is used, AND where it must be installed.
>
> If you put /usr/bin env on the #! line, then the script author is saying
> 'use python", and the script user can have python installed anywhere they
> want, and can use PATH to tell everything what the default it.
>
> I think that _should_ be encouraged -- but ideally with some versioning:
>
> #!/usr/bin/env python2
>
> or even:
>
> #!/usr/bin/env python3.4
>
>
Please, just explicit major versions, not minor ones. python3, not
python3.4. Just earlier this year I had to fix about 20 scripts that ran
perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.


> If any of you remember what a pain it was when RedHat installed system
> tools with (I think):
>
> !#/usr/bin/env python
>
> You'll know what a nightmare that was -- you could not upgrade the
> 'default" python without breaking stuff.
>
> So system tools should probably use the full specific path -- the script
> is tied closely to the environment. But if they had at least put a version
> in there, it would have worked fine:
>
> #!/usr/bin/env python1.5
>
> These days, I try to use distutils or setuptools to install scripts, and
> they can do the right thing to use the correct python to start up anyway...
>
> -CHB
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/b38c4a30/attachment.html>

From chris.barker at noaa.gov  Wed Mar 11 22:27:55 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Mar 2015 14:27:55 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500B14A.6090705@stoneleaf.us>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com> <5500B14A.6090705@stoneleaf.us>
Message-ID: <CALGmxE+4OsouOP526b9JLQkrmZ+DMc8BUE0zSgOR9DYrMzRHog@mail.gmail.com>

On Wed, Mar 11, 2015 at 2:19 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/11/2015 01:38 PM, M.-A. Lemburg wrote:
> > On 11.03.2015 21:03, David Mertz wrote:
>
> > I think the migration should start with modifying scripts to use
> >
> >   #!/usr/bin/env python2
> >
> > when they are Python 2 only (*) and
> >
> >   #!/usr/bin/env python3
> >
> > when they are Python 3 only and
> >
> >   #!/usr/bin/env python
> >
> > only when they support both Python 2 and 3.
>

and for that matter:

#!/usr/bin/env python3.4

If the minor version matters.

We all should have been doing this for years!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/a33caba8/attachment.html>

From chris.barker at noaa.gov  Thu Mar 12 00:06:22 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 11 Mar 2015 16:06:22 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
 <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
Message-ID: <CALGmxE+-D1+MHRvC+RzTcAF31GG-QW3Fw=At_fBEHNQfTHLONA@mail.gmail.com>

On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> Maybe setuptools could check "there's something on the path called python
> that doesn't run the same version of Python that I'm running right now, so
> only install ipython3.4 and ipython3, not ipython".
>

hmm -- that seems odd -- if you re-arranged the path after it was
installed, it would be different???

I think you can count on folks use PATH to determine what they want as
default -- so in python2, setuptools would put
ipython and ipython2

and in the python3 install, it would put:

ipython and ipython3

so if you asked for ipython3 or ipython2, you'd get what you asked for --
and if you asked for ipython the PATH would determine what you'd get.

-CHB





> ~ Ian Lee
> On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>
>> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>>
>> +1
>>
>> An issue that I've noticed recently (I haven't had the time to fully
>> track this one down yet, but I suspect it's a setuptools issue) takes a
>> "last one wins" approach to scripts that get written into the same bin
>> directory (e.g. --user installs).
>>
>>
>> Many packages will install spamX and spamX.Y scripts as well as plain
>> spam, so you can deal with this the same way you deal with Python
>> instead--run ipython3 or ipython3.4 instead of just ipython. (Not to
>> mention ipython_pypy3!) Also, sometimes python2 and python3 will install to
>> different directories (/usr for the one that's part of the system,
>> /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...),
>> so "last one wins" isn't even predictable cross-platform; you have to be
>> able to deal with either "last one wins" or "controlled by PATH" when you
>> want to just run "ipython". But really, since these scripts don't need to
>> be run by a shbang or other automated mechanism, and the user who
>> intentionally installs IPython for both 2.7 and 3.4 is going to need some
>> way to manually select the one they want to run each time anyway. So I
>> think this solution isn't too bad: 90% of the time, a script is only
>> installed for one Python version so "last one wins" works trivially; most
>> of the rest of the time, the user has two or more and needs to use
>> "ipython2" vs. "ipython3" rather than just "ipython" because how else could
>> be use both as desired, so "last one wins" is irrelevant.The thing is, I
>> don't think you get this script versioning automatically from setuptools.
>> If you don't, maybe you should?
>>
>>
>> $ python2.7 -m pip install --user ipython
>>>
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python2.7
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>>
>>> $ python3.4 -m pip install --user ipython
>>>
>>> $ head -n1 ~/.local/bin/ipython*
>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>> #!/usr/bin/python3.4
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>> #!/usr/bin/python2.7
>>>
>>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>>> #!/usr/bin/python3.4
>>
>>
>> So the packages end up in the separate, versioned directories
>> (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in
>> the same bin directory, the implicit script "~/.local/bin/ipython" ends up
>> changing to use the hashbang of the Python version which last installed the
>> package (see highlighted above).
>>
>> I would expect that the implicit script ("~/.local/bin/ipython") should
>> use the implicit python version, e.g. "#!/usr/bin/python".
>>
>> Alternatively, maybe I just shouldn't ever be installing user versions of
>> both Python 2 and Python 3 versions of a package.
>>
>>
>>
>> ~ Ian Lee
>>
>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> On 11.03.2015 21:03, David Mertz wrote:
>>> > https://www.python.org/dev/peps/pep-0394/
>>>
>>> I think the migration should start with modifying scripts to use
>>>
>>> #!/usr/bin/env python2
>>>
>>> when they are Python 2 only (*) and
>>>
>>> #!/usr/bin/env python3
>>>
>>> when they are Python 3 only and
>>>
>>> #!/usr/bin/env python
>>>
>>> only when they support both Python 2 and 3.
>>>
>>> "Explicit is better than implicit" and all that Zen :-)
>>>
>>> Once that's done, switching the symlink is really a no-brainer.
>>>
>>> The recipe for this is easy too:
>>>
>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>> 2. migrate your scripts one by one to either Python 3.x or
>>>    to Python 2.7 + 3.4+
>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>>
>>> (*) Some OSes may require to use python2.7, if they don't come
>>> with a symlink from python2 -> python2.7.
>>>
>>> --
>>> Marc-Andre Lemburg
>>> eGenix.com
>>>
>>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>>> ________________________________________________________________________
>>>
>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>>
>>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>>                http://www.egenix.com/company/contact/
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/8070bf9f/attachment-0001.html>

From random832 at fastmail.us  Thu Mar 12 00:42:38 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 11 Mar 2015 19:42:38 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150311194325.GA15390@python.ca>
References: <20150311194325.GA15390@python.ca>
Message-ID: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>

On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
> Traditionally the OS only passes the first option to the
> interpreter so anything after that could be used to pass 3.x
> specific options.

I think some OSes pass everything after the interpreter as one long
string rather than ignoring extra arguments. Is there any standard for
this?

From ethan at stoneleaf.us  Thu Mar 12 00:43:27 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 11 Mar 2015 16:43:27 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CALGmxE+-D1+MHRvC+RzTcAF31GG-QW3Fw=At_fBEHNQfTHLONA@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
 <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
 <CALGmxE+-D1+MHRvC+RzTcAF31GG-QW3Fw=At_fBEHNQfTHLONA@mail.gmail.com>
Message-ID: <5500D31F.9080700@stoneleaf.us>

On 03/11/2015 04:06 PM, Chris Barker wrote:

> I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put
> ipython and ipython2
> 
> and in the python3 install, it would put:
> 
> ipython and ipython3
> 
> so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would
> determine what you'd get.

+1

Use the Path, Luke!  *ducks and runs*

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/c6689e53/attachment.sig>

From mal at egenix.com  Thu Mar 12 01:19:32 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 12 Mar 2015 01:19:32 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com>
References: <20150311194325.GA15390@python.ca>	<CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>	<5500A7AE.9020400@egenix.com>
 <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com>
Message-ID: <5500DB94.3020606@egenix.com>

On 11.03.2015 23:45, Andrew Barnert wrote:
> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> On 11.03.2015 21:03, David Mertz wrote:
>>> https://www.python.org/dev/peps/pep-0394/
>>
>> I think the migration should start with modifying scripts to use
>>
>> #!/usr/bin/env python2
>>
>> when they are Python 2 only (*) and
>>
>> #!/usr/bin/env python3
>>
>> when they are Python 3 only and
>>
>> #!/usr/bin/env python
>>
>> only when they support both Python 2 and 3.
> 
> Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".
> 
> So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.
> 
> In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)

I won't comment on the Mac Python installations. There's a reason
why we've been using our own installations since the early days of
Max OS X :-)

In any case, if those binaries don't exist on the target system,
the user will get a nice error message saying e.g.:

python2: Command not found.

and even better: modern OSes will then point the user to the
missing packages on their system to make the script run again.

I think that's a lot more informative than

SyntaxError: Missing parentheses in call to 'print'

:-)

Anyway, the above is just what I recommend.

I'm not a big fan of doing setuptools like magic all over the
place when there's a clean solution right in front of you (ok,
one may need to take off those cool Python monkey-patching and
meta-class sunglasses occasionally to see it ;-)).

>> "Explicit is better than implicit" and all that Zen :-)
>>
>> Once that's done, switching the symlink is really a no-brainer.
>>
>> The recipe for this is easy too:
>>
>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>> 2. migrate your scripts one by one to either Python 3.x or
>>   to Python 2.7 + 3.4+
>> 3. after migration replace "#!/usr/bin/env python2" with
>>   "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>
>> (*) Some OSes may require to use python2.7, if they don't come
>> with a symlink from python2 -> python2.7.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 12 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From steve at pearwood.info  Thu Mar 12 01:20:49 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Mar 2015 11:20:49 +1100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
Message-ID: <20150312002049.GC7655@ando.pearwood.info>

On Wed, Mar 11, 2015 at 06:12:00PM -0500, Ryan Gonzalez wrote:

> Please, just explicit major versions, not minor ones. python3, not
> python3.4. Just earlier this year I had to fix about 20 scripts that ran
> perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.

Funny about that, I had exactly the opposite experience: a bunch of 
scripts which worked fine with Python2.3 but failed with 2.6. (`raise 
"string"` was removed in 2.6.) Fortunately, the author of the scripts 
was an old-school Unix grey-beard who used the full path to the 
versioned interpreter, so I was able to install 2.3 and have them just 
work.

Fixing hashbang lines is much simpler than trying to fix the scripts 
themselves.


-- 
Steve

From abarnert at yahoo.com  Thu Mar 12 01:45:39 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 17:45:39 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CALGmxE+-D1+MHRvC+RzTcAF31GG-QW3Fw=At_fBEHNQfTHLONA@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CA+GyjM=Ew7uLGmJmS9TS9NbqoatuE+M5McoPU9NPDA+707VSgQ@mail.gmail.com>
 <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com>
 <CA+GyjMnV7EYkQ-mup_u64eT6AV_LxR69_T-LyWE0Fugx4e16Cg@mail.gmail.com>
 <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com>
 <CALGmxE+-D1+MHRvC+RzTcAF31GG-QW3Fw=At_fBEHNQfTHLONA@mail.gmail.com>
Message-ID: <84D7FEF9-E1DA-4878-99B3-CF26399CD218@yahoo.com>

On Mar 11, 2015, at 4:06 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
>> On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython".
> 
> hmm -- that seems odd -- if you re-arranged the path after it was installed, it would be different???

Yes. That's essentially why I said Unix doesn't naturally work that way. But it may be appropriate for this specific use case, even if it's a hack.

> I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put
> ipython and ipython2
> 
> and in the python3 install, it would put:
> 
> ipython and ipython3
> 
> so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would determine what you'd get.

That's exactly what we already have; the problem is when python2 and python3 have the same scripts directory--say, /usr/local/bin--they both install a script called /usr/local/bin/ipython, with whichever was installed last overwriting the one that was installed first.

You might argue that this should never come up, but the fact is that it does. When you install python2 and python3 packages from most package managers, they both go into /usr/local, and both install scripts to /usr/local/bin. And on many platforms, even if the system Python is in /usr, it installs scripts to /usr/local/bin (and that's the right thing to do if /usr/bin is only for system binaries or binaries owned by the system package manager).

So, Ian (I think it was him; it's hard to see from this pruned reply...) suggested that just "ipython" should be able to run whatever version of Python is just "python" rather than whichever was installed last. How could we make that work?

One option is to have the script #!/usr/bin/env python. But, while that works great for the case where you've installed ipython2 and ipython3 and want ipython to follow your python, the case Ian is trying to solve, it doesn't work an all for the case where you've installed only ipython3 and want to be able to access it as just plain ipython, which is an even more common case. (I do have 4 versions of ipython installed for different CPython and PyPy versions, but I only have one chardetect, etc., because I don't care which version of Python runs the script, I just care what the script does.

So, the installer hack would solve both problems, on every kind of platform, for every kind of script. But it's a pretty bad hack. Is it worth doing? I'm not sure. But it's worth thinking about why it would get what Ian wants, at least to see why other solutions won't.


> 
> -CHB
> 
> 
> 
>  
>>> ~ Ian Lee
>>> 
>>>> On Mar 11, 2015 3:29 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>>>>> On Mar 11, 2015, at 2:52 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>>>>> 
>>>>> +1
>>>>> 
>>>>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs).
>>>> 
>>>> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should?
>>>> 
>>>>> 
>>>>>> $ python2.7 -m pip install --user ipython
>>>>>>  
>>>>>> $ head -n1 ~/.local/bin/ipython*
>>>>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>>>>> #!/usr/bin/python2.7
>>>>>>  
>>>>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>>>>> #!/usr/bin/python2.7
>>>>>>  
>>>>>> $ python3.4 -m pip install --user ipython
>>>>>>  
>>>>>> $ head -n1 ~/.local/bin/ipython*
>>>>>> ==> /home/ianlee1521/.local/bin/ipython <==
>>>>>> #!/usr/bin/python3.4
>>>>>>  
>>>>>> ==> /home/ianlee1521/.local/bin/ipython2 <==
>>>>>> #!/usr/bin/python2.7
>>>>>>  
>>>>>> ==> /home/ianlee1521/.local/bin/ipython3 <==
>>>>>> #!/usr/bin/python3.4
>>>>> 
>>>>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above).
>>>>> 
>>>>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python".
>>>>> 
>>>>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package.
>>>>> 
>>>>> 
>>>>> ~ Ian Lee
>>>>> 
>>>>>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>>> On 11.03.2015 21:03, David Mertz wrote:
>>>>>> > https://www.python.org/dev/peps/pep-0394/
>>>>>> 
>>>>>> I think the migration should start with modifying scripts to use
>>>>>> 
>>>>>> #!/usr/bin/env python2
>>>>>> 
>>>>>> when they are Python 2 only (*) and
>>>>>> 
>>>>>> #!/usr/bin/env python3
>>>>>> 
>>>>>> when they are Python 3 only and
>>>>>> 
>>>>>> #!/usr/bin/env python
>>>>>> 
>>>>>> only when they support both Python 2 and 3.
>>>>>> 
>>>>>> "Explicit is better than implicit" and all that Zen :-)
>>>>>> 
>>>>>> Once that's done, switching the symlink is really a no-brainer.
>>>>>> 
>>>>>> The recipe for this is easy too:
>>>>>> 
>>>>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>>>>> 2. migrate your scripts one by one to either Python 3.x or
>>>>>>    to Python 2.7 + 3.4+
>>>>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>>>>    "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>>>>> 
>>>>>> (*) Some OSes may require to use python2.7, if they don't come
>>>>>> with a symlink from python2 -> python2.7.
>>>>>> 
>>>>>> --
>>>>>> Marc-Andre Lemburg
>>>>>> eGenix.com
>>>>>> 
>>>>>> Professional Python Services directly from the Source  (#1, Mar 11 2015)
>>>>>> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>>>> >>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>>>> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
>>>>>> ________________________________________________________________________
>>>>>> 
>>>>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>>>>>> 
>>>>>>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>>>>>>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>>>>>>            Registered at Amtsgericht Duesseldorf: HRB 46611
>>>>>>                http://www.egenix.com/company/contact/
>>>>>> _______________________________________________
>>>>>> Python-ideas mailing list
>>>>>> Python-ideas at python.org
>>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>> 
>>>>> _______________________________________________
>>>>> Python-ideas mailing list
>>>>> Python-ideas at python.org
>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/c112bb69/attachment-0001.html>

From skip.montanaro at gmail.com  Thu Mar 12 01:58:55 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 11 Mar 2015 19:58:55 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
Message-ID: <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>

On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>
> No; this won't work:
>
> #!/usr/bin/env python -x -3
>
> As Oleg said, anything in the hashbang can only be passed one argument.
It's not the kernel that invokes that; it's the shell.

What is to prevent Python itself from going back and reading that line?
Will the shell barf completely if there are two args? Or will it exec
.../env python?

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150311/8bb1d180/attachment.html>

From abarnert at yahoo.com  Thu Mar 12 01:59:32 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 17:59:32 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <5500DB94.3020606@egenix.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com>
 <5500DB94.3020606@egenix.com>
Message-ID: <9D539754-054B-498A-B095-B94744852071@yahoo.com>

On Mar 11, 2015, at 5:19 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 11.03.2015 23:45, Andrew Barnert wrote:
>>> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> 
>>>> On 11.03.2015 21:03, David Mertz wrote:
>>>> https://www.python.org/dev/peps/pep-0394/
>>> 
>>> I think the migration should start with modifying scripts to use
>>> 
>>> #!/usr/bin/env python2
>>> 
>>> when they are Python 2 only (*) and
>>> 
>>> #!/usr/bin/env python3
>>> 
>>> when they are Python 3 only and
>>> 
>>> #!/usr/bin/env python
>>> 
>>> only when they support both Python 2 and 3.
>> 
>> Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".
>> 
>> So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.
>> 
>> In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)
> 
> I won't comment on the Mac Python installations. There's a reason
> why we've been using our own installations since the early days of
> Max OS X :-)

In the early days of Mac OS X, Apple shipped an incomplete and broken version of a 4-year-old Python, so that made perfect sense. For half a decade, Apple has been shipping a complete version of the latest 2.x, and with things like PyObjC available out of the box. On top of that, when you tell people to use a separate Python 2.7 installation, they invariably run into problems where they install things for Apple's Python and then try to use them in the other Python or vice-versa, which is compounded by the fact that many people want to use /usr/local/bin as their script directory, but that's the same directory Apple uses, which is one of the ways you get to Ian's problem--whichever ipython was installed last wins--but of course here version numbers don't even help because they're both 2.7. There are hundreds of people on StackOverflow with this problem every year, and surely orders of magnitude more who don't go there for help.

The root problem is that PEP 394 only solves the problem of having two different versions of Python, not two different Pythons of the same version, but you're encouraging people to do the latter. And if PATH doesn't solve it for them (because they're using /usr/local), what does?

Also, if your program only works on Macs with an extra Python installation, then you don't work on the majority of Unix desktops out there, and it's hard to call that "portable". If you're not interested in working out of the box on most *nix systems, just require Python 3 and all these problems go away. :)

> In any case, if those binaries don't exist on the target system,
> the user will get a nice error message saying e.g.:
> 
> python2: Command not found.
> 
> and even better: modern OSes will then point the user to the
> missing packages on their system to make the script run again.

So nothing other than some of the linux distros counts as a modern OS?

> I think that's a lot more informative than
> 
> SyntaxError: Missing parentheses in call to 'print'
> 
> :-)
> 
> Anyway, the above is just what I recommend.
> 
> I'm not a big fan of doing setuptools like magic all over the
> place when there's a clean solution right in front of you (ok,
> one may need to take off those cool Python monkey-patching and
> meta-class sunglasses occasionally to see it ;-)).
> 
>>> "Explicit is better than implicit" and all that Zen :-)
>>> 
>>> Once that's done, switching the symlink is really a no-brainer.
>>> 
>>> The recipe for this is easy too:
>>> 
>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>> 2. migrate your scripts one by one to either Python 3.x or
>>>  to Python 2.7 + 3.4+
>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>  "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>> 
>>> (*) Some OSes may require to use python2.7, if they don't come
>>> with a symlink from python2 -> python2.7.
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Mar 12 2015)
>>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/

From mal at egenix.com  Thu Mar 12 02:30:29 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 12 Mar 2015 02:30:29 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <9D539754-054B-498A-B095-B94744852071@yahoo.com>
References: <20150311194325.GA15390@python.ca>	<CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>	<5500A7AE.9020400@egenix.com>	<3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com>	<5500DB94.3020606@egenix.com>
 <9D539754-054B-498A-B095-B94744852071@yahoo.com>
Message-ID: <5500EC35.40105@egenix.com>

On 12.03.2015 01:59, Andrew Barnert wrote:
> On Mar 11, 2015, at 5:19 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> On 11.03.2015 23:45, Andrew Barnert wrote:
>>>> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>
>>>>> On 11.03.2015 21:03, David Mertz wrote:
>>>>> https://www.python.org/dev/peps/pep-0394/
>>>>
>>>> I think the migration should start with modifying scripts to use
>>>>
>>>> #!/usr/bin/env python2
>>>>
>>>> when they are Python 2 only (*) and
>>>>
>>>> #!/usr/bin/env python3
>>>>
>>>> when they are Python 3 only and
>>>>
>>>> #!/usr/bin/env python
>>>>
>>>> only when they support both Python 2 and 3.
>>>
>>> Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems".
>>>
>>> So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply.
>>>
>>> In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?)
>>
>> I won't comment on the Mac Python installations. There's a reason
>> why we've been using our own installations since the early days of
>> Max OS X :-)
> 
> In the early days of Mac OS X, Apple shipped an incomplete and broken version of a 4-year-old Python, so that made perfect sense. For half a decade, Apple has been shipping a complete version of the latest 2.x, and with things like PyObjC available out of the box. On top of that, when you tell people to use a separate Python 2.7 installation, they invariably run into problems where they install things for Apple's Python and then try to use them in the other Python or vice-versa, which is compounded by the fact that many people want to use /usr/local/bin as their script directory, but that's the same directory Apple uses, which is one of the ways you get to Ian's problem--whichever ipython was installed last wins--but of course here version numbers don't even help because they're both 2.7. There are hundreds of people on StackOverflow with this problem every year, and surely orders of magnitude more who don't go there for help.

As I said, we've been using our own installations. That doesn't
mean the resulting packages won't run with the stock Apple
versions, it just means that we're not relying on those Apple
versions when compiling our software.

> The root problem is that PEP 394 only solves the problem of having two different versions of Python, not two different Pythons of the same version, but you're encouraging people to do the latter. And if PATH doesn't solve it for them (because they're using /usr/local), what does?

This thread is not trying to solve the problem of having
multiple Python installations of the same version. The subject problem
is migration to Python 3, not running multiple installations
of Python 2.7. Let's not get side tracked (I know it's python-ideas
and side tracking is fun, but still :-)).

> Also, if your program only works on Macs with an extra Python installation, then you don't work on the majority of Unix desktops out there, and it's hard to call that "portable". If you're not interested in working out of the box on most *nix systems, just require Python 3 and all these problems go away. :)

Actually, we don't work much on Macs, but mostly on Linux,
FreeBSD and also on Windows, where you have a different set of problems
than what's being discussed here, but which can be solved in a
similar way by the way of the py launcher:

https://www.python.org/dev/peps/pep-0397/

With the right py.ini file, the above approach will work on Windows
as well.

>> In any case, if those binaries don't exist on the target system,
>> the user will get a nice error message saying e.g.:
>>
>> python2: Command not found.
>>
>> and even better: modern OSes will then point the user to the
>> missing packages on their system to make the script run again.
> 
> So nothing other than some of the linux distros counts as a modern OS?

Of course they do. This was just a tangent I couldn't resist,
since I think that more modern OSes should provide the same usability
features. It's one of the nice Linux distro features I'd love to
see in other OSes as well.

But this was not the main argument. I think that even the simple

python2: Command not found.

is better than the standard Python SyntaxError (or other
similar Python 3 vs. 2 related error) you get when running
a script with the wrong Python version.

>> I think that's a lot more informative than
>>
>> SyntaxError: Missing parentheses in call to 'print'
>>
>> :-)
>>
>> Anyway, the above is just what I recommend.
>>
>> I'm not a big fan of doing setuptools like magic all over the
>> place when there's a clean solution right in front of you (ok,
>> one may need to take off those cool Python monkey-patching and
>> meta-class sunglasses occasionally to see it ;-)).
>>
>>>> "Explicit is better than implicit" and all that Zen :-)
>>>>
>>>> Once that's done, switching the symlink is really a no-brainer.
>>>>
>>>> The recipe for this is easy too:
>>>>
>>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2"
>>>> 2. migrate your scripts one by one to either Python 3.x or
>>>>  to Python 2.7 + 3.4+
>>>> 3. after migration replace "#!/usr/bin/env python2" with
>>>>  "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp.
>>>>
>>>> (*) Some OSes may require to use python2.7, if they don't come
>>>> with a symlink from python2 -> python2.7.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 12 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From abarnert at yahoo.com  Thu Mar 12 02:39:11 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 11 Mar 2015 18:39:11 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
Message-ID: <4402E693-1F8A-4865-BD6B-F5F3BE60417B@yahoo.com>

On Mar 11, 2015, at 4:42 PM, random832 at fastmail.us wrote:
> 
>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
>> Traditionally the OS only passes the first option to the
>> interpreter so anything after that could be used to pass 3.x
>> specific options.
> 
> I think some OSes pass everything after the interpreter as one long
> string rather than ignoring extra arguments.

And some pass all the arguments separately. Try "#! /usr/bin/env python -O -i" on different platforms; you'll find all three behaviors.

> Is there any standard for
> this?

Nope. POSIX allows for compliant sh implementations to do shbang processing by saying that 'If the first line of a file of shell commands starts with the characters "#!", the results are unspecified". That means there's actually no way to write portable #! scripts according to POSIX. Given that POSIX doesn't require any particular path to env, that's not too surprising; you're already non-portable using /usr/bin/env

At least (as of 2008) POSIX does ban doing shbang processing inside the exec functions instead of in sh. (Although even there, Linux's pluggable executable formats provide a way around that...) Somewhere in the POSIX standard, there's a non-normative recommendation that if you want an application to depend on #!, you should include an installer that uses getconf PATH, searches for a suitable interpreter, and used sed to change the shbang to point at the absolute path to that interpreter.

So, forget theory; in practice, what's portable? Using exactly one argument works as expected on any system you're likely to find. Which means if you want to use /usr/bin/env instead of using an installer, you don't get any arguments.


From phd at phdru.name  Thu Mar 12 09:24:01 2015
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 12 Mar 2015 09:24:01 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
Message-ID: <20150312082401.GA12046@phdru.name>

Hi!

On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote:
> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
> > Traditionally the OS only passes the first option to the
> > interpreter so anything after that could be used to pass 3.x
> > specific options.
> 
> I think some OSes pass everything after the interpreter as one long
> string rather than ignoring extra arguments. Is there any standard for
> this?

   There *are* standards ;-) See
http://www.in-ulm.de/~mascheck/various/shebang/#splitting
   and the table below.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From phd at phdru.name  Thu Mar 12 09:25:43 2015
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 12 Mar 2015 09:25:43 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
Message-ID: <20150312082543.GB12046@phdru.name>

Hi!

On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
> >
> > No; this won't work:
> >
> > #!/usr/bin/env python -x -3
> >
> > As Oleg said, anything in the hashbang can only be passed one argument.
> It's not the kernel that invokes that; it's the shell.

   Why do you think it's the shell? It's exec() system call.

> Skip

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From solipsis at pitrou.net  Thu Mar 12 10:25:03 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Mar 2015 10:25:03 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
Message-ID: <20150312102503.02b22f00@fsol>

On Wed, 11 Mar 2015 13:43:25 -0600
Neil Schemenauer <nas-python at arctrix.com>
wrote:
> This has been brought up elsewhere, I think this is a better forum
> to discuss it.  Changing /usr/bin/python directly to python3.x is
> bad idea, in my option.  It's going to cause users trouble and the
> benefit to Python 3.x users is not worth it.  Instead, let's try to
> find a smooth migration path.

As a data point, when you install Python 3 using Conda, the executable
is already called "python":

$ python
Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 21 2014,
17:16:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 


Regards

Antoine.



From abarnert at yahoo.com  Thu Mar 12 11:17:47 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 03:17:47 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312082401.GA12046@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
 <20150312082401.GA12046@phdru.name>
Message-ID: <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com>

On Mar 12, 2015, at 1:24 AM, Oleg Broytman <phd at phdru.name> wrote:
> 
> Hi!
> 
>> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote:
>>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
>>> Traditionally the OS only passes the first option to the
>>> interpreter so anything after that could be used to pass 3.x
>>> specific options.
>> 
>> I think some OSes pass everything after the interpreter as one long
>> string rather than ignoring extra arguments. Is there any standard for
>> this?
> 
>   There *are* standards ;-) See
> http://www.in-ulm.de/~mascheck/various/shebang/#splitting
>   and the table below.

A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard. 

> 
> Oleg.
> -- 
>     Oleg Broytman            http://phdru.name/            phd at phdru.name
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From phd at phdru.name  Thu Mar 12 11:23:54 2015
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 12 Mar 2015 11:23:54 +0100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
 <20150312082401.GA12046@phdru.name>
 <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com>
Message-ID: <20150312102354.GA17548@phdru.name>

On Thu, Mar 12, 2015 at 03:17:47AM -0700, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Mar 12, 2015, at 1:24 AM, Oleg Broytman <phd at phdru.name> wrote:
> >> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote:
> >>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
> >>> Traditionally the OS only passes the first option to the
> >>> interpreter so anything after that could be used to pass 3.x
> >>> specific options.
> >> 
> >> I think some OSes pass everything after the interpreter as one long
> >> string rather than ignoring extra arguments. Is there any standard for
> >> this?
> > 
> >   There *are* standards ;-) See
> > http://www.in-ulm.de/~mascheck/various/shebang/#splitting
> >   and the table below.
> 
> A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard.

   Isn't that how all standards are implemented in real life?! ;-)

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From abarnert at yahoo.com  Thu Mar 12 11:29:59 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 03:29:59 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312082543.GB12046@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
Message-ID: <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>

On Mar 12, 2015, at 1:25 AM, Oleg Broytman <phd at phdru.name> wrote:
> 
> Hi!
> 
>> On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro <skip.montanaro at gmail.com> wrote:
>>> On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>>> 
>>> No; this won't work:
>>> 
>>> #!/usr/bin/env python -x -3
>>> 
>>> As Oleg said, anything in the hashbang can only be passed one argument.
>> It's not the kernel that invokes that; it's the shell.
> 
>   Why do you think it's the shell? It's exec() system call.

Maybe he thinks that because he read the POSIX standards, which forbid doing it in exec, or the man pages for his system's execl or sh?

The earliest versions of Unix shbang processing were done in exec, but that turned out to lead to a variety of problems, so most stopped, and after a few decades, POSIX finally (in 2008) forbade it: exec must use the standard shell processor for anything that's not a recognized, loadable executable format. The standard shell processor (usually /bin/sh) is then allowed to do anything it wants for files starting with "#!" (the behavior is left undefined by the standard); on pretty much all systems what it does is (one of the three variations of) the usual shbang process.

(As I briefly mentioned earlier, there is a way to weasel around the wording, and Linux does give you a way to use it: the linux kernel has pluggable executable formats, and there's nothing stopping you from, say, defining an executable format that's recognized by treating "#!/usr/bin/env python" as a magic number and "loaded" by exec'ing the Python interpreter, which would still be within the letter of the standard.)

> 
>> Skip
> 
> Oleg.
> -- 
>     Oleg Broytman            http://phdru.name/            phd at phdru.name
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Thu Mar 12 11:42:51 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 03:42:51 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312102354.GA17548@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
 <20150312082401.GA12046@phdru.name>
 <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com>
 <20150312102354.GA17548@phdru.name>
Message-ID: <D73D5464-3559-410C-87EF-1FC1B8AA4308@yahoo.com>

On Mar 12, 2015, at 3:23 AM, Oleg Broytman <phd at phdru.name> wrote:
> 
>> On Thu, Mar 12, 2015 at 03:17:47AM -0700, Andrew Barnert <abarnert at yahoo.com> wrote:
>> On Mar 12, 2015, at 1:24 AM, Oleg Broytman <phd at phdru.name> wrote:
>>>>> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote:
>>>>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote:
>>>>> Traditionally the OS only passes the first option to the
>>>>> interpreter so anything after that could be used to pass 3.x
>>>>> specific options.
>>>> 
>>>> I think some OSes pass everything after the interpreter as one long
>>>> string rather than ignoring extra arguments. Is there any standard for
>>>> this?
>>> 
>>>  There *are* standards ;-) See
>>> http://www.in-ulm.de/~mascheck/various/shebang/#splitting
>>>  and the table below.
>> 
>> A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard.
> 
>   Isn't that how all standards are implemented in real life?! ;-)

Sure, there are cases where there is a standard but nobody follows it, but that's not the case here: there is no standard to follow in the first place. The relevant standard document (POSIX 2008 or 2013, for most *nixes) explicitly says the behavior is not defined by the standard.

(And practically, that makes even more of a difference than it should, as various C libs have at times _intentionally_ done things differently when the standard leaves it open, for good and bad reasons ranging from "so people don't rely on 'it works on Linux and FreeBSD, so it must be standard and portable, so I won't look it up'" to "screw those BSD people for implying something is portable without checking with us".)

> -- 
>     Oleg Broytman            http://phdru.name/            phd at phdru.name
>           Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From guettliml at thomas-guettler.de  Thu Mar 12 12:56:58 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Thu, 12 Mar 2015 12:56:58 +0100
Subject: [Python-ideas] datetime: Support infinity
Message-ID: <55017F0A.4040109@thomas-guettler.de>

Hi,

at the end of january there was a mail thread about "datetime: Support infinity".

Some people liked it :-)

What can I do to get this done?

Regards,
   Thomas

From skip.montanaro at gmail.com  Thu Mar 12 14:21:33 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 12 Mar 2015 08:21:33 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312082543.GB12046@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
Message-ID: <CANc-5UwYaXMKsLdEX=92=eGKhtJ4Vju9bz413XZC9beOMpU0xg@mail.gmail.com>

I think the quoting got screwed up somewhere. I wrote:

skip> If I understood correctly, /usr/bin/python would be started by
the kernel, and it would be passed the -x flag.

to which Ryan replied:

ryan> It's not the kernel that invokes that; it's the shell.

then Oleg responded (but it looked like he was quoting me, not Ryan):

oleg> Why do you think it's the shell? It's exec() system call.

I ignored Ryan's original reply, since it's been so long that I
considered things at this level, I thought I might have been mistaken
or that the field had changed in the meantime. So, are we all on the
same page now? The shell calls exec(...whatever...), and within that
call (in the kernel), the #! line is interpreted. There is still some
question about whether it consumes the first two "words" after the #!
(command and one argument), or in some cases consumes all words,
treating them as separate arguments (seems feasible, though
problematic from a security POV) or as one big argument (seems
unlikely, as that would foster an enormous amount of breakage).

Skip

From skip.montanaro at gmail.com  Thu Mar 12 14:23:11 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 12 Mar 2015 08:23:11 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CANc-5UwYaXMKsLdEX=92=eGKhtJ4Vju9bz413XZC9beOMpU0xg@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <CANc-5UwYaXMKsLdEX=92=eGKhtJ4Vju9bz413XZC9beOMpU0xg@mail.gmail.com>
Message-ID: <CANc-5UwTFfFAnH5FGy_abCOoqG0=19vD6_df024_eeH-v-mFDQ@mail.gmail.com>

On Thu, Mar 12, 2015 at 8:21 AM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> ... I thought I might have been mistaken
> or that the field had changed in the meantime.

Which, I see by reading the rest of the thread, it has...

S

From wes.turner at gmail.com  Thu Mar 12 14:55:15 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 12 Mar 2015 08:55:15 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312102503.02b22f00@fsol>
References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol>
Message-ID: <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>

On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Wed, 11 Mar 2015 13:43:25 -0600
> Neil Schemenauer <nas-python at arctrix.com>
> wrote:
> > This has been brought up elsewhere, I think this is a better forum
> > to discuss it.  Changing /usr/bin/python directly to python3.x is
> > bad idea, in my option.  It's going to cause users trouble and the
> > benefit to Python 3.x users is not worth it.  Instead, let's try to
> > find a smooth migration path.
>
> As a data point, when you install Python 3 using Conda, the executable
> is already called "python":
>
> $ python

Brew and pyenv also rely upon $PATH to determine which executable named
'python[maj.min]' to run a script marked as executable with.

+1 for the /usr/bin/env python[maj[.min]] approach is likely most portable
(because it uses $PATH for what it is for)

* OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!))
* OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent)
* I think it would be reasonable to request an update that adds a
'/usr/bin/python2' symlink, so that paths can be '/usr/bin/env
python[maj[.min]'.
* They seem more focused on (!OpenStack) Swift lately.

[1] https://www.apple.com/opensource/

Again, +1 for the /usr/bin/env python[maj[.min]] approach (because it uses
$PATH for what it is for)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/42e78241/attachment.html>

From random832 at fastmail.us  Thu Mar 12 16:43:36 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 12 Mar 2015 11:43:36 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312082401.GA12046@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com>
 <20150312082401.GA12046@phdru.name>
Message-ID: <1426175016.2903413.239449765.7FD189FA@webmail.messagingengine.com>

On Thu, Mar 12, 2015, at 04:24, Oleg Broytman wrote:
> 
>    There *are* standards ;-) See
> http://www.in-ulm.de/~mascheck/various/shebang/#splitting
>    and the table below.
> 
> Oleg.

One additional data point of purely historical interest is that DEMOS (a
soviet unix clone derived from 3BSD but the functionality here is
apparently its unique innovation) supports "#!CMD A1 $* A2 A3" which
does more or less what it looks like it does.

It also supports "/*#!" as the magic prefix.

From random832 at fastmail.us  Thu Mar 12 16:51:06 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 12 Mar 2015 11:51:06 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
Message-ID: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>

On Thu, Mar 12, 2015, at 06:29, Andrew Barnert wrote:
> The earliest versions of Unix shbang processing were done in exec, but
> that turned out to lead to a variety of problems, so most stopped, and
> after a few decades, POSIX finally (in 2008) forbade it: exec must use
> the standard shell processor for anything that's not a recognized,
> loadable executable format.

Show me where POSIX defines "recognized, loadable executable format" to
exclude #! as a magic number with its usual behavior.

What the standard requires is that execvp and execlp treat what would be
a failure of the other members of the exec family of functions (i.e.
execve, execle, execv, execl), with [ENOEXEC] by falling back to sh.
However, shebang does not cause these other functions to fail with
[ENOEXEC], and therefore does not fall under this clause.

It does not forbid any processing from being done "in exec" (i.e. in
execve).

From random832 at fastmail.us  Thu Mar 12 16:58:22 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 12 Mar 2015 11:58:22 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
Message-ID: <1426175902.2908116.239457549.634FE590@webmail.messagingengine.com>

On Thu, Mar 12, 2015, at 11:51, random832 at fastmail.us wrote:
> It does not forbid any processing from being done "in exec" (i.e. in
> execve).

Just for completeness, I will note that it _probably_ does forbid exec
from returning [ENOENT], [EACCES], etc, on the basis of problems
executing the interpreter specified in the shbang line, and
implementations should simply return [EINVAL] in that case. However,
that's quite different from the processing being forbidden entirely.

From rymg19 at gmail.com  Thu Mar 12 18:50:53 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 12 Mar 2015 12:50:53 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312082543.GB12046@phdru.name>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
Message-ID: <CAO41-mN5FALjQc-9sAqJrzA7Ng_D2wKomfgiWAYTt9CPoii87w@mail.gmail.com>

On Thu, Mar 12, 2015 at 3:25 AM, Oleg Broytman <phd at phdru.name> wrote:

> Hi!
>
> On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro <
> skip.montanaro at gmail.com> wrote:
> > On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
> > >
> > > No; this won't work:
> > >
> > > #!/usr/bin/env python -x -3
> > >
> > > As Oleg said, anything in the hashbang can only be passed one argument.
> > It's not the kernel that invokes that; it's the shell.
>
>    Why do you think it's the shell? It's exec() system call.
>
>
Right. My bad. I was thinking about the parsing part, not the execution
part.


> > Skip
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            phd at phdru.name
>            Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/a7a3bd38/attachment.html>

From abarnert at yahoo.com  Thu Mar 12 21:38:01 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 13:38:01 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
Message-ID: <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>

> On Mar 12, 2015, at 8:51 AM, random832 at fastmail.us wrote:
> 
> Show me where POSIX defines "recognized, loadable executable format" to
> exclude #! as a magic number with its usual behavior.

I already quoted the standard earlier in the thread, and I already explained how something like linux's pluggable binfmt provides a way around it that's still at least according to the letter of the standard. I don't see much point in repeating what I already wrote. But notice that in the RATIONALE section of the 2001, 2004, 2008, and 2013 versions of the standard (e.g., http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html) effectively describes that as legacy behavior in contrast to the behavior that's "now required by IEEE Std 1003.1-2001":

> Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute.

At any rate, this is getting way off track. The point here is that if you're looking for a standard on what #! does everywhere, you're not going to find one. The working group on executable scripts for POSIX-2001 eventually decided to add nothing more than the statement that scripts that start with #! have behavior undefined by the standard, and the recommendation apps that want to use #! should include an installer that sed's the script based on what's on the PATH at install time. Which makes the standard useless if you want to write portable shbang scripts.

But practicality beats purity. If all of the *nix platforms that Python currently supports do shbang processing, all have env in /usr/bin (also specifically not required by POSIX, which explicitly says that path locations are not determined by the standard), and none of them swallow all args (as early BSD did), but some drop all args after the first, some pass all, and some pass a single arg with spaces, then that's what you have to work with. You can use "#!/usr/bin/env python", but you can't use "#!/usr/bin/env python -m ipython".

Of course if you're building a Mac installer or packages for a particular linux distro or the like, you don't have to worry about portability at all, and can do what makes sense for your platform. (This also means not having to worry about other distros that provide a Python 3.x as /usr/bin/python, etc.)

From random832 at fastmail.us  Thu Mar 12 21:49:36 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 12 Mar 2015 16:49:36 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
 <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>
Message-ID: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>

On Thu, Mar 12, 2015, at 16:38, Andrew Barnert wrote:
> effectively describes that as legacy behavior in contrast to the behavior
> that's "now required by IEEE Std 1003.1-2001":

The legacy behavior being discussed is that of _requiring_ it and
_failing_ to execute the shell for files that didn't have it.

There's nothing forbidding an implementation from having a magic number
of "#!" as a standard non-"pluggable" executable format.

> > Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute.
> 
> At any rate, this is getting way off track.

You have a point - sorry about that.

Though, speaking of the posix shell script exec issue, and to wrench
things back on topic... what about allowing python scripts to start with
the line "exec python [options...] $0", a line which will be ignored by
the python interpreter?

From abarnert at yahoo.com  Thu Mar 12 22:05:16 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 14:05:16 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
 <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>
 <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>
Message-ID: <3C2D8DFB-3F09-483D-B8E2-5F1D1291215D@yahoo.com>

On Mar 12, 2015, at 1:49 PM, random832 at fastmail.us wrote:

> Though, speaking of the posix shell script exec issue, and to wrench
> things back on topic... what about allowing python scripts to start with
> the line "exec python [options...] $0", a line which will be ignored by
> the python interpreter?

I vaguely remember either perl or Tcl considering something equivalent long ago as a way of dealing with BSD4.x versions that didn't pass any arguments from the shbang line (before that became irrelevant but env became a de facto practical standard and ate up the first arg, putting us right back in the equivalent place), so it might be worth searching their archives to see what came of that. (Although given the languages we're talking about, I wouldn't be surprised if the answer was "you can already do it in one line in these six different hacky ways, so we don't need to add a special way to do it", which won't apply to Python...)

However, I'm not sure this solves the problem the thread is trying to solve. Yes, it gives you a portable way to let 3.5+ scripts pass extra arguments to the interpreter, which might be useful for other reasons (and if so, I think a new thread laying out the rationale would be handy). But the problem people were trying to solve here is that `#!/usr/bin/env python2 -m spam` isn't portable (and neither is `#!/usr/local/bin/spam` where spam is a Python script), and you can't retroactively change 2.x.

From p.f.moore at gmail.com  Thu Mar 12 22:33:35 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 12 Mar 2015 21:33:35 +0000
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
 <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>
 <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>
Message-ID: <CACac1F_L=gwcPKRcQd4eKmhMrcJk+AxbcfxQQVkgYZ0tx=5T4g@mail.gmail.com>

On 12 March 2015 at 20:49,  <random832 at fastmail.us> wrote:
> Though, speaking of the posix shell script exec issue, and to wrench
> things back on topic... what about allowing python scripts to start with
> the line "exec python [options...] $0", a line which will be ignored by
> the python interpreter?

Can't you use the -x option (skip first line of source) for that?
Paul

From abarnert at yahoo.com  Thu Mar 12 22:34:49 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 14:34:49 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>
References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol>
 <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>
Message-ID: <2D3F4998-7AF3-4FAF-96CF-9A6E733C7067@yahoo.com>

On Mar 12, 2015, at 6:55 AM, Wes Turner <wes.turner at gmail.com> wrote:
> 
> 
> On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> >
> > On Wed, 11 Mar 2015 13:43:25 -0600
> > Neil Schemenauer <nas-python at arctrix.com>
> > wrote:
> > > This has been brought up elsewhere, I think this is a better forum
> > > to discuss it.  Changing /usr/bin/python directly to python3.x is
> > > bad idea, in my option.  It's going to cause users trouble and the
> > > benefit to Python 3.x users is not worth it.  Instead, let's try to
> > > find a smooth migration path.
> >
> > As a data point, when you install Python 3 using Conda, the executable
> > is already called "python":
> >
> > $ python
> 
> Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with.
> 
> +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for)
> 
> * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!))
> * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent)
> 
2.7.6 on both my base 10.10 box and my 10.10.3 prerelease. (I think it's no longer violating NDA to answer questions about prereleases now that they're public betas rather than DRs... If I'm wrong, pretend I didn't say anything and just consider that they've only changed the Python version in a dot release once in the history of OS X, and that was before they got semi-serious about Python...)
> * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'.
> 
Is there someone who can make that request who Apple will listen to more carefully than just a random user's radar? Or should we all just go file radars and link them on OpenRadar for tracking?

Also, is there any change of getting it added to 10.9 and 10.10 bug fix releases?

Anyway, this would get us a step closer, but as long as there are millions more OS X 10.8, RHEL5, etc. boxes out there (that have python but not python2) than Arch boxes (where python means 3.x); 2.x-only app's still can't portably follow what PEP 394 recommends.
> * They seem more focused on (!OpenStack) Swift lately.
> 
It's not as if Python was ever their top priority in the first place. But they've clearly got at least someone working on Python integration given that they included a newer Python 2.7 (and things like newer PyObjC, newer and better-optimized NumPy, PyOpenSSL linked to the same dylib as Python itself, etc.) in each new major OS release, so it's not like there's nobody who could add the symlink...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/ef71eab7/attachment.html>

From random832 at fastmail.us  Thu Mar 12 22:49:42 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Thu, 12 Mar 2015 17:49:42 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CACac1F_L=gwcPKRcQd4eKmhMrcJk+AxbcfxQQVkgYZ0tx=5T4g@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAO41-mOfWJZK+MMLyAqZJufBkw03=jfbw=2NAW8Om+G9JFyQCQ@mail.gmail.com>
 <CANc-5Uwd8=TQHUco-_dDA06075xT3ocNZbS7X0njZGOCqZgXGw@mail.gmail.com>
 <CAO41-mM4XSf+YMP9Hx7_+qMNr=36Aa5x120nWg+sj3QmC1=53w@mail.gmail.com>
 <CANc-5UwvPA0eLqzxH9=Vf7W1SjA1Oknn6OgArDgtDqd58pMA9Q@mail.gmail.com>
 <20150312082543.GB12046@phdru.name>
 <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com>
 <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com>
 <F0F886DD-2F70-4817-95F7-B7C83CDE611F@yahoo.com>
 <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com>
 <CACac1F_L=gwcPKRcQd4eKmhMrcJk+AxbcfxQQVkgYZ0tx=5T4g@mail.gmail.com>
Message-ID: <1426196982.2999114.239678065.6219273A@webmail.messagingengine.com>

On Thu, Mar 12, 2015, at 17:33, Paul Moore wrote:
> On 12 March 2015 at 20:49,  <random832 at fastmail.us> wrote:
> > Though, speaking of the posix shell script exec issue, and to wrench
> > things back on topic... what about allowing python scripts to start with
> > the line "exec python [options...] $0", a line which will be ignored by
> > the python interpreter?
> 
> Can't you use the -x option (skip first line of source) for that?
> Paul

I'd guess the "DOS specific hack" it is intended for is, in fact,
precisely "run the python interpreter using the native scripting
language in the first line". So... I guess you could.

Might be worth cleaning up the line number issue and replacing the env
hack with it going forward, given, you know, POSIX compliance.

From barry at python.org  Thu Mar 12 22:52:20 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 12 Mar 2015 17:52:20 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
Message-ID: <20150312175220.6f125d03@anarchist.wooz.org>

On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:

>I think the migration should start with modifying scripts to use
>
>#!/usr/bin/env python2
>
>when they are Python 2 only (*) and
>
>#!/usr/bin/env python3
>
>when they are Python 3 only and
>
>#!/usr/bin/env python
>
>only when they support both Python 2 and 3.

Mostly, I agree, although I would not recommend /usr/bin/env for system
installed scripts.  Using /usr/bin/env is fine for development trees, but a
really bad idea for anything installed in system locations.  Modify your
suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)

I would very much like to see explicit use of /usr/bin/python2 replace
/usr/bin/python for scripts.

Then I think the problem is, what do users expect when they type

$ python foo.py

at their shell prompt?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/31800003/attachment.sig>

From abarnert at yahoo.com  Thu Mar 12 23:00:47 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 12 Mar 2015 15:00:47 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>
References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol>
 <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>
Message-ID: <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com>

On Mar 12, 2015, at 6:55 AM, Wes Turner <wes.turner at gmail.com> wrote:
> On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> >
> > On Wed, 11 Mar 2015 13:43:25 -0600
> > Neil Schemenauer <nas-python at arctrix.com>
> > wrote:
> > > This has been brought up elsewhere, I think this is a better forum
> > > to discuss it.  Changing /usr/bin/python directly to python3.x is
> > > bad idea, in my option.  It's going to cause users trouble and the
> > > benefit to Python 3.x users is not worth it.  Instead, let's try to
> > > find a smooth migration path.
> >
> > As a data point, when you install Python 3 using Conda, the executable
> > is already called "python":
> >
> > $ python
> 
> Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with.
> 
> +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for)
> 
> * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!))
> * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent)
> * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'.
> 
http://openradar.appspot.com/radar?id=5833516721897472

If you want to file your own bug report with Apple, you can mention that it's a dup of radar 20143311.

> * They seem more focused on (!OpenStack) Swift lately.
> 
> [1] https://www.apple.com/opensource/
> 
> Again, +1 for the /usr/bin/env python[maj[.min]] approach (because it uses $PATH for what it is for)
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/ee7d349d/attachment-0001.html>

From wes.turner at gmail.com  Thu Mar 12 23:01:13 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 12 Mar 2015 17:01:13 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312175220.6f125d03@anarchist.wooz.org>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <20150312175220.6f125d03@anarchist.wooz.org>
Message-ID: <CACfEFw_9NSwmcFaT=SjYAMzzkqnzoovmQzfPsB_oH0kK=m-YdQ@mail.gmail.com>

On Mar 12, 2015 4:53 PM, "Barry Warsaw" <barry at python.org> wrote:
>
> On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:
>
> >I think the migration should start with modifying scripts to use
> >
> >#!/usr/bin/env python2
> >
> >when they are Python 2 only (*) and
> >
> >#!/usr/bin/env python3
> >
> >when they are Python 3 only and
> >
> >#!/usr/bin/env python
> >
> >only when they support both Python 2 and 3.
>
> Mostly, I agree, although I would not recommend /usr/bin/env for system
> installed scripts.  Using /usr/bin/env is fine for development trees, but
a
> really bad idea for anything installed in system locations.  Modify your
> suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)

For (re-)packaged scripts, it seems reasonable to expect package
maintainers to use explicit platform-specific paths.

Otherwise, for checksums and diff noise, does /usr/bin/env python2 make
sense?

>
> I would very much like to see explicit use of /usr/bin/python2 replace
> /usr/bin/python for scripts.

+1. This will require documented guidelines and many emails to package
maintainers.

Where should a note about this go in
/docs?

>
> Then I think the problem is, what do users expect when they type
>
> $ python foo.py
>
> at their shell prompt?

For the first python binary on $PATH to supersede whatever might be defined
in the shebang line of foo.py (thus skipping any shebang path and args).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/a97a91ca/attachment.html>

From wes.turner at gmail.com  Thu Mar 12 23:04:30 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Thu, 12 Mar 2015 17:04:30 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com>
References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol>
 <CACfEFw_E_f9tAi+JyibL6gFBzZEBms-Bwjh_c10HPd=XKMWAjw@mail.gmail.com>
 <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com>
Message-ID: <CACfEFw8ivccO4HfRt-wv3y-DhPudn9SQ6W1ghBj3skQhnL9B+g@mail.gmail.com>

On Mar 12, 2015 5:00 PM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>
> On Mar 12, 2015, at 6:55 AM, Wes Turner <wes.turner at gmail.com> wrote:
>>
>> On Mar 12, 2015 4:25 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>> >
>> > On Wed, 11 Mar 2015 13:43:25 -0600
>> > Neil Schemenauer <nas-python at arctrix.com>
>> > wrote:
>> > > This has been brought up elsewhere, I think this is a better forum
>> > > to discuss it.  Changing /usr/bin/python directly to python3.x is
>> > > bad idea, in my option.  It's going to cause users trouble and the
>> > > benefit to Python 3.x users is not worth it.  Instead, let's try to
>> > > find a smooth migration path.
>> >
>> > As a data point, when you install Python 3 using Conda, the executable
>> > is already called "python":
>> >
>> > $ python
>>
>> Brew and pyenv also rely upon $PATH to determine which executable named
'python[maj.min]' to run a script marked as executable with.
>>
>> +1 for the /usr/bin/env python[maj[.min]] approach is likely most
portable (because it uses $PATH for what it is for)
>>
>> * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!))
>> * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent)
>> * I think it would be reasonable to request an update that adds a
'/usr/bin/python2' symlink, so that paths can be '/usr/bin/env
python[maj[.min]'.
>
> http://openradar.appspot.com/radar?id=5833516721897472
>
> If you want to file your own bug report with Apple, you can mention that
it's a dup of radar 20143311.

I sent an email to product-security AT apple.com (
https://www.apple.com/support/security/) with a link to this thread;
because TLS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/1c30e162/attachment.html>

From barry at python.org  Fri Mar 13 02:40:42 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 12 Mar 2015 21:40:42 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CAO41-mOypBaYPcmcm2+bdU1t0s4Nx00-Fh7eV9Fk1tFTeQbwuw@mail.gmail.com>
Message-ID: <20150312214042.31055748@anarchist.wooz.org>

On Mar 11, 2015, at 06:02 PM, Ryan Gonzalez wrote:

>Bad idea? What about when you're running simple scripts you've downloaded
>and Python 3 is on an external hard drive, not /usr/bin?

I don't get it.  Regardless of what a script's shebang says, you can always be
explicit about which version you want to use to run the script.  You don't
*have* to use the shebang's interpreter.  Okay, so it's a little more typing -
is that really a problem for this use case?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/023451e3/attachment.sig>

From barry at python.org  Fri Mar 13 02:45:36 2015
From: barry at python.org (Barry Warsaw)
Date: Thu, 12 Mar 2015 21:45:36 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <20150312175220.6f125d03@anarchist.wooz.org>
 <CACfEFw_9NSwmcFaT=SjYAMzzkqnzoovmQzfPsB_oH0kK=m-YdQ@mail.gmail.com>
Message-ID: <20150312214536.65583928@anarchist.wooz.org>

On Mar 12, 2015, at 05:01 PM, Wes Turner wrote:

>For (re-)packaged scripts, it seems reasonable to expect package
>maintainers to use explicit platform-specific paths.
>
>Otherwise, for checksums and diff noise, does /usr/bin/env python2 make
>sense?

Sorry, I don't understand the question. ;)

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150312/a09246c4/attachment.sig>

From bkabrda at redhat.com  Fri Mar 13 10:00:14 2015
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Fri, 13 Mar 2015 05:00:14 -0400 (EDT)
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150312175220.6f125d03@anarchist.wooz.org>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org>
Message-ID: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com>

----- Original Message -----
> On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:
> 
> >I think the migration should start with modifying scripts to use
> >
> >#!/usr/bin/env python2
> >
> >when they are Python 2 only (*) and
> >
> >#!/usr/bin/env python3
> >
> >when they are Python 3 only and
> >
> >#!/usr/bin/env python
> >
> >only when they support both Python 2 and 3.
> 
> Mostly, I agree, although I would not recommend /usr/bin/env for system
> installed scripts.  Using /usr/bin/env is fine for development trees, but a
> really bad idea for anything installed in system locations.  Modify your
> suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)

+1, the system scripts should use system Python explicitly to keep working when user's environment changes. I'm not sure about Debian/Ubuntu, but we do this in Fedora as much as we can.
And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while coding, because it's just more handy during development.

> I would very much like to see explicit use of /usr/bin/python2 replace
> /usr/bin/python for scripts.

+1 (or, better yet, /usr/bin/python3 ;))

> Then I think the problem is, what do users expect when they type
> 
> $ python foo.py
> 
> at their shell prompt?

For more than a year now, I've been trying to push python 3 as default in Fedora. "Default" means that it should be the only Python on live media/cloud images/... and various system applications and system scripts will run on it. I want Fedora to follow PEP 394, so this doesn't mean switching /usr/bin/python to Python 3, but that's one of the biggest sources of confusion. Most users expect /usr/bin/python to point to "the" Python.

We're not switching for switching /usr/bin/python (for now) for several reasons:
1) PEP 394
2) System packages still using /usr/bin/python
3) Custom user/sysadmin scripts pointing to /usr/bin/python

Point 1 is a recommendation that can change in time, since it's in hands of Python upstream (but then again, it's mostly there because of 3, I think); we'll make point 2 go away in time, because that's something that we can change. Point 3 is the real issue, because (from my experience) there are tons of users/sysadmins that use Python as a shell with nice features and don't really care about what's going on in Python upstream. These scripts are outside of Python upstream and distro control and we can't do anything about them. My thinking is that these won't change - not now, not in two years, not in ten years. The only way to change them is to break them - and then the real question is: do we want to break them or do we want to keep them working forever? (well, at least until we retire Python 2, which would make /usr/bin/python disappear completely)

So what can be done:
a) PEP 394 changes in few years to recommend pointing /usr/bin/python to Python 3, thus breaking (3). Hopefully this would happen at point where most distros are ready (and willing) to do the change.
b) PEP 394 never changes while Python 2 is supported, suggesting switching /usr/bin/python after Python 2 EOL (2020?).
c) PEP 394 never changes, not even after Python 2 EOL.
d) We remove /usr/bin/python. I've already heard several people suggesting this, but I'm not sure what that should mean for us on distro level. For example, should the system-packaged pip have /usr/bin/pip then? Or should that be removed too? Either way, this would nicely solve (3), since people would just get "python not found" and would have to investigate and learn what's going on, doing an explicit decision about which Python they want to use. On the other hand, I can pretty much guarantee that lots of people would hate this, since they just want to type "python" :)
e) Something else?

I have to say I prefer a), since I think that we should just move forward at one point and make Python 3 "the Python". My second preference would be d), since that at least doesn't make it look that Python 2 is still "the Python".

> Cheers,
> -Barry

-- 
Regards,
Slavek Kabrda

From wes.turner at gmail.com  Fri Mar 13 11:38:26 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 13 Mar 2015 05:38:26 -0500
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <20150312175220.6f125d03@anarchist.wooz.org>
 <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com>
Message-ID: <CACfEFw-x2qdgaOMi7WDzEb-o8mga8uuMo03Yua3WxPb1GtA=ZQ@mail.gmail.com>

On Mar 13, 2015 4:02 AM, "Bohuslav Kabrda" <bkabrda at redhat.com> wrote:
>
> ----- Original Message -----
> > On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote:
> >
> > >I think the migration should start with modifying scripts to use
> > >
> > >#!/usr/bin/env python2
> > >
> > >when they are Python 2 only (*) and
> > >
> > >#!/usr/bin/env python3
> > >
> > >when they are Python 3 only and
> > >
> > >#!/usr/bin/env python
> > >
> > >only when they support both Python 2 and 3.
> >
> > Mostly, I agree, although I would not recommend /usr/bin/env for system
> > installed scripts.  Using /usr/bin/env is fine for development trees,
but a
> > really bad idea for anything installed in system locations.  Modify your
> > suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :)
>
> +1, the system scripts should use system Python explicitly to keep
working when user's environment changes. I'm not sure about Debian/Ubuntu,
but we do this in Fedora as much as we can.
> And while I'm primarily a packager, I also use /usr/bin/env python[2|3]
while coding, because it's just more handy during development.
>
> > I would very much like to see explicit use of /usr/bin/python2 replace
> > /usr/bin/python for scripts.
>
> +1 (or, better yet, /usr/bin/python3 ;))
>
> > Then I think the problem is, what do users expect when they type
> >
> > $ python foo.py
> >
> > at their shell prompt?
>
> For more than a year now, I've been trying to push python 3 as default in
Fedora. "Default" means that it should be the only Python on live
media/cloud images/... and various system applications and system scripts
will run on it. I want Fedora to follow PEP 394, so this doesn't mean
switching /usr/bin/python to Python 3, but that's one of the biggest
sources of confusion. Most users expect /usr/bin/python to point to "the"
Python.
>
> We're not switching for switching /usr/bin/python (for now) for several
reasons:
> 1) PEP 394
> 2) System packages still using /usr/bin/python
> 3) Custom user/sysadmin scripts pointing to /usr/bin/python
>
> Point 1 is a recommendation that can change in time, since it's in hands
of Python upstream (but then again, it's mostly there because of 3, I
think); we'll make point 2 go away in time, because that's something that
we can change. Point 3 is the real issue, because (from my experience)
there are tons of users/sysadmins that use Python as a shell with nice
features and don't really care about what's going on in Python upstream.
These scripts are outside of Python upstream and distro control and we
can't do anything about them. My thinking is that these won't change - not
now, not in two years, not in ten years. The only way to change them is to
break them - and then the real question is: do we want to break them or do
we want to keep them working forever? (well, at least until we retire
Python 2, which would make /usr/bin/python disappear completely)
>
> So what can be done:
> a) PEP 394 changes in few years to recommend pointing /usr/bin/python to
Python 3, thus breaking (3). Hopefully this would happen at point where
most distros are ready (and willing) to do the change.
> b) PEP 394 never changes while Python 2 is supported, suggesting
switching /usr/bin/python after Python 2 EOL (2020?).
> c) PEP 394 never changes, not even after Python 2 EOL.
> d) We remove /usr/bin/python. I've already heard several people
suggesting this, but I'm not sure what that should mean for us on distro
level. For example, should the system-packaged pip have /usr/bin/pip then?
Or should that be removed too? Either way, this would nicely solve (3),
since people would just get "python not found" and would have to
investigate and learn what's going on, doing an explicit decision about
which Python they want to use. On the other hand, I can pretty much
guarantee that lots of people would hate this, since they just want to type
"python" :)
> e) Something else?
>
> I have to say I prefer a), since I think that we should just move forward
at one point and make Python 3 "the Python". My second preference would be
d), since that at least doesn't make it look that Python 2 is still "the
Python".

Values / Criteria (for dealing with change):

* backward compatibility (non-breaking)
* cross-platform support
* consistent file checksums
* interpreter arguments
   * PYTHONHASHSEED
   * -O2
* simplest solution
* convenience

Things that require/generate shebang lines:

* OS Packages in ./bin (can be expected to sed /usr/bin/env \(python.*\))
* ./scripts (should not need to be sed-ed to work w/ the current virtualenv
(?))
* console_scripts in ./bin (stubs generated with an explicit path; that can
later be 'made relocatable')

PEP 394: The "python" Command on Unix-Like Systems

* https://www.python.org/dev/peps/pep-0394/#recommendation
* http://legacy.python.org/dev/peps/pep-0394/#recommendation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/209b6b46/attachment.html>

From ncoghlan at gmail.com  Fri Mar 13 12:57:36 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 Mar 2015 21:57:36 +1000
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io>
References: <20150311194325.GA15390@python.ca>
 <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io>
Message-ID: <CADiSq7dej2Ax6OcQ9tzwXXiu3tkuhG9Pcm-ksdEvbK1zY2sNjg@mail.gmail.com>

On 12 March 2015 at 06:14, Donald Stufft <donald at stufft.io> wrote:
> What's worse. Breaking everything that uses an unversioned shebang because /usr/bin/python doesn't exist or breaking some things that doesn't work with Python 3 because /usr/bin/python is now 3?
>
> Generally discussions along this line talk about Python 2 not being installed by default at all (whereas Python 3 would be). So really it's two different broken by default.

Right, this most recently came up in the context of Fedora's plans to
transition to only having Python 3 in the base install
(https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the
moment the expected consequence is that there will come a Fedora
version (tentatively Fedora 23) where, out of the box,
"/usr/bin/python" and "/usr/bin/env python" will just stop working.

Packages that are part of the distro aren't a problem - those are all
being dealt with as the transition progresses, and any which
specifically need Python 2 will depend on it accordingly.

The problem arises with custom user packages and scripts, and third
party packages and scripts. For those, we either just let them break,
or we automatically try running them in Python 3 by change the package
that provides the "/usr/bin/python" symlink. This is actually going to
happen well in advance of Python 3.6, so my original suggestion of
deferring a decision until then (the other discussion Neal was
referring to when starting this thread) won't actually help (as Fedora
23 is due out later this year, while even a slip to Fedora 24 would
still put this change in early 2016).

In favour of moving the symlink to the Python 3 package when Python 3
is the only version a distro installs by default:

* Python 2/3 compatible scripts will "just work"
* Python 3 incompatible scripts will hopefully get a useful error that
points them in the right direction (such as
https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python)
* if the script author is only worried about PEP 394 compliant
environments, they can update their shebang line to "python2", or else
they can set it to "python2.7" if they aren't worried about older
Linux distros that only provide Python 2.6 or earlier.

If we went down this path, Fedora would likely consider enabling
deprecation warnings and Py3k warnings in Rawhide and the Alphas of
the previous release (where Python 2 was still the default) to help
gauge the likely impact.

The case against moving the symlink:

* while it's convenient, running arbitrary user code in the system
Python is also going to be a compatibility/maintainability issue for
both distro developers and our users
* for the Fedora/CentOS/RHEL ecosystem, Software Collections use
environment modules to switch the active Python:
https://www.softwarecollections.org/en/
* if the python symlink is taken away entirely, then it's an
opportunity to give users the choice of either installing the "system
Python 2 stack" (which will be effectively unchanged, just not
installed by default) *or* of installing the Python 2.7 Software
Collection, and getting used to the idea of leaving the system Python
alone
* the advantage of Software Collections is that they support parallel
installation by design, so user's can upgrade their distro without
upgrading the collections they're using, and vice-versa
* other utilities like pyenv and conda similarly support parallel
installs by design

Going down this path would basically mean saying that "python" at the
system level would no longer be defined at all once Python 2 goes EOL
in 2020. A "default" Python would only exist inside an activated
environment module, whether defined by scl, pyenv, conda, or another
tool.

Adopting the latter approach would mean saying "No change to PEP 394"
upstream, and thus passing the situation back to Fedora to decide how
to handle.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Mar 13 13:15:22 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 13 Mar 2015 22:15:22 +1000
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
 <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
 <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>
Message-ID: <CADiSq7fYgtEaZR=nyPvpzu=C7NrS5nmk-oiGRifSOd=HQiZFtg@mail.gmail.com>

On 12 March 2015 at 04:30, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon <brett at python.org> wrote:
>>
>>
>>
>> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>>
>>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>
>>>> On 10 March 2015 at 21:20, Russell Keith-Magee <russell at keith-magee.com>
>>>> wrote:
>>>> >
>>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
>>>> >>
>>>> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
>>>> >> >Hi Ryan,
>>>> >> >
>>>> >> >First off I'm no fan of autoconf, but...
>>>> >> >
>>>> >>
>>>> >> Who is other than the one who designed it? :)
>>>> >
>>>> > Let me be clear - there's a difference between "not a fan" and
>>>> > "calling for
>>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating -
>>>> > but it
>>>> > *works*, and has had some pretty seriously battle testing - both in
>>>> > the
>>>> > abstract "autoconf as a product", and in the specific "autoconf as
>>>> > used in
>>>> > Python". This isn't something that can be said of most of the
>>>> > alternatives
>>>> > in the abstract case, or *any* of the alternatives in the Python case.
>>>>
>>>> Building architecture dependent software is a genuinely hard problem,
>>>> which is why it's full of "80%" solutions like autotools and the
>>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but
>>>> they also basically do their job, so folks tend to develop a high
>>>> tolerance level for their quirks.
>>>>
>>>> In our particular case, we also have the fun problem of "How do you
>>>> build CPython locally when you don't have a working Python interpreter
>>>> yet?". A number of the more interesting alternative C/C++ build
>>>> systems (like Meson, Scons, Waf and the Google "gyp" project that
>>>> powers the node-gyp native extension build system for Node.js) require
>>>> an already working Python interpreter to get things started (when
>>>> starting from a completely blank slate, even our own build sequence
>>>> has to build a deliberately broken partial interpreter just to freeze
>>>> importlib._bootstrap before going on to build a properly configured
>>>> one that can actually import modules).
>>>
>>>
>>> What about something like Google's new GN? It's designed to replace Gyp
>>> in building Chromium and is written in C++.
>>>
>>> I'm pretty sure using a build system written in Lua to build Python is
>>> kind of weird.
>>
>>
>>>
>>>
>>> With all that in mind...what exactly would be requirements that Python
>>> would need in an autotools replacement?
>>
>>
>> Can't require anything more than an ANSI C compiler basically. People run
>> Python in places where they don't have much more than gcc.
>
>
> Not even C99? Darn it...

C99 would probably be OK these days. If there's a good enough
cross-compiling story, I'm also not sure we should entirely rule out
C++, as it's potentially OK if *developing* CPython requires a
mainstream platform with a C++ compiler.

But yeah, you can see why folks look at this problem and end up
thinking "well, you know, maybe autotools isn't *that* bad..."

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From bcannon at gmail.com  Fri Mar 13 13:27:15 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 13 Mar 2015 12:27:15 +0000
Subject: [Python-ideas] About people's accounts being deactivated due to
	bounces
Message-ID: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>

We're trying to figure out what's going on. Just re-enable your account if
you got a notice that it was temporarily disabled.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/1d21849c/attachment.html>

From barry at python.org  Fri Mar 13 18:32:04 2015
From: barry at python.org (Barry Warsaw)
Date: Fri, 13 Mar 2015 13:32:04 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <20150312175220.6f125d03@anarchist.wooz.org>
 <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com>
Message-ID: <20150313133204.6753253b@marathon>

On Mar 13, 2015, at 05:00 AM, Bohuslav Kabrda wrote:

>+1, the system scripts should use system Python explicitly to keep working
>when user's environment changes. I'm not sure about Debian/Ubuntu, but we do
>this in Fedora as much as we can.

We do this in Debian/Ubuntu too, although it looks like there are one or two
outliers so they probably aren't built with the common Python build tools.

>And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while
>coding, because it's just more handy during development.

Oh, definitely.

>For more than a year now, I've been trying to push python 3 as default in
>Fedora. "Default" means that it should be the only Python on live media/cloud
>images/... and various system applications and system scripts will run on
>it.

This is a great goal, and something that we've been working toward in
Debian/Ubuntu also.  I hope all the Linux distros can do the same thing.

>I want Fedora to follow PEP 394, so this doesn't mean switching
>/usr/bin/python to Python 3, but that's one of the biggest sources of
>confusion. Most users expect /usr/bin/python to point to "the" Python.

This is what we need to tease out I think.  For the most part, users don't
care what the shebang line of their scripts is, or even if they have one.
They *do* care about what interpreter fires up when they type `python` or any
of the variants at their shell, as well as...

>3) Custom user/sysadmin scripts pointing to /usr/bin/python

[...]

>Point 3 is the real issue, because (from my experience) there are tons of
>users/sysadmins that use Python as a shell with nice features and don't
>really care about what's going on in Python upstream. These scripts are
>outside of Python upstream and distro control and we can't do anything about
>them. My thinking is that these won't change - not now, not in two years, not
>in ten years. The only way to change them is to break them - and then the
>real question is: do we want to break them or do we want to keep them working
>forever? (well, at least until we retire Python 2, which would make
>/usr/bin/python disappear completely)

That is indeed the question.  I think if we break them, we're just pissing
people off for a not compelling enough reason (notice I didn't say "good"
reason :).

It think the first step is to strongly encourage people to start using
/usr/bin/python2 for their Python 2 scripts, and to strongly encourage all
distributions of Python to be including a 'python2' symlink.  Of course, for
anyone with even a rudimentary understanding of *nix, it's easy enough to
create that symlink if it's missing, but we might even go a step further and
include a 'fix_py2_usr_bin' script that people can run which wouldn't do
anything more than ensure /usr/bin/python2 exists.

We've committed to bug fix releases for 2.7 until 2020.  To me, that's the
time frame for any change to /usr/bin/python, whether that's complete removal
or repointing the symlink.  By then, we'll have 5 more years of educating
people to be using 'python2' and 'python3' explicitly so the impact of
resolving the ambiguity around 'python' should be lessened.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/859bb888/attachment-0001.sig>

From rymg19 at gmail.com  Fri Mar 13 19:05:21 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 13:05:21 -0500
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CADiSq7fYgtEaZR=nyPvpzu=C7NrS5nmk-oiGRifSOd=HQiZFtg@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
 <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
 <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>
 <CADiSq7fYgtEaZR=nyPvpzu=C7NrS5nmk-oiGRifSOd=HQiZFtg@mail.gmail.com>
Message-ID: <CAO41-mOVsEXdRMHnx2tjopuwRcG-YSJOXUcTV0dgCgFDC5FF2g@mail.gmail.com>

If C99's ok, there's hope. :)

C++'s main issue is that there are platforms that support C99 but have
out-of-date or non-complaint C++ compilers. I love the language (yes, I'm
weird), but it's tricky to have truly portable C++ programs because if all
the holes in various compilers. Just look at all the #ifdef's in Boost.

On Fri, Mar 13, 2015 at 7:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 12 March 2015 at 04:30, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> > On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon <brett at python.org> wrote:
> >>
> >>
> >>
> >> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez <rymg19 at gmail.com>
> wrote:
> >>>
> >>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >>>>
> >>>> On 10 March 2015 at 21:20, Russell Keith-Magee <
> russell at keith-magee.com>
> >>>> wrote:
> >>>> >
> >>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
> >>>> >>
> >>>> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
> >>>> >> >Hi Ryan,
> >>>> >> >
> >>>> >> >First off I'm no fan of autoconf, but...
> >>>> >> >
> >>>> >>
> >>>> >> Who is other than the one who designed it? :)
> >>>> >
> >>>> > Let me be clear - there's a difference between "not a fan" and
> >>>> > "calling for
> >>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating
> -
> >>>> > but it
> >>>> > *works*, and has had some pretty seriously battle testing - both in
> >>>> > the
> >>>> > abstract "autoconf as a product", and in the specific "autoconf as
> >>>> > used in
> >>>> > Python". This isn't something that can be said of most of the
> >>>> > alternatives
> >>>> > in the abstract case, or *any* of the alternatives in the Python
> case.
> >>>>
> >>>> Building architecture dependent software is a genuinely hard problem,
> >>>> which is why it's full of "80%" solutions like autotools and the
> >>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but
> >>>> they also basically do their job, so folks tend to develop a high
> >>>> tolerance level for their quirks.
> >>>>
> >>>> In our particular case, we also have the fun problem of "How do you
> >>>> build CPython locally when you don't have a working Python interpreter
> >>>> yet?". A number of the more interesting alternative C/C++ build
> >>>> systems (like Meson, Scons, Waf and the Google "gyp" project that
> >>>> powers the node-gyp native extension build system for Node.js) require
> >>>> an already working Python interpreter to get things started (when
> >>>> starting from a completely blank slate, even our own build sequence
> >>>> has to build a deliberately broken partial interpreter just to freeze
> >>>> importlib._bootstrap before going on to build a properly configured
> >>>> one that can actually import modules).
> >>>
> >>>
> >>> What about something like Google's new GN? It's designed to replace Gyp
> >>> in building Chromium and is written in C++.
> >>>
> >>> I'm pretty sure using a build system written in Lua to build Python is
> >>> kind of weird.
> >>
> >>
> >>>
> >>>
> >>> With all that in mind...what exactly would be requirements that Python
> >>> would need in an autotools replacement?
> >>
> >>
> >> Can't require anything more than an ANSI C compiler basically. People
> run
> >> Python in places where they don't have much more than gcc.
> >
> >
> > Not even C99? Darn it...
>
> C99 would probably be OK these days. If there's a good enough
> cross-compiling story, I'm also not sure we should entirely rule out
> C++, as it's potentially OK if *developing* CPython requires a
> mainstream platform with a C++ compiler.
>
> But yeah, you can see why folks look at this problem and end up
> thinking "well, you know, maybe autotools isn't *that* bad..."
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/5699fdf1/attachment.html>

From rymg19 at gmail.com  Fri Mar 13 19:43:46 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 13:43:46 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
Message-ID: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>

So...I've been looking stuff up. Don't have too much free time right now,
but it was mostly Googleing.

I have deemed three ideas to be best for Python:

- Write a custom makefile generator. I have some ideas, but they likely
won't come to light any time soon.

- Use Boost.Build. It seems ugly and complicated, but it actually has a
full configuring system that can detect libraries and try to compile source
files. It's written in C89, not C++, and its dependency is the Boehm GC,
which probably supports everywhere that Python runs on.

Only issue? It's undergoing a rewrite in Python. However, at the rate it's
going, Python will likely be at version 7 by the time they finish.

- Stick with autotools. Trying to avoid this.

Thoughts? Again?

-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/f817664d/attachment.html>

From wes.turner at gmail.com  Fri Mar 13 20:27:29 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 13 Mar 2015 14:27:29 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
Message-ID: <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>

Scons? http://www.scons.org/doc/production/HTML/scons-user.html
On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:

> So...I've been looking stuff up. Don't have too much free time right now,
> but it was mostly Googleing.
>
> I have deemed three ideas to be best for Python:
>
> - Write a custom makefile generator. I have some ideas, but they likely
> won't come to light any time soon.
>
> - Use Boost.Build. It seems ugly and complicated, but it actually has a
> full configuring system that can detect libraries and try to compile source
> files. It's written in C89, not C++, and its dependency is the Boehm GC,
> which probably supports everywhere that Python runs on.
>
> Only issue? It's undergoing a rewrite in Python. However, at the rate it's
> going, Python will likely be at version 7 by the time they finish.
>
> - Stick with autotools. Trying to avoid this.
>
> Thoughts? Again?
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/7c66d44e/attachment-0001.html>

From rymg19 at gmail.com  Fri Mar 13 20:40:50 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 14:40:50 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
Message-ID: <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>

That written in Python, so Python would require Python to build which would
require Python to build which would require...

Besides, SCons sucks. If I were using a build system written in Python, I'd
use fbuild or Waf.


On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner <wes.turner at gmail.com> wrote:

> Scons? http://www.scons.org/doc/production/HTML/scons-user.html
> On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>
>> So...I've been looking stuff up. Don't have too much free time right now,
>> but it was mostly Googleing.
>>
>> I have deemed three ideas to be best for Python:
>>
>> - Write a custom makefile generator. I have some ideas, but they likely
>> won't come to light any time soon.
>>
>> - Use Boost.Build. It seems ugly and complicated, but it actually has a
>> full configuring system that can detect libraries and try to compile source
>> files. It's written in C89, not C++, and its dependency is the Boehm GC,
>> which probably supports everywhere that Python runs on.
>>
>> Only issue? It's undergoing a rewrite in Python. However, at the rate
>> it's going, Python will likely be at version 7 by the time they finish.
>>
>> - Stick with autotools. Trying to avoid this.
>>
>> Thoughts? Again?
>>
>> --
>> Ryan
>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>> program. Something?s wrong.
>> http://kirbyfan64.github.io/
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/5a640978/attachment.html>

From wes.turner at gmail.com  Fri Mar 13 20:46:17 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 13 Mar 2015 14:46:17 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
Message-ID: <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>

The subject line mentions "Part 2". Is there a link to "Part 1"? What is
the rationale for changing?

Make, autoconf, and automake are already widely ported/portable.

What are the criteria for evaluating alternatives?
On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:

> That written in Python, so Python would require Python to build which
> would require Python to build which would require...
>
> Besides, SCons sucks. If I were using a build system written in Python,
> I'd use fbuild or Waf.
>
>
> On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner <wes.turner at gmail.com> wrote:
>
>> Scons? http://www.scons.org/doc/production/HTML/scons-user.html
>> On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>>
>>> So...I've been looking stuff up. Don't have too much free time right
>>> now, but it was mostly Googleing.
>>>
>>> I have deemed three ideas to be best for Python:
>>>
>>> - Write a custom makefile generator. I have some ideas, but they likely
>>> won't come to light any time soon.
>>>
>>> - Use Boost.Build. It seems ugly and complicated, but it actually has a
>>> full configuring system that can detect libraries and try to compile source
>>> files. It's written in C89, not C++, and its dependency is the Boehm GC,
>>> which probably supports everywhere that Python runs on.
>>>
>>> Only issue? It's undergoing a rewrite in Python. However, at the rate
>>> it's going, Python will likely be at version 7 by the time they finish.
>>>
>>> - Stick with autotools. Trying to avoid this.
>>>
>>> Thoughts? Again?
>>>
>>> --
>>> Ryan
>>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>>> program. Something?s wrong.
>>> http://kirbyfan64.github.io/
>>>
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/1462c545/attachment.html>

From jeanpierreda at gmail.com  Fri Mar 13 20:47:45 2015
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 13 Mar 2015 12:47:45 -0700
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
Message-ID: <CABicbJ+q7urU_dQqoZtk-NLe=p1_nufm-ia4HDJOWGvtqjYy7w@mail.gmail.com>

On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> I've been trying to work on Issue 23496. And want to tear my hair out. I've
> noticed a few problems with Python's current build system:
>
> - There is no separation of the host C compiler and the target C compiler.
> When you give configure a C compiler for cross-compiling, make will happily
> try to build Parser/pgen with the cross-compiler. And try to run the
> generated executable. And fail. I know you can use `make touch` to get
> around this, but it still sucks.

I just want to echo this. I had really horrible problems trying to get
Python working with emscripten a year ago, because of this and other
problems. (Gave up.)  It turned out emscripten was a bigger cause for
complaint, but, still.

-- Devin

> - For some reason, autoconf decided to re-configure everything about 2
> minutes into make. But it didn't pass any of the flags I needed, resulting
> in everything failing.
>
> - I do not like having to use this for configuring:
>
>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ ./configure
> --host=arm-linux-androideabi --build=x86_64 --disable-ipv6
> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
> ac_cv_have_long_long_format=yes
>
> At the same time, I'm not quite sure how many people would want to sift
> through the 5000 lines in configure.ac or the 1650 lines in Makefile.pre.in.
>
> I just get the feeling that, whenever I build Python, there are too many
> holes/hacks to get around make/autoconf things.
>
> --
> Ryan
> If I were in a 10-story building glass-sided building and forced to write
> either Go or autconf scripts, I?d jump out a window.
> http://kirbyfan64.github.io/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rymg19 at gmail.com  Fri Mar 13 20:58:06 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 14:58:06 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
Message-ID: <CAO41-mO5_i7zjjih-w2Kk0abwG44k5o-HmfDhr3=ZWMaOF99ug@mail.gmail.com>

https://mail.python.org/pipermail/python-ideas/2015-March/032381.html

On Fri, Mar 13, 2015 at 2:46 PM, Wes Turner <wes.turner at gmail.com> wrote:

> The subject line mentions "Part 2". Is there a link to "Part 1"? What is
> the rationale for changing?
>
> Make, autoconf, and automake are already widely ported/portable.
>
> What are the criteria for evaluating alternatives?
> On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>
>> That written in Python, so Python would require Python to build which
>> would require Python to build which would require...
>>
>> Besides, SCons sucks. If I were using a build system written in Python,
>> I'd use fbuild or Waf.
>>
>>
>> On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner <wes.turner at gmail.com> wrote:
>>
>>> Scons? http://www.scons.org/doc/production/HTML/scons-user.html
>>> On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>>>
>>>> So...I've been looking stuff up. Don't have too much free time right
>>>> now, but it was mostly Googleing.
>>>>
>>>> I have deemed three ideas to be best for Python:
>>>>
>>>> - Write a custom makefile generator. I have some ideas, but they likely
>>>> won't come to light any time soon.
>>>>
>>>> - Use Boost.Build. It seems ugly and complicated, but it actually has a
>>>> full configuring system that can detect libraries and try to compile source
>>>> files. It's written in C89, not C++, and its dependency is the Boehm GC,
>>>> which probably supports everywhere that Python runs on.
>>>>
>>>> Only issue? It's undergoing a rewrite in Python. However, at the rate
>>>> it's going, Python will likely be at version 7 by the time they finish.
>>>>
>>>> - Stick with autotools. Trying to avoid this.
>>>>
>>>> Thoughts? Again?
>>>>
>>>> --
>>>> Ryan
>>>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>>>> program. Something?s wrong.
>>>> http://kirbyfan64.github.io/
>>>>
>>>>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>>
>>
>>
>> --
>> Ryan
>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>> program. Something?s wrong.
>> http://kirbyfan64.github.io/
>>
>>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/ad6c2cf7/attachment-0001.html>

From rymg19 at gmail.com  Fri Mar 13 21:00:03 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 15:00:03 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
Message-ID: <CAO41-mMJEfMeUvGy0aeT9kGpoacmiGc4SiSNOGhBbN_8HrBY6A@mail.gmail.com>

On Fri, Mar 13, 2015 at 2:47 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> That written in Python, so Python would require Python to build which
>> would require Python to build which would require...
>>
>
> Well, you can't buid a C compiler without a C compiler either (or SOME
> compiler). And given that you should be able to do cross-compilation for
> platforms that don't currently have a python build, I'm not sure that's a
> killer.
>
> It it so wrong that python eats its own dog food in this manner?
>
>
It's not as much "Python eat its own dog food" as "cross-compiling natively
would be even harder than it is right now."

C compilers are available on virtually every UNIX that's ever been created
since the 70's, including AIX, Plan 9, Solaris, etc.

Python? Not as much.


> -CHB
>
>>
>> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/3c9e6a41/attachment.html>

From wes.turner at gmail.com  Fri Mar 13 21:01:30 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 13 Mar 2015 15:01:30 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
Message-ID: <CACfEFw8x1ABQT5VtCakFxT_RcDOt=xU2iQ4aupZfOtkEQ2KmWA@mail.gmail.com>

On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>
> That written in Python, so Python would require Python to build which
would require Python to build which would require...
>
> Besides, SCons sucks. If I were using a build system written in Python,
I'd use fbuild or Waf.

Apparently, Waf started as a fork of SCons. [1][2]

[1]
http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects
[2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf

I don't know anything about fbuild; others might.

A feature/alternative matrix could be helpful (in addition to a preliminary
implementation) for making a cogent argument (in PEP form [3][4]). This is
possible in RST with table or list-table syntax.

[3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates
[4] https://www.python.org/dev/peps/pep-0009/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/e30bf681/attachment.html>

From rymg19 at gmail.com  Fri Mar 13 21:11:41 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 13 Mar 2015 15:11:41 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CACfEFw8x1ABQT5VtCakFxT_RcDOt=xU2iQ4aupZfOtkEQ2KmWA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw8x1ABQT5VtCakFxT_RcDOt=xU2iQ4aupZfOtkEQ2KmWA@mail.gmail.com>
Message-ID: <CAO41-mPRKOBnPSTwvtZVVvYYUKG0AGh6y4WJZFJRCQ1npTWb0g@mail.gmail.com>

On Fri, Mar 13, 2015 at 3:01 PM, Wes Turner <wes.turner at gmail.com> wrote:

>
> On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
> >
> > That written in Python, so Python would require Python to build which
> would require Python to build which would require...
> >
> > Besides, SCons sucks. If I were using a build system written in Python,
> I'd use fbuild or Waf.
>
> Apparently, Waf started as a fork of SCons. [1][2]
>
> [1]
> http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects
> [2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf
>

But it's now better.


> I don't know anything about fbuild; others might.
>

https://github.com/felix-lang/fbuild


> A feature/alternative matrix could be helpful (in addition to a
> preliminary implementation) for making a cogent argument (in PEP form
> [3][4]). This is possible in RST with table or list-table syntax.
>
> [3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates
> [4] https://www.python.org/dev/peps/pep-0009/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/cddab011/attachment.html>

From chris.barker at noaa.gov  Fri Mar 13 20:47:05 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 13 Mar 2015 12:47:05 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
Message-ID: <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>

On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> That written in Python, so Python would require Python to build which
> would require Python to build which would require...
>

Well, you can't buid a C compiler without a C compiler either (or SOME
compiler). And given that you should be able to do cross-compilation for
platforms that don't currently have a python build, I'm not sure that's a
killer.

It it so wrong that python eats its own dog food in this manner?

-CHB

>
> --

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150313/b35d9e97/attachment-0001.html>

From abarnert at yahoo.com  Fri Mar 13 22:46:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 13 Mar 2015 14:46:20 -0700
Subject: [Python-ideas] About people's accounts being deactivated due to
	bounces
In-Reply-To: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
References: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
Message-ID: <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com>

I think this may be related to a problem with changes in DKIM policy at Yahoo, or an expiration of DKIM certs at Python.org, or something else that may have changed somewhere in that area.

I've noticed that since around 15 January, mailing lists (at least those that munge headers and preserve the original sender's From, as many do) have had two problems: (1) messages from Yahoo users (or other users whose provider has a DKIM policy but isn't gmail) through the list to gmail get marked as spam, and (2) a flood of bounces. I don't know why this didn't just affect every user on every mailing list all at once instead of randomly hitting some but not all users at some random time in the last two months.

Sent from my iPhone

> On Mar 13, 2015, at 5:27 AM, Brett Cannon <bcannon at gmail.com> wrote:
> 
> We're trying to figure out what's going on. Just re-enable your account if you got a notice that it was temporarily disabled.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From robertc at robertcollins.net  Sat Mar 14 01:07:57 2015
From: robertc at robertcollins.net (Robert Collins)
Date: Sat, 14 Mar 2015 13:07:57 +1300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
Message-ID: <CAJ3HoZ2PedQJ5K1yhe=f52ndx4s4ZAqWgFspkPfo39vnNQTG-g@mail.gmail.com>

On 14 Mar 2015 09:44, "Chris Barker" <chris.barker at noaa.gov> wrote:
>
> On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>> That written in Python, so Python would require Python to build which
would require Python to build which would require...
>
>
> Well, you can't buid a C compiler without a C compiler either (or SOME
compiler). And given that you should be able to do cross-compilation for
platforms that don't currently have a python build, I'm not sure that's a
killer.
>
> It it so wrong that python eats its own dog food in this manner?

Wrong, no.  harder yes. C compilers have mature cross compiler build chains
including stage aware test suites and the like. When a tool is it's own
build dependency bootstrapping it on a new platform gains a hard
requirement on cross compiling rather than it being a thing for special
cases like platforms that cannot compile the tool. This makes the life of
distribution maintainers harder too, since cross arch platform
bootstrapping becomes even more manual.

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/3e837bd8/attachment.html>

From random832 at fastmail.us  Sat Mar 14 06:10:43 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Sat, 14 Mar 2015 01:10:43 -0400
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAJ3HoZ2PedQJ5K1yhe=f52ndx4s4ZAqWgFspkPfo39vnNQTG-g@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAJ3HoZ2PedQJ5K1yhe=f52ndx4s4ZAqWgFspkPfo39vnNQTG-g@mail.gmail.com>
Message-ID: <1426309843.2639153.240240573.03996CBC@webmail.messagingengine.com>



On Fri, Mar 13, 2015, at 20:07, Robert Collins wrote:
> Wrong, no.  harder yes. C compilers have mature cross compiler build
> chains
> including stage aware test suites and the like. When a tool is it's own
> build dependency bootstrapping it on a new platform gains a hard
> requirement on cross compiling rather than it being a thing for special
> cases like platforms that cannot compile the tool.


That's what bootstrapping is. If you're not doing this, you're not
bootstrapping.

From stefan at bytereef.org  Sat Mar 14 12:54:52 2015
From: stefan at bytereef.org (Stefan Krah)
Date: Sat, 14 Mar 2015 11:54:52 +0000 (UTC)
Subject: [Python-ideas] CPython needs to revamp/replace it's build
	system && Cross-compiling Python sucks
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CABicbJ+q7urU_dQqoZtk-NLe=p1_nufm-ia4HDJOWGvtqjYy7w@mail.gmail.com>
Message-ID: <loom.20150314T124603-461@post.gmane.org>

Devin Jeanpierre <jeanpierreda at ...> writes:
> On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez <rymg19 <at> gmail.com> wrote:
> > I've been trying to work on Issue 23496. And want to tear my hair out. I've
> > noticed a few problems with Python's current build system:
> >
> > - There is no separation of the host C compiler and the target C compiler.
> > When you give configure a C compiler for cross-compiling, make will happily
> > try to build Parser/pgen with the cross-compiler. And try to run the
> > generated executable. And fail. I know you can use `make touch` to get
> > around this, but it still sucks.
> 
> I just want to echo this. I had really horrible problems trying to get
> Python working with emscripten a year ago, because of this and other
> problems. (Gave up.)  It turned out emscripten was a bigger cause for
> complaint, but, still.

Someone just needs to open an issue that we need 'make bootstrap'
or something similar.

Replacing autotools does not seem very realistic or desirable.
A couple of years ago I did LinuxFromScratch and autotools was
*by far* the most reliable and problem free build system.

I can't say the same about Cmake.


Stefan Krah





From solipsis at pitrou.net  Sat Mar 14 14:03:23 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 14 Mar 2015 14:03:23 +0100
Subject: [Python-ideas] Rewriting the build system (part 2)
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAJ3HoZ2PedQJ5K1yhe=f52ndx4s4ZAqWgFspkPfo39vnNQTG-g@mail.gmail.com>
Message-ID: <20150314140323.1964d347@fsol>

On Sat, 14 Mar 2015 13:07:57 +1300
Robert Collins
<robertc at robertcollins.net> wrote:
> On 14 Mar 2015 09:44, "Chris Barker" <chris.barker at noaa.gov> wrote:
> >
> > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> >>
> >> That written in Python, so Python would require Python to build which
> would require Python to build which would require...
> >
> >
> > Well, you can't buid a C compiler without a C compiler either (or SOME
> compiler). And given that you should be able to do cross-compilation for
> platforms that don't currently have a python build, I'm not sure that's a
> killer.
> >
> > It it so wrong that python eats its own dog food in this manner?
> 
> Wrong, no.  harder yes. C compilers have mature cross compiler build chains
> including stage aware test suites and the like.

Not to mention that any decent platform manufacturer or system vendor
will do the job of writing or porting a C compiler for the platform.
Few of them will port Python for us, though.

Regards

Antoine.



From flying-sheep at web.de  Sat Mar 14 14:32:42 2015
From: flying-sheep at web.de (Philipp A.)
Date: Sat, 14 Mar 2015 13:32:42 +0000
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
Message-ID: <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>

the PEP is pretty perfect.

#!/usr/bin/env python_binary_this_script_runs_with

if it runs with python 2: #!/usr/bin/env python2
if it runs with python 3: #!/usr/bin/env python3
if it runs with both: #!/usr/bin/env python

the only thing worth discussing IMHO is if python 3 should start shipping
with *all* symlinks from past 3.x minor versions to the newest minor
version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g.
pathlib and would be sure that the script will work with future versions.
but then again, users can install pathlib for 3.3?
?

Ryan Gonzalez <rymg19 at gmail.com> schrieb am Do., 12. M?rz 2015 um 00:13 Uhr:

> On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker <chris.barker at noaa.gov>
> wrote:
>
>> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg at krypto.org>
>> wrote:
>>
>>> I think /usr/bin/env is a bad idea to promote. It changes from an
>>> explicit designation of what you need in order to execute to something
>>> being derived from the users $PATH.  It is *less specific* than
>>> /usr/bin/python. For some people that is the goal, but I discourage it.
>>>
>>
>> I don't think the goal is specificity, or lack there of, but a shift in
>> control:
>>
>> Who controls where python is installed?
>>
>> If you put an explicit full path on the #! line, then the script author
>> controls which python is used, AND where it must be installed.
>>
>> If you put /usr/bin env on the #! line, then the script author is saying
>> 'use python", and the script user can have python installed anywhere they
>> want, and can use PATH to tell everything what the default it.
>>
>> I think that _should_ be encouraged -- but ideally with some versioning:
>>
>> #!/usr/bin/env python2
>>
>> or even:
>>
>> #!/usr/bin/env python3.4
>>
>>
> Please, just explicit major versions, not minor ones. python3, not
> python3.4. Just earlier this year I had to fix about 20 scripts that ran
> perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.
>
>
>> If any of you remember what a pain it was when RedHat installed system
>> tools with (I think):
>>
>> !#/usr/bin/env python
>>
>> You'll know what a nightmare that was -- you could not upgrade the
>> 'default" python without breaking stuff.
>>
>> So system tools should probably use the full specific path -- the script
>> is tied closely to the environment. But if they had at least put a version
>> in there, it would have worked fine:
>>
>> #!/usr/bin/env python1.5
>>
>> These days, I try to use distutils or setuptools to install scripts, and
>> they can do the right thing to use the correct python to start up anyway...
>>
>> -CHB
>>
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>  _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/1f4f4575/attachment.html>

From abarnert at yahoo.com  Sat Mar 14 16:03:41 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 14 Mar 2015 08:03:41 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
Message-ID: <B66187B2-C320-4DD1-BE4F-9C155C4E94B6@yahoo.com>

On Mar 14, 2015, at 6:32 AM, Philipp A. <flying-sheep at web.de> wrote:
> 
> the PEP is pretty perfect.
> 
> #!/usr/bin/env python_binary_this_script_runs_with
> if it runs with python 2: #!/usr/bin/env python2
> if it runs with python 3: #!/usr/bin/env python3
> if it runs with both: #!/usr/bin/env python
> 
I don't see how this is perfect in the real world, or even a good recommendation.

Continuing to use python rather than python2 will work out of the box on nearly every *nix system today, with no exceptions larger than Arch Linux. Switching to python2 means you no longer work on a number of widely-deployed and still-supported versions (including even the latest betas of Mac OS X). What script developer in his right mind would take that tradeoff? (Especially what developer who isn't willing to move to 3.x, or to distribute or deploy their scripts in a more reasonable way, either of which eliminates the problem...)

And the only way that's likely to change over the next decade is that some platforms just won't include Python 2.7 out of the box at all--in which case python2 won't work any better than plain python.
> the only thing worth discussing IMHO is if python 3 should start shipping with all symlinks from past 3.x minor versions to the newest minor version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and would be sure that the script will work with future versions. but then again, users can install pathlib for 3.3?
> 
> 
> Ryan Gonzalez <rymg19 at gmail.com> schrieb am Do., 12. M?rz 2015 um 00:13 Uhr:
>>> On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>> 
>>>> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith <greg at krypto.org> wrote:
>>>> I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH.  It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it.
>>> 
>>> I don't think the goal is specificity, or lack there of, but a shift in control:
>>> 
>>> Who controls where python is installed?
>>> 
>>> If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed.
>>> 
>>> If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it.
>>> 
>>> I think that _should_ be encouraged -- but ideally with some versioning:
>>> 
>>> #!/usr/bin/env python2
>>> 
>>> or even:
>>> 
>>> #!/usr/bin/env python3.4
>> 
>> Please, just explicit major versions, not minor ones. python3, not python3.4. Just earlier this year I had to fix about 20 scripts that ran perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks.
>>  
>>> If any of you remember what a pain it was when RedHat installed system tools with (I think):
>>> 
>>> !#/usr/bin/env python
>>> 
>>> You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff. 
>>> 
>>> So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine:
>>> 
>>> #!/usr/bin/env python1.5
>>> 
>>> These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway...
>>> 
>>> -CHB
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> Christopher Barker, Ph.D.
>>> Oceanographer
>>> 
>>> Emergency Response Division
>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>> Seattle, WA  98115       (206) 526-6317   main reception
>>> 
>>> Chris.Barker at noaa.gov
>>> 
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> 
>> -- 
>> Ryan
>> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong.
>> http://kirbyfan64.github.io/ 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/6180a707/attachment-0001.html>

From rosuav at gmail.com  Sat Mar 14 16:27:02 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 15 Mar 2015 02:27:02 +1100
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
Message-ID: <CAPTjJmpkSupLgY5z-OeuO2qCn-k05JDUir3ZgksSB2b0vVcDRQ@mail.gmail.com>

On Sun, Mar 15, 2015 at 12:32 AM, Philipp A. <flying-sheep at web.de> wrote:
> the only thing worth discussing IMHO is if python 3 should start shipping
> with all symlinks from past 3.x minor versions to the newest minor version.
> then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and
> would be sure that the script will work with future versions. but then
> again, users can install pathlib for 3.3?

I would strongly advise NOT doing this. If you say "python3.4", then
it should run on Python 3.4, not on Python 3.4-or-anything-later.

ChrisA

From barry at python.org  Sat Mar 14 16:47:59 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 14 Mar 2015 11:47:59 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
 <B66187B2-C320-4DD1-BE4F-9C155C4E94B6@yahoo.com>
Message-ID: <20150314114759.5fb7f8e3@anarchist.wooz.org>

On Mar 14, 2015, at 08:03 AM, Andrew Barnert wrote:

>Continuing to use python rather than python2 will work out of the box on
>nearly every *nix system today, with no exceptions larger than Arch
>Linux.

The part of the PEP aimed at Python distribution providers should recommend
that `python2` be used in preference of just `python` and that the symlink be
present.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/88a45ad6/attachment.sig>

From barry at python.org  Sat Mar 14 16:57:19 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 14 Mar 2015 11:57:19 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
References: <20150311194325.GA15390@python.ca>
 <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io>
 <CADiSq7dej2Ax6OcQ9tzwXXiu3tkuhG9Pcm-ksdEvbK1zY2sNjg@mail.gmail.com>
Message-ID: <20150314115719.23ea9d75@anarchist.wooz.org>

On Mar 13, 2015, at 09:57 PM, Nick Coghlan wrote:

>Right, this most recently came up in the context of Fedora's plans to
>transition to only having Python 3 in the base install
>(https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the
>moment the expected consequence is that there will come a Fedora
>version (tentatively Fedora 23) where, out of the box,
>"/usr/bin/python" and "/usr/bin/env python" will just stop working.
>
>Packages that are part of the distro aren't a problem - those are all
>being dealt with as the transition progresses, and any which
>specifically need Python 2 will depend on it accordingly.
>
>The problem arises with custom user packages and scripts, and third
>party packages and scripts.

Sure, agreed so far.  But you're not going to be removing Python 2 from the
Fedora archive right?  So any user with third party packages that require
Python 2 could just install it and their stuff would continue to work.

At least that's the plan on Debian/Ubuntu.  I'd like to be at the point[*]
where the default installs would not contain Python 2, but a simple `apt-get`
would install it from the archives for anybody who needs it.  That seems like
a safer option than changing the symlink.

Cheers,
-Barry

[*] I'm aiming for Debian Stretch and Ubuntu 16.04 Xylophonic X-ray fish LTS.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/07a16c15/attachment.sig>

From barry at python.org  Sat Mar 14 16:59:23 2015
From: barry at python.org (Barry Warsaw)
Date: Sat, 14 Mar 2015 11:59:23 -0400
Subject: [Python-ideas] About people's accounts being deactivated due to
	bounces
References: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
 <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com>
Message-ID: <20150314115923.27aa5ead@anarchist.wooz.org>

On Mar 13, 2015, at 02:46 PM, Andrew Barnert wrote:

>I think this may be related to a problem with changes in DKIM policy at
>Yahoo, or an expiration of DKIM certs at Python.org, or something else that
>may have changed somewhere in that area.

Please contact postmaster at python.org.  Mark recently upgraded Mailman on
mail.python.org and the new version has some DKIM mitigation stuff.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/89a1ed94/attachment.sig>

From rosuav at gmail.com  Sat Mar 14 17:44:38 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 15 Mar 2015 03:44:38 +1100
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
Message-ID: <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>

On Sat, Mar 14, 2015 at 6:47 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>> That written in Python, so Python would require Python to build which
>> would require Python to build which would require...
>
>
> Well, you can't buid a C compiler without a C compiler either (or SOME
> compiler). And given that you should be able to do cross-compilation for
> platforms that don't currently have a python build, I'm not sure that's a
> killer.
>
> It it so wrong that python eats its own dog food in this manner?
>
> -CHB

I've been just skimming this thread rather than reading in detail, so
please forgive me if this question is completely unrelated.

Will this affect the commands required to build Python from source, or
is all this just for actual development?

Consider two completely different workflows here: Firstly, a core
Python dev. S/he wants to be able to efficiently build, edit, etc, and
wants to be able to edit any part of the code. and secondly, someone
who has little knowledge of Python's internals, and just wants to
build the latest version (either from hg or from a source tarball).

In the first case, it's fine to require that some version of Python be
already installed. If editing file XYZ requires that file ABC be
rebuilt from it, and the rebuild process requires Python 3.3 or newer,
that's not going to hurt people. The modified ABC gets checked in
along with the change to XYZ, and all's well.

But in the second case, it'd be much safer to minimize the
requirements. If you can untar the source and just "./configure; make;
sudo make install" your way to having the latest Python, that's a LOT
easier than I've just been doing trying to get my Dad onto a newer
Mozilla Thunderbird. (I don't understand what it's doing all along
there, but it pulls a whole bunch of separate repositories when I say
update, and there's a separate bootstrap step, and ... I dunno.)

ChrisA

From brett at python.org  Sat Mar 14 17:52:22 2015
From: brett at python.org (Brett Cannon)
Date: Sat, 14 Mar 2015 16:52:22 +0000
Subject: [Python-ideas] About people's accounts being deactivated due to
	bounces
In-Reply-To: <20150314115923.27aa5ead@anarchist.wooz.org>
References: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
 <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com>
 <20150314115923.27aa5ead@anarchist.wooz.org>
Message-ID: <CAP1=2W6XhdfR7Y4LeryDSnZ--CJVRXkBMnGYA+ZY_6wGJ-oUkA@mail.gmail.com>

Already did.

On Sat, Mar 14, 2015, 12:02 Barry Warsaw <barry at python.org> wrote:

> On Mar 13, 2015, at 02:46 PM, Andrew Barnert wrote:
>
> >I think this may be related to a problem with changes in DKIM policy at
> >Yahoo, or an expiration of DKIM certs at Python.org, or something else
> that
> >may have changed somewhere in that area.
>
> Please contact postmaster at python.org.  Mark recently upgraded Mailman on
> mail.python.org and the new version has some DKIM mitigation stuff.
>
> Cheers,
> -Barry
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150314/b919c827/attachment-0001.html>

From abarnert at yahoo.com  Sun Mar 15 04:50:37 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 14 Mar 2015 20:50:37 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150314114759.5fb7f8e3@anarchist.wooz.org>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
 <B66187B2-C320-4DD1-BE4F-9C155C4E94B6@yahoo.com>
 <20150314114759.5fb7f8e3@anarchist.wooz.org>
Message-ID: <BAADCB62-415F-4105-9A5B-F24FDD42F63E@yahoo.com>

On Mar 14, 2015, at 8:47 AM, Barry Warsaw <barry at python.org> wrote:
> 
>> On Mar 14, 2015, at 08:03 AM, Andrew Barnert wrote:
>> 
>> Continuing to use python rather than python2 will work out of the box on
>> nearly every *nix system today, with no exceptions larger than Arch
>> Linux.
> 
> The part of the PEP aimed at Python distribution providers should recommend
> that `python2` be used in preference of just `python` and that the symlink be
> present.

Sure, that's exactly what it recommends. But you can't retroactively make the symlink present on, say, OS X 10.10 or CentOS 5, so what good does that recommendation actually do?

Maybe in 5-10 years we'll be at a point where you can count on the symlink being there on most *nix systems that have Python 2 pre-installed. Except, of course, that most *nix systems won't have Python 2 pre-installed anymore by then (and those that do will probably overlap quite a bit with the ones that don't have the symlink).

From barry at python.org  Mon Mar 16 00:06:34 2015
From: barry at python.org (Barry Warsaw)
Date: Sun, 15 Mar 2015 19:06:34 -0400
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <BAADCB62-415F-4105-9A5B-F24FDD42F63E@yahoo.com>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
 <B66187B2-C320-4DD1-BE4F-9C155C4E94B6@yahoo.com>
 <20150314114759.5fb7f8e3@anarchist.wooz.org>
 <BAADCB62-415F-4105-9A5B-F24FDD42F63E@yahoo.com>
Message-ID: <20150315190634.4a41c33c@marathon>

On Mar 14, 2015, at 08:50 PM, Andrew Barnert wrote:

>Maybe in 5-10 years we'll be at a point where you can count on the symlink
>being there on most *nix systems that have Python 2 pre-installed. Except, of
>course, that most *nix systems won't have Python 2 pre-installed anymore by
>then (and those that do will probably overlap quite a bit with the ones that
>don't have the symlink).

That's the point.  By the time Python 2.7 EOLs and (IMHO) we make any change
to /usr/bin/python, there will have been plenty of time to make sure anybody
running a modern OS version will have the python2 symlink.

Given how easy it is to add yourself, I'm not worried about legacy systems or
rebels who wouldn't add the link.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150315/0c8059c2/attachment.sig>

From tjreedy at udel.edu  Mon Mar 16 00:09:39 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 15 Mar 2015 19:09:39 -0400
Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars,
	for tkinter use
Message-ID: <me53fr$nsv$1@ger.gmane.org>

3.x comes with builtin ascii(obj) to return a string representation of 
obj that only has ascii characters.

 >>> s = 'a\xaa\ua000\U0001a000'
 >>> len(s)
4
 >>> sa = ascii(s)
 >>> sa, len(sa)
("'a\\xaa\\ua000\\U0001a000'", 23)

This allows any string to be printed on even a minimal ascii terminal.

Python also comes with the tkinter interface to tk.  Tk widgets are not 
limited to ascii but support the full BMP subset of unicode.  (This is 
better than Windows consoles limited by codepages.)  Thus, for use with 
tkinter, ascii() has two faults: it adds a quote at beginning and end of 
the string (like repr); it expands too much.

I looked at repr, which expands less, but it seems to be buggy in that 
it is not consistent in its handling of non-BMP chars.

 >>> s1 = 'a\xaa\ua000\U00011000'
 >>> sa = 'a\xaa\ua000\U0001a000'
 >>> s1r = repr(s1); len(s1r)
6
 >>> sar = repr(sa); len(sar)
15

'\U0001a000' gets expanded, and can be printed.
'\U00011000' does not, and cannot be consistently printed.

 >>> s1r  # only works at >>> prompt
"'a\xaa\ua000\U00011000'"
 >>> print(s1r)  # required in programs
Traceback (most recent call last):
   File "<pyshell#43>", line 1, in <module>
     print(s1r)
   File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write
     return self.shell.write(s, self.tags)
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 
4-4: Non-BMP character not supported in Tk

Printing s1 or sa directly, by either means, gives the same error. 
(Since '>>> expr'  is supposed to be the same as 'print(expr)' the above 
difference puzzles me.)

Even if repr always worked as it does for '\U0001a000', there would 
still be the problem of the added quotes.  I therefore proposed the 
addition of a new str method, such as 's.bmp()', that returns s with all 
non-BMP chars, and only such chars, expanded.  Since strings (in 
CPython) are internally marked by 'kind', the method would just return s 
when no expansion is needed.  I presume it could otherwise re-use the 
expansion code already in repr.

Aside from tkinter programmers in general, this issue bites Idle in at 
least two ways.  Internally, filenames can contain non-BMP chars and 
Idle displays them in 3 places.
See http://bugs.python.org/issue23672.
Externally, Idle users sometimes want to print strings with non-BMP 
chars.  I believe the automatic use of .bmp() with console prints could 
be user selectable.  There have been issues about this on both our 
tracker and StackOverflow.

I believe that the use of non-BMP chars is becoming more common and can 
no longer be simply dismissed as too rare to worry about.  Telling 
Windows users that they are better off than if they use python directly, 
with the windows console, does not solve the inability to print any 
Python string.  This proposal would.

-- 
Terry Jan Reedy


From cfkaran2 at gmail.com  Mon Mar 16 01:02:31 2015
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 15 Mar 2015 20:02:31 -0400
Subject: [Python-ideas] Modify inspect.getmembers() to return object members
	in source code order
Message-ID: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>

I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order.  For people that want the output in sorted order, the call would become 'sorted(inspect.getmembers())', which is relatively easy to do.  For me, and anyone else that is generating code from source code, the advantage is that the generated code is 'in the right order'.  As an example, consider generating unit tests for code that really should have had unit tests as the code was written, but didn't[1].  If inspect.getmembers() returned the elements in the same order as they appear in the source code, then we can generate code similar to the following[2]:

"""
members = inspect.getmembers(SomeoneNeedsToCleanThisUp)
names = [x[0] for x in members]

def test_gen(name):
    return ("    def test_{0!s}(self):\n".format(name) +
            "        raise NotImplementedError('He Who Shall Not Be Named, But Is On My Special List had better fill this in NOW!')\n" +
            "    # End of test_{0!s}()\n".format(name))

tests = "\n".join([test_gen(x) for x in names])
# Writing out names to the right location, etc.
"""

Thoughts?

Thanks,
Cem Karan

[1] This may possibly be the reason why I want to propose this change...
[2] Banged out in Mail.app very quickly, may have bugs, and definitely is not production code.

From storchaka at gmail.com  Mon Mar 16 01:03:03 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 16 Mar 2015 02:03:03 +0200
Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars,
	for tkinter use
In-Reply-To: <me53fr$nsv$1@ger.gmane.org>
References: <me53fr$nsv$1@ger.gmane.org>
Message-ID: <me56jo$5uv$1@ger.gmane.org>

On 16.03.15 01:09, Terry Reedy wrote:
> 3.x comes with builtin ascii(obj) to return a string representation of
> obj that only has ascii characters.
>
>  >>> s = 'a\xaa\ua000\U0001a000'
>  >>> len(s)
> 4
>  >>> sa = ascii(s)
>  >>> sa, len(sa)
> ("'a\\xaa\\ua000\\U0001a000'", 23)
>
> This allows any string to be printed on even a minimal ascii terminal.
>
> Python also comes with the tkinter interface to tk.  Tk widgets are not
> limited to ascii but support the full BMP subset of unicode.  (This is
> better than Windows consoles limited by codepages.)  Thus, for use with
> tkinter, ascii() has two faults: it adds a quote at beginning and end of
> the string (like repr); it expands too much.
>
> I looked at repr, which expands less, but it seems to be buggy in that
> it is not consistent in its handling of non-BMP chars.
>
>  >>> s1 = 'a\xaa\ua000\U00011000'
>  >>> sa = 'a\xaa\ua000\U0001a000'
>  >>> s1r = repr(s1); len(s1r)
> 6
>  >>> sar = repr(sa); len(sar)
> 15
>
> '\U0001a000' gets expanded, and can be printed.
> '\U00011000' does not, and cannot be consistently printed.
>
>  >>> s1r  # only works at >>> prompt
> "'a\xaa\ua000\U00011000'"
>  >>> print(s1r)  # required in programs
> Traceback (most recent call last):
>    File "<pyshell#43>", line 1, in <module>
>      print(s1r)
>    File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write
>      return self.shell.write(s, self.tags)
> UnicodeEncodeError: 'UCS-2' codec can't encode characters in position
> 4-4: Non-BMP character not supported in Tk
>
> Printing s1 or sa directly, by either means, gives the same error.
> (Since '>>> expr'  is supposed to be the same as 'print(expr)' the above
> difference puzzles me.)
>
> Even if repr always worked as it does for '\U0001a000', there would
> still be the problem of the added quotes.  I therefore proposed the
> addition of a new str method, such as 's.bmp()', that returns s with all
> non-BMP chars, and only such chars, expanded.  Since strings (in
> CPython) are internally marked by 'kind', the method would just return s
> when no expansion is needed.  I presume it could otherwise re-use the
> expansion code already in repr.
>
> Aside from tkinter programmers in general, this issue bites Idle in at
> least two ways.  Internally, filenames can contain non-BMP chars and
> Idle displays them in 3 places.
> See http://bugs.python.org/issue23672.
> Externally, Idle users sometimes want to print strings with non-BMP
> chars.  I believe the automatic use of .bmp() with console prints could
> be user selectable.  There have been issues about this on both our
> tracker and StackOverflow.
>
> I believe that the use of non-BMP chars is becoming more common and can
> no longer be simply dismissed as too rare to worry about.  Telling
> Windows users that they are better off than if they use python directly,
> with the windows console, does not solve the inability to print any
> Python string.  This proposal would.
>

Right now I'm writing a patch that implements similar idea for issue18814.

 >>> codecs.convert_astral('a\u20ac\U000e007f', 'strict')
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/home/serhiy/py/cpython/Lib/codecs.py", line 1159, in 
convert_astral
     'astral characters'))
UnicodeTranslateError: can't translate character '\U000e007f' in 
position 2: astral characters
 >>> codecs.convert_astral('a\u20ac\U000e007f', 'ignore')
'a?'
 >>> codecs.convert_astral('a\u20ac\U000e007f', 'replace')
'a??'
 >>> codecs.convert_astral('a\u20ac\U000e007f', 'backslashreplace')
'a?\\U000e007f'
 >>> codecs.convert_astral('a\u20ac\U000e007f', 'namereplace')
'a?\\N{CANCEL TAG}'
 >>> codecs.convert_astral('a\u20ac\U000e007f', 'xmlcharrefreplace')
'a?&#917631;'



From rosuav at gmail.com  Mon Mar 16 01:08:55 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Mar 2015 11:08:55 +1100
Subject: [Python-ideas] Modify inspect.getmembers() to return object
 members in source code order
In-Reply-To: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
Message-ID: <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order.
>

I'm not sure that that's really possible, unless you also are
proposing that OrderedDict be used internally instead of a regular
dict for all namespaces. That information doesn't exist.

> As an example, consider generating unit tests for code that really
> should have had unit tests as the code was written, but didn't

If I had to solve this problem, I'd parse the source code. Chances are
your functions/methods follow a straight-forward layout, so you should
be able to read them and parse them fairly easily.

ChrisA

From njs at pobox.com  Mon Mar 16 01:13:05 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 15 Mar 2015 17:13:05 -0700
Subject: [Python-ideas] Modify inspect.getmembers() to return object
 members in source code order
In-Reply-To: <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
 <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
Message-ID: <CAPJVwBnuAjqReghn+jUhJMT+x7VDUz_S1w4py96j+y+MghjCAg@mail.gmail.com>

Line number information is retained, so you could sort by that. In fact you
could do this without modifying inspect.getmembers...

-n
On Mar 15, 2015 5:09 PM, "Chris Angelico" <rosuav at gmail.com> wrote:

> On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> > I'd like to suggest that inspect.getmembers() be modified to return
> elements in the order they are defined in the source code rather than in
> alphabetical order.
> >
>
> I'm not sure that that's really possible, unless you also are
> proposing that OrderedDict be used internally instead of a regular
> dict for all namespaces. That information doesn't exist.
>
> > As an example, consider generating unit tests for code that really
> > should have had unit tests as the code was written, but didn't
>
> If I had to solve this problem, I'd parse the source code. Chances are
> your functions/methods follow a straight-forward layout, so you should
> be able to read them and parse them fairly easily.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150315/cefe82d7/attachment.html>

From random832 at fastmail.us  Mon Mar 16 01:23:53 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Sun, 15 Mar 2015 20:23:53 -0400
Subject: [Python-ideas] Modify inspect.getmembers() to return object
 members in source code order
In-Reply-To: <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
 <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
Message-ID: <1426465433.1977968.240744977.79DC0319@webmail.messagingengine.com>

On Sun, Mar 15, 2015, at 20:08, Chris Angelico wrote:
> I'm not sure that that's really possible, unless you also are
> proposing that OrderedDict be used internally instead of a regular
> dict for all namespaces. That information doesn't exist.

There are dict implementations where all items that were added with no
deletions starting from when the dict was empty will be returned by
iteration in the order they were originally added. Adding such an
implementation (or changing dict itself to have this guarantee) and
using it for various things like keyword args is perennially suggested.

From cfkaran2 at gmail.com  Mon Mar 16 01:29:25 2015
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 15 Mar 2015 20:29:25 -0400
Subject: [Python-ideas] Modify inspect.getmembers() to return object
	members in source code order
In-Reply-To: <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
 <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
Message-ID: <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com>


On Mar 15, 2015, at 8:08 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>> I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order.
>> 
> 
> I'm not sure that that's really possible, unless you also are
> proposing that OrderedDict be used internally instead of a regular
> dict for all namespaces. That information doesn't exist.

That was what I was afraid of.  

>> As an example, consider generating unit tests for code that really
>> should have had unit tests as the code was written, but didn't
> 
> If I had to solve this problem, I'd parse the source code. Chances are
> your functions/methods follow a straight-forward layout, so you should
> be able to read them and parse them fairly easily.

They are, and I can do so, but I got my hopes up that inspect.getmembers() could be modified easily.  Oh well, such is life.

Thanks,
Cem Karan

From rosuav at gmail.com  Mon Mar 16 01:37:48 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Mar 2015 11:37:48 +1100
Subject: [Python-ideas] Modify inspect.getmembers() to return object
 members in source code order
In-Reply-To: <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
 <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
 <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com>
Message-ID: <CAPTjJmqwPv-S6AK_2wTy6gwHQZz3Rjh-dN-dw_OoN3OR37JMuA@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:29 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> On Mar 15, 2015, at 8:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>>> I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order.
>>>
>>
>> I'm not sure that that's really possible, unless you also are
>> proposing that OrderedDict be used internally instead of a regular
>> dict for all namespaces. That information doesn't exist.
>
> That was what I was afraid of.

Nathaniel offered information which I hadn't known, which probably
solves your problem. You get line number info, and can sort by that -
just on the result of getmembers.

ChrisA

From cfkaran2 at gmail.com  Mon Mar 16 01:39:59 2015
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 15 Mar 2015 20:39:59 -0400
Subject: [Python-ideas] Modify inspect.getmembers() to return object
	members in source code order
In-Reply-To: <CAPJVwBnuAjqReghn+jUhJMT+x7VDUz_S1w4py96j+y+MghjCAg@mail.gmail.com>
References: <C291F07B-F1F4-4B5B-8520-B357DB12C420@gmail.com>
 <CAPTjJmonQVXe=_KcSWahBetjGG02iy16bm41dYPj4dhtu-cz6Q@mail.gmail.com>
 <CAPJVwBnuAjqReghn+jUhJMT+x7VDUz_S1w4py96j+y+MghjCAg@mail.gmail.com>
Message-ID: <CC48EBD3-47D9-4A9F-9509-F12828D4EA2B@gmail.com>


On Mar 15, 2015, at 8:13 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Line number information is retained, so you could sort by that. In fact you could do this without modifying inspect.getmembers...

You mean via a combinations of inspect.getmembers() and inspect.getsourcelines()?  Yeah, I think that could work.  I'll have to play around a bit to be sure I've got the hang of it, but I'm going to provisionally withdraw my suggestion about changing getmembers().  Thank you Nathaniel!

> -n
> 
> On Mar 15, 2015 5:09 PM, "Chris Angelico" <rosuav at gmail.com> wrote:
> On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> > I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order.
> >
> 
> I'm not sure that that's really possible, unless you also are
> proposing that OrderedDict be used internally instead of a regular
> dict for all namespaces. That information doesn't exist.
> 
> > As an example, consider generating unit tests for code that really
> > should have had unit tests as the code was written, but didn't
> 
> If I had to solve this problem, I'd parse the source code. Chances are
> your functions/methods follow a straight-forward layout, so you should
> be able to read them and parse them fairly easily.
> 
> ChrisA

Thanks,
Cem Karan

From abarnert at yahoo.com  Mon Mar 16 06:43:46 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 15 Mar 2015 22:43:46 -0700
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150315190634.4a41c33c@marathon>
References: <20150311194325.GA15390@python.ca>
 <CAEbHw4Y0Ek3cnnatehphhUoE1GcoFH7AbixN1ryx0T7BkOG5Lg@mail.gmail.com>
 <5500A7AE.9020400@egenix.com>
 <CAGE7PN+ucFKQcg+QGdTc1S4BqNsH8c=p-pFNp9D0oCZyrAxrTg@mail.gmail.com>
 <CALGmxEK+8Q+0ygMzxMA-C5t0ssyiXZE5PEH9p1fLU38nbECMAA@mail.gmail.com>
 <CAO41-mMO9nR6RNG0R0_1TkCiVWqo5-bRRBHMO58QRi-_B4r3WQ@mail.gmail.com>
 <CAN8d9gkBACbAeYprxBt819Lej20-dTUZ3kTLxFiSND9YaZ2kPw@mail.gmail.com>
 <B66187B2-C320-4DD1-BE4F-9C155C4E94B6@yahoo.com>
 <20150314114759.5fb7f8e3@anarchist.wooz.org>
 <BAADCB62-415F-4105-9A5B-F24FDD42F63E@yahoo.com>
 <20150315190634.4a41c33c@marathon>
Message-ID: <7512BEB1-7654-43C3-B35F-21F9638CD7EB@yahoo.com>

On Mar 15, 2015, at 4:06 PM, Barry Warsaw <barry at python.org> wrote:
> 
>> On Mar 14, 2015, at 08:50 PM, Andrew Barnert wrote:
>> 
>> Maybe in 5-10 years we'll be at a point where you can count on the symlink
>> being there on most *nix systems that have Python 2 pre-installed. Except, of
>> course, that most *nix systems won't have Python 2 pre-installed anymore by
>> then (and those that do will probably overlap quite a bit with the ones that
>> don't have the symlink).
> 
> That's the point.  By the time Python 2.7 EOLs and (IMHO) we make any change
> to /usr/bin/python, there will have been plenty of time to make sure anybody
> running a modern OS version will have the python2 symlink.

If you're suggesting that people distributing Python 2.7 scripts in 5-10 years should use python2, then sure, I agree. But you're suggesting they should do that today (or, if not you, then the PEP, and others in this thread). And there hasn't been plenty of time today.

And people shouldn't be distributing new Python 2.7 scripts in 5-10 years. If this is still a major issue when 2.7 EOLs, then 2.7 shouldn't EOL, and 3.x has been a failure. But I personally think that's unlikely. Most people will switch to 3 by then, and those who don't will mostly have special circumstances (e.g., they're adding to a server that has other components that require legacy software).

> Given how easy it is to add yourself, I'm not worried about legacy systems or
> rebels who wouldn't add the link.

People using their brand-new MacBook or a hosted CentOS server aren't rebels, and they're not using legacy systems. (They're also less likely to know how to deal with a problem than someone running a linux desktop or maintaining their own server.)


From abarnert at yahoo.com  Mon Mar 16 06:45:19 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 15 Mar 2015 22:45:19 -0700
Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars,
	for tkinter use
In-Reply-To: <me53fr$nsv$1@ger.gmane.org>
References: <me53fr$nsv$1@ger.gmane.org>
Message-ID: <B697DEAF-7B53-4AAC-93A0-BFD78FBBD4F4@yahoo.com>

It seems like what you really want here is a new codec, like unicode-escape but only escaping non-BMP characters, not a new repr-like function.

On Mar 15, 2015, at 4:09 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> 
> 3.x comes with builtin ascii(obj) to return a string representation of obj that only has ascii characters.
> 
> >>> s = 'a\xaa\ua000\U0001a000'
> >>> len(s)
> 4
> >>> sa = ascii(s)
> >>> sa, len(sa)
> ("'a\\xaa\\ua000\\U0001a000'", 23)
> 
> This allows any string to be printed on even a minimal ascii terminal.
> 
> Python also comes with the tkinter interface to tk.  Tk widgets are not limited to ascii but support the full BMP subset of unicode.  (This is better than Windows consoles limited by codepages.)  Thus, for use with tkinter, ascii() has two faults: it adds a quote at beginning and end of the string (like repr); it expands too much.
> 
> I looked at repr, which expands less, but it seems to be buggy in that it is not consistent in its handling of non-BMP chars.
> 
> >>> s1 = 'a\xaa\ua000\U00011000'
> >>> sa = 'a\xaa\ua000\U0001a000'
> >>> s1r = repr(s1); len(s1r)
> 6
> >>> sar = repr(sa); len(sar)
> 15
> 
> '\U0001a000' gets expanded, and can be printed.
> '\U00011000' does not, and cannot be consistently printed.
> 
> >>> s1r  # only works at >>> prompt
> "'a\xaa\ua000\U00011000'"
> >>> print(s1r)  # required in programs
> Traceback (most recent call last):
>  File "<pyshell#43>", line 1, in <module>
>    print(s1r)
>  File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write
>    return self.shell.write(s, self.tags)
> UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 4-4: Non-BMP character not supported in Tk
> 
> Printing s1 or sa directly, by either means, gives the same error. (Since '>>> expr'  is supposed to be the same as 'print(expr)' the above difference puzzles me.)
> 
> Even if repr always worked as it does for '\U0001a000', there would still be the problem of the added quotes.  I therefore proposed the addition of a new str method, such as 's.bmp()', that returns s with all non-BMP chars, and only such chars, expanded.  Since strings (in CPython) are internally marked by 'kind', the method would just return s when no expansion is needed.  I presume it could otherwise re-use the expansion code already in repr.
> 
> Aside from tkinter programmers in general, this issue bites Idle in at least two ways.  Internally, filenames can contain non-BMP chars and Idle displays them in 3 places.
> See http://bugs.python.org/issue23672.
> Externally, Idle users sometimes want to print strings with non-BMP chars.  I believe the automatic use of .bmp() with console prints could be user selectable.  There have been issues about this on both our tracker and StackOverflow.
> 
> I believe that the use of non-BMP chars is becoming more common and can no longer be simply dismissed as too rare to worry about.  Telling Windows users that they are better off than if they use python directly, with the windows console, does not solve the inability to print any Python string.  This proposal would.
> 
> -- 
> Terry Jan Reedy
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From ncoghlan at gmail.com  Mon Mar 16 09:27:29 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Mar 2015 18:27:29 +1000
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CAO41-mOVsEXdRMHnx2tjopuwRcG-YSJOXUcTV0dgCgFDC5FF2g@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
 <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
 <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>
 <CADiSq7fYgtEaZR=nyPvpzu=C7NrS5nmk-oiGRifSOd=HQiZFtg@mail.gmail.com>
 <CAO41-mOVsEXdRMHnx2tjopuwRcG-YSJOXUcTV0dgCgFDC5FF2g@mail.gmail.com>
Message-ID: <CADiSq7eyXBVfEqbFzHOXNmfQNjn=t7MX03nANVjQwAJAxLPiHA@mail.gmail.com>

On 14 Mar 2015 04:05, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
>
> If C99's ok, there's hope. :)

As far as I can recall, the main culprit there in relation to CPython
itself has long been MSVC's lack of support for C99, but the very fact we
use the native build chain on Windows means it isn't relevant to the choice
of build system on other platforms.

I may have forgotten another reason for the C89 requirement, but if I'm
right then a build tool for non-Windows systems could potentially be
allowed to require C99 support.

Regards,
Nick.

>
> C++'s main issue is that there are platforms that support C99 but have
out-of-date or non-complaint C++ compilers. I love the language (yes, I'm
weird), but it's tricky to have truly portable C++ programs because if all
the holes in various compilers. Just look at all the #ifdef's in Boost.
>
> On Fri, Mar 13, 2015 at 7:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 12 March 2015 at 04:30, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>> > On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon <brett at python.org>
wrote:
>> >>
>> >>
>> >>
>> >> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez <rymg19 at gmail.com>
wrote:
>> >>>
>> >>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan <ncoghlan at gmail.com>
wrote:
>> >>>>
>> >>>> On 10 March 2015 at 21:20, Russell Keith-Magee <
russell at keith-magee.com>
>> >>>> wrote:
>> >>>> >
>> >>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan <rymg19 at gmail.com> wrote:
>> >>>> >>
>> >>>> >> Russell Keith-Magee <russell at keith-magee.com> wrote:
>> >>>> >> >Hi Ryan,
>> >>>> >> >
>> >>>> >> >First off I'm no fan of autoconf, but...
>> >>>> >> >
>> >>>> >>
>> >>>> >> Who is other than the one who designed it? :)
>> >>>> >
>> >>>> > Let me be clear - there's a difference between "not a fan" and
>> >>>> > "calling for
>> >>>> > it's head". Autoconf is crufty, annoying, and sometimes
aggravating -
>> >>>> > but it
>> >>>> > *works*, and has had some pretty seriously battle testing - both
in
>> >>>> > the
>> >>>> > abstract "autoconf as a product", and in the specific "autoconf as
>> >>>> > used in
>> >>>> > Python". This isn't something that can be said of most of the
>> >>>> > alternatives
>> >>>> > in the abstract case, or *any* of the alternatives in the Python
case.
>> >>>>
>> >>>> Building architecture dependent software is a genuinely hard
problem,
>> >>>> which is why it's full of "80%" solutions like autotools and the
>> >>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but
>> >>>> they also basically do their job, so folks tend to develop a high
>> >>>> tolerance level for their quirks.
>> >>>>
>> >>>> In our particular case, we also have the fun problem of "How do you
>> >>>> build CPython locally when you don't have a working Python
interpreter
>> >>>> yet?". A number of the more interesting alternative C/C++ build
>> >>>> systems (like Meson, Scons, Waf and the Google "gyp" project that
>> >>>> powers the node-gyp native extension build system for Node.js)
require
>> >>>> an already working Python interpreter to get things started (when
>> >>>> starting from a completely blank slate, even our own build sequence
>> >>>> has to build a deliberately broken partial interpreter just to
freeze
>> >>>> importlib._bootstrap before going on to build a properly configured
>> >>>> one that can actually import modules).
>> >>>
>> >>>
>> >>> What about something like Google's new GN? It's designed to replace
Gyp
>> >>> in building Chromium and is written in C++.
>> >>>
>> >>> I'm pretty sure using a build system written in Lua to build Python
is
>> >>> kind of weird.
>> >>
>> >>
>> >>>
>> >>>
>> >>> With all that in mind...what exactly would be requirements that
Python
>> >>> would need in an autotools replacement?
>> >>
>> >>
>> >> Can't require anything more than an ANSI C compiler basically. People
run
>> >> Python in places where they don't have much more than gcc.
>> >
>> >
>> > Not even C99? Darn it...
>>
>> C99 would probably be OK these days. If there's a good enough
>> cross-compiling story, I'm also not sure we should entirely rule out
>> C++, as it's potentially OK if *developing* CPython requires a
>> mainstream platform with a C++ compiler.
>>
>> But yeah, you can see why folks look at this problem and end up
>> thinking "well, you know, maybe autotools isn't *that* bad..."
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
>
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
> http://kirbyfan64.github.io/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/a230fc5b/attachment-0001.html>

From solipsis at pitrou.net  Mon Mar 16 09:55:30 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Mar 2015 09:55:30 +0100
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CAJxq848Sxqo1meFuTrh+K0rxHJJqDJeWAgFp4esT7y93-czm1Q@mail.gmail.com>
 <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com>
 <CAJxq84_SRzUXn543OZbT-ScuPKhUii7+pCB2Yr7LhA9Qq1yUeg@mail.gmail.com>
 <CADiSq7cP5ktuQ6SLBdGXtOq2GzV9zCJYTNE2wAFOdZbNy=Qp0g@mail.gmail.com>
 <CAO41-mO3j=tRZ=7OPa8=SDYXCLOH150ajxQDM4z7BOuX2-0HKA@mail.gmail.com>
 <CAP1=2W5oJnhURxSpwbMQZ+ebcBrP6wzFA6_jOGA84GN4euZ7rg@mail.gmail.com>
 <CAO41-mMZ5xa=MYaUkGrEz9qsixCpjUo=K-gDp9BdmmY2k44_PQ@mail.gmail.com>
 <CADiSq7fYgtEaZR=nyPvpzu=C7NrS5nmk-oiGRifSOd=HQiZFtg@mail.gmail.com>
 <CAO41-mOVsEXdRMHnx2tjopuwRcG-YSJOXUcTV0dgCgFDC5FF2g@mail.gmail.com>
 <CADiSq7eyXBVfEqbFzHOXNmfQNjn=t7MX03nANVjQwAJAxLPiHA@mail.gmail.com>
Message-ID: <20150316095530.548e590b@fsol>

On Mon, 16 Mar 2015 18:27:29 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 14 Mar 2015 04:05, "Ryan Gonzalez" <rymg19 at gmail.com> wrote:
> >
> > If C99's ok, there's hope. :)
> 
> As far as I can recall, the main culprit there in relation to CPython
> itself has long been MSVC's lack of support for C99, but the very fact we
> use the native build chain on Windows means it isn't relevant to the choice
> of build system on other platforms.

VS 2015 (which Python 3.5 is gonna use) should support at least a
subset of C99.

> I may have forgotten another reason for the C89 requirement, but if I'm
> right then a build tool for non-Windows systems could potentially be
> allowed to require C99 support.

Other (legacy? enterprise?) C compilers may not support C99, but at this
point we can probably declare them unsupported.

Regards

Antoine.



From victor.stinner at gmail.com  Mon Mar 16 11:39:29 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Mar 2015 11:39:29 +0100
Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars,
 for tkinter use
In-Reply-To: <B697DEAF-7B53-4AAC-93A0-BFD78FBBD4F4@yahoo.com>
References: <me53fr$nsv$1@ger.gmane.org>
 <B697DEAF-7B53-4AAC-93A0-BFD78FBBD4F4@yahoo.com>
Message-ID: <CAMpsgwakAqQhvQ_mEt-VHPLWLwGpowFxgeYxGuGHLskstPfUPg@mail.gmail.com>

I agree that a codec would be more appropriate. See:
http://bugs.python.org/issue18814

Victor

2015-03-16 6:45 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
> It seems like what you really want here is a new codec, like unicode-escape but only escaping non-BMP characters, not a new repr-like function.
>
> On Mar 15, 2015, at 4:09 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> 3.x comes with builtin ascii(obj) to return a string representation of obj that only has ascii characters.
>>
>> >>> s = 'a\xaa\ua000\U0001a000'
>> >>> len(s)
>> 4
>> >>> sa = ascii(s)
>> >>> sa, len(sa)
>> ("'a\\xaa\\ua000\\U0001a000'", 23)
>>
>> This allows any string to be printed on even a minimal ascii terminal.
>>
>> Python also comes with the tkinter interface to tk.  Tk widgets are not limited to ascii but support the full BMP subset of unicode.  (This is better than Windows consoles limited by codepages.)  Thus, for use with tkinter, ascii() has two faults: it adds a quote at beginning and end of the string (like repr); it expands too much.
>>
>> I looked at repr, which expands less, but it seems to be buggy in that it is not consistent in its handling of non-BMP chars.
>>
>> >>> s1 = 'a\xaa\ua000\U00011000'
>> >>> sa = 'a\xaa\ua000\U0001a000'
>> >>> s1r = repr(s1); len(s1r)
>> 6
>> >>> sar = repr(sa); len(sar)
>> 15
>>
>> '\U0001a000' gets expanded, and can be printed.
>> '\U00011000' does not, and cannot be consistently printed.
>>
>> >>> s1r  # only works at >>> prompt
>> "'a\xaa\ua000\U00011000'"
>> >>> print(s1r)  # required in programs
>> Traceback (most recent call last):
>>  File "<pyshell#43>", line 1, in <module>
>>    print(s1r)
>>  File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write
>>    return self.shell.write(s, self.tags)
>> UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 4-4: Non-BMP character not supported in Tk
>>
>> Printing s1 or sa directly, by either means, gives the same error. (Since '>>> expr'  is supposed to be the same as 'print(expr)' the above difference puzzles me.)
>>
>> Even if repr always worked as it does for '\U0001a000', there would still be the problem of the added quotes.  I therefore proposed the addition of a new str method, such as 's.bmp()', that returns s with all non-BMP chars, and only such chars, expanded.  Since strings (in CPython) are internally marked by 'kind', the method would just return s when no expansion is needed.  I presume it could otherwise re-use the expansion code already in repr.
>>
>> Aside from tkinter programmers in general, this issue bites Idle in at least two ways.  Internally, filenames can contain non-BMP chars and Idle displays them in 3 places.
>> See http://bugs.python.org/issue23672.
>> Externally, Idle users sometimes want to print strings with non-BMP chars.  I believe the automatic use of .bmp() with console prints could be user selectable.  There have been issues about this on both our tracker and StackOverflow.
>>
>> I believe that the use of non-BMP chars is becoming more common and can no longer be simply dismissed as too rare to worry about.  Telling Windows users that they are better off than if they use python directly, with the windows console, does not solve the inability to print any Python string.  This proposal would.
>>
>> --
>> Terry Jan Reedy
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From victor.stinner at gmail.com  Mon Mar 16 11:55:03 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Mar 2015 11:55:03 +0100
Subject: [Python-ideas] About people's accounts being deactivated due to
	bounces
In-Reply-To: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
References: <CAP1=2W4GOwOTEn4ZXR23CpHvGjDWvkjCQexx9Hei+oj6LRYSHQ@mail.gmail.com>
Message-ID: <CAMpsgwaV3FaYDXKbDRc2eo=oHpC=zvqLHdPPApy83TyaHMzJMQ@mail.gmail.com>

Hi,

Be careful: recently, many emails sent to python-ideas and
core-mentorship were sorted as spam in my Gmail account. It may be
related to the recent upgrade of Mailman of these mailing lists.

Victor

2015-03-13 13:27 GMT+01:00 Brett Cannon <bcannon at gmail.com>:
> We're trying to figure out what's going on. Just re-enable your account if
> you got a notice that it was temporarily disabled.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From random832 at fastmail.us  Mon Mar 16 14:55:59 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 16 Mar 2015 09:55:59 -0400
Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars,
 for tkinter use
In-Reply-To: <me53fr$nsv$1@ger.gmane.org>
References: <me53fr$nsv$1@ger.gmane.org>
Message-ID: <1426514159.1203554.240988925.5344514A@webmail.messagingengine.com>

On Sun, Mar 15, 2015, at 19:09, Terry Reedy wrote:
> Tk widgets ... support the full BMP subset of unicode.

Is it actually true that Tk widgets cannot support non-BMP characters?
On my Python 2.7 on Windows (which is a narrow build, and therefore is
passing in a UTF-16 string containing surrogates), it appears to work
just fine. The character itself appears as a box, because there is no
font support, but it is one box, not two. It seems like on the versions
where people are getting the error message, Tkinter is arbitrarily (i.e.
not based on a real limitation in Tk) refusing to convert extended
characters to surrogates.

From techtonik at gmail.com  Mon Mar 16 13:12:44 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 15:12:44 +0300
Subject: [Python-ideas] math.tau
Message-ID: <CAPkN8xKA5weB=T1QOhpjanzO7u4FGajsUzFYkvKrOU2e8toRUQ@mail.gmail.com>

>>> import math
>>> math.tau
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'tau'
>>> math.pi * 2
6.283185307179586

https://www.youtube.com/watch?v=2hhjsSN-AiU

=)

From techtonik at gmail.com  Mon Mar 16 14:09:36 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 16:09:36 +0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
Message-ID: <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>

On Sun, Mar 8, 2015 at 4:07 PM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
>>
>> On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico <rosuav at gmail.com> wrote:
>> > If it says "decode", the result is a Unicode string. If it says
>> > "encode", the result is bytes. I'm not sure what is difficult here.
>>
>> Yep. When I teach, I use this mnemonic, which I can now quote from my
>> book [1] ;-)
>>
>> [TIP]
>> ====
>> If you need a memory aid to distinguish `.decode()` from `.encode()`,
>> convince yourself that a Unicode `str` contains "human" text, while
>> byte sequences can be cryptic machine core dumps. Therefore, it makes
>> sense that we *decode* `bytes` to `str` to get human readable text,
>> and we *encode* text to `bytes` for storage or transmission.
>> ====
>
>
> This is a great advice (and yes, I also often get confused by the two).

I use the analogy of "encoding abstract numbers into specific bytes" myself.
It is still hard to review the source, which may use it wrong.

db.get(query).from('utf-8') gives me a clear indication that a person expects
that db contents is in 'utf-8'.

db.get(query).decode('utf-8') gives me a feeling that query result should be
decoded from whatever format it is in to 'utf-8'.

So if Python supported .from and .to, the part about encoding wouldn't feel
like math.

From techtonik at gmail.com  Mon Mar 16 14:24:05 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 16:24:05 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
Message-ID: <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>

On Fri, Mar 13, 2015 at 10:46 PM, Wes Turner <wes.turner at gmail.com> wrote:
> The subject line mentions "Part 2". Is there a link to "Part 1"? What is the
> rationale for changing?
>
> Make, autoconf, and automake are already widely ported/portable.

"Autotools has reached the Chandrasekhar mass limit of software and
become a black hole of suck that consumes endless hours of bright
peoples? time without any good result. It is well past time for it to
die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877

I couldn't sum up this better. It looks even worse from Windows machine.

> What are the criteria for evaluating alternatives?

Python 3 support and beautiful output to console? =)

From techtonik at gmail.com  Mon Mar 16 14:43:40 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 16 Mar 2015 16:43:40 +0300
Subject: [Python-ideas] CPython needs to revamp/replace it's build
 system && Cross-compiling Python sucks
In-Reply-To: <CABicbJ+q7urU_dQqoZtk-NLe=p1_nufm-ia4HDJOWGvtqjYy7w@mail.gmail.com>
References: <CAO41-mPqhGyGq_m1=DxqS7c=nNORvDE=sVBzRa-qSEHT5S5Qxg@mail.gmail.com>
 <CABicbJ+q7urU_dQqoZtk-NLe=p1_nufm-ia4HDJOWGvtqjYy7w@mail.gmail.com>
Message-ID: <CAPkN8xJth9pLCTjS4zt9OHZuFdHGybN7e1eD_m6ETeF03k5YOg@mail.gmail.com>

On Fri, Mar 13, 2015 at 10:47 PM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>> I've been trying to work on Issue 23496. And want to tear my hair out. I've
>> noticed a few problems with Python's current build system:
>>
>> - There is no separation of the host C compiler and the target C compiler.
>> When you give configure a C compiler for cross-compiling, make will happily
>> try to build Parser/pgen with the cross-compiler. And try to run the
>> generated executable. And fail. I know you can use `make touch` to get
>> around this, but it still sucks.
>
> I just want to echo this. I had really horrible problems trying to get
> Python working with emscripten a year ago, because of this and other
> problems. (Gave up.)  It turned out emscripten was a bigger cause for
> complaint, but, still.

There is an awesome SCons extension called Parts, but is not integrated
back due to a lack of time and/or funding. Here is the lightning talk:
https://www.youtube.com/watch?v=Ft6-r_vksDI#t=1188

http://parts.stage.tigris.org/
-- 
anatoly t.

From bcannon at gmail.com  Mon Mar 16 16:11:09 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Mon, 16 Mar 2015 15:11:09 +0000
Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b`
Message-ID: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>

One of the rather subtle issues with writing Python 2/3 code is that
indexing on bytes in Python 2 returns a length-1 bytes object while in
Python 3 it returns an int. Because ==/!= always returns True/False it can
be a very subtle failure and tough to track down.

What do people think of extending -b/-bb in Python 3 to warn when
performing equality between an int and a bytes object of any length? I
don't want to restrict to length-1 bytes objects because people may be
doing comparisons where the result can be length-1 or any other length and
thus would still have a subtle bug to pick up. Do people think this would
raise a ton of false-positives? Would people find it useful?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/723cca66/attachment.html>

From victor.stinner at gmail.com  Mon Mar 16 16:15:01 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Mar 2015 16:15:01 +0100
Subject: [Python-ideas] math.tau
In-Reply-To: <CAPkN8xKA5weB=T1QOhpjanzO7u4FGajsUzFYkvKrOU2e8toRUQ@mail.gmail.com>
References: <CAPkN8xKA5weB=T1QOhpjanzO7u4FGajsUzFYkvKrOU2e8toRUQ@mail.gmail.com>
Message-ID: <CAMpsgwaPXmXsUjTd_ZiJ+ecAVMyhyFuDrjzAKd3oLfMdTsOAFQ@mail.gmail.com>

Hi Anatoly,

It looks like you forgot to write the body of your email, there is no
text, only code?

2015-03-16 13:12 GMT+01:00 anatoly techtonik <techtonik at gmail.com>:
>>>> import math
>>>> math.tau
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> AttributeError: 'module' object has no attribute 'tau'
>>>> math.pi * 2
> 6.283185307179586

Once a year, someone asks for math.tau and then disappears. I don't
think that there is a real need to add math.tau. It's really trivial
to compute such constant in your own application or in the Python
interpreter. They are much more math constants, it's not useful to add
more constants to the math modules. We added recently math.inf and
math.nan which are float("int") and float("nan")! Even if I'm not
convinced that it was useful to add them :-p
http://bugs.python.org/issue23185
(see also http://bugs.python.org/issue23229 for cmath)

The previous attempt to add math.tau was rejected:
http://bugs.python.org/issue12345

Please read the issue for the rationale.

Victor

From guido at python.org  Mon Mar 16 16:16:22 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Mar 2015 08:16:22 -0700
Subject: [Python-ideas] Warn about comparing bytes to int for `python3
	-b`
In-Reply-To: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
References: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
Message-ID: <CAP7+vJ+5+BQ-kV2kxOV6PDy3mjyx3tWvO1c+zCb8OzkNZOZO9g@mail.gmail.com>

As long as it's part of -b/-bb this sounds like a useful (though small) bit
of help for people in the last throes of porting a big package to PY3. As
for how many false positives it will trigger, who knows? The most likely
case would be if people use dicts whose keys can be bytestrings or ints --
I know that's a popular hobby when it comes to str/int, but I don't know if
it's also common with bytes/int. I guess the only way to find out is to
build and release it.

On Mon, Mar 16, 2015 at 8:11 AM, Brett Cannon <bcannon at gmail.com> wrote:

> One of the rather subtle issues with writing Python 2/3 code is that
> indexing on bytes in Python 2 returns a length-1 bytes object while in
> Python 3 it returns an int. Because ==/!= always returns True/False it can
> be a very subtle failure and tough to track down.
>
> What do people think of extending -b/-bb in Python 3 to warn when
> performing equality between an int and a bytes object of any length? I
> don't want to restrict to length-1 bytes objects because people may be
> doing comparisons where the result can be length-1 or any other length and
> thus would still have a subtle bug to pick up. Do people think this would
> raise a ton of false-positives? Would people find it useful?
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/c142dadf/attachment.html>

From victor.stinner at gmail.com  Mon Mar 16 16:21:57 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Mar 2015 16:21:57 +0100
Subject: [Python-ideas] Warn about comparing bytes to int for `python3
	-b`
In-Reply-To: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
References: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
Message-ID: <CAMpsgwbW0NaZMpeSxJ4VibRAQ=R04RNrzXKSMY-31198PYWvkA@mail.gmail.com>

Hi,

2015-03-16 16:11 GMT+01:00 Brett Cannon <bcannon at gmail.com>:
> One of the rather subtle issues with writing Python 2/3 code is that
> indexing on bytes in Python 2 returns a length-1 bytes object while in
> Python 3 it returns an int. Because ==/!= always returns True/False it can
> be a very subtle failure and tough to track down.

I worked on such patch in the past, but I lost it :-) I can help to
rewrite it if needed.

So yes, it *is* very useful to port a large Python 2 project to Python 3.

For example, you may not be able to run your application with Python 3
because a third party library cannot be imported on Python 3, so it
blocks the whole work on porting an application to Python 3. Until the
module is ported, you may want to prepare the port. Checking
bytes==str just by reading the source code is difficult.

Other issues which can only be "seen" at runtime when running an
application on Python 3 :

- "x > 0" with x=None => TypeError is raised in Python 3
- x / 8 where x is an int => becomes a float in Python 3, it's hard to
detect this issue in Python 2 just by reading the source code :-/

> What do people think of extending -b/-bb in Python 3 to warn when performing
> equality between an int and a bytes object of any length? I don't want to
> restrict to length-1 bytes objects because people may be doing comparisons
> where the result can be length-1 or any other length and thus would still
> have a subtle bug to pick up. Do people think this would raise a ton of
> false-positives? Would people find it useful?

First ensure that the stdlib doesn't raise any BytesWarning exception.

For example, os.get_exec_path() has to modify warnings filters
temporary in Python 3 :-/

    # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a
    # BytesWarning when using python -b or python -bb: ignore the warning

Victor

From chris.barker at noaa.gov  Mon Mar 16 17:27:14 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 09:27:14 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
Message-ID: <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>

>
> >> That written in Python, so Python would require Python to build which
> >> would require Python to build which would require...
>

A bit more here:

It seems there are two cases:

1) Development on and for a common, mature platform:

In this case finding a functional python with which to run the build tool
sis trivial. We're all set, no need to exclude python-based tools.

2) Development for / porting to a new and/or obscure platform:

In this case, then there is no existing python with which to run the build
tools. And there are two ways you might go about this (in general):

  a) cross-compiling -- run the compiler on a existing mature platform, and
generate a compiled python for the new platform. IN this case, the build
tools only need to run on the development platform, so we are back to (1),
and all set. Note that the hard part of cross-compiling is still done by
the C compiler, not really the build/configuration tools, The build tools
need to be able to set up the cross-compile -- but presumable a new tool is
chosen be cause it's easier to do things like that with it...

  b) Native compiling -- compiling python on the new platform. This is the
tricky one. In this case, the entire goal is to get python building/running
on the new platform. Clearly you can't just use the nifty python-based
build system to do that. But as that's the goal, you'll still need to do
the tricky parts anyway. So you could cross-compile the first version --
with maybe as few modules as possible to get started. Another option is a
build system that needs python for the configuration stage, but not the
compilation stage -- kind of like autotools: autotools is used in a
three-step process:

  i) run autoconf, etc, to generate a configure script -- this is often
distributed with the source tarball, so that end users don't need the
autoconf tools

 ii) run the configure script to generate Makefiles, etc. configured for
the platform at hand.

 iii) run make to do the actual build.

Usually the last two steps are run natively -- but it's not too much of a
stretch to run i and ii on a known build machine, and then only ned to run
make on the final target. That would require the build tools to have a
cross-configure ability.

I have no idea if any of the known tools can do this, but if so, it would
be pretty usable.

And also worth keeping in mind that truly new platforms are pretty rare --
this kind of bootstrapping would not have to be done often --not a killer
if it takes some hand-work.

But the grain of salt : I have neer and am unlikely ever, to try to port a
significant project to a new platform. Folks that actually need to do this
should make these decisions.

Will this affect the commands required to build Python from source, or
> is all this just for actual development?
>

well, it may or may not be both:

If the system supported a three-step process I outlined above, then
makefiles (or the like) could be shipped for the common platforms, though
probably easier to run the configure step on the final target. But the vast
majority of final targets do have a python ready to run the configure/build
machinery.

 someone
> who has little knowledge of Python's internals, and just wants to
> build the latest version (either from hg or from a source tarball).
>

That's the assumption I'm following -- the user wants the latest version --
so probably already has easy access to an older version with which to run
the build tools...

But in the second case, it'd be much safer to minimize the
>
requirements. If you can untar the source and just "./configure; make;
> sudo make install" your way to having the latest Python, that's a LOT
> easier than I've just been doing trying to get my Dad onto a newer
> Mozilla Thunderbird. (I don't understand what it's doing all along
> there, but it pulls a whole bunch of separate repositories when I say
> update, and there's a separate bootstrap step, and ... I dunno.)
>

requiring _some_, not very recent, python is not exactly

 """a whole bunch of separate repositories when I say update, and there's a
separate bootstrap step"""

If there is an already build python (like OS-X, any general purpose linux
distro, Windows if you don't mind point and clicking the python.org
installer, etc, then it's not really a "bootstrap".

In short -- using a python-based build system would make porting to a
brand-new platform trickier -- but not really affect most other building.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/a8b611d9/attachment-0001.html>

From chris.barker at noaa.gov  Mon Mar 16 17:41:05 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 09:41:05 -0700
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
Message-ID: <CALGmxELSc7u6ozNDKEQtoNMWeCxwOur91hrnu4VikDP9y66ZRQ@mail.gmail.com>

On Mon, Mar 16, 2015 at 6:09 AM, anatoly techtonik <techtonik at gmail.com>
wrote:

> db.get(query).from('utf-8') gives me a clear indication that a person
> expects
> that db contents is in 'utf-8'.
>
> db.get(query).decode('utf-8') gives me a feeling that query result should
> be
> decoded from whatever format it is in to 'utf-8'.
>

In teaching this, yes, newbies do get confused about what "decode" vs.
"encode" mean. The fact that py2 strings have BOTH methods does confuse the
heck out of it! -- but that problem is solved in py3. And in this case
"from" might be a bit more obvious to a newbie. (though only might -- hard
to tell without usability testing...)

But "decode" and "encode" are widely used terms that clearly specify what's
going on -- I think that even with teh "from", it's clear to you because
you already understand a bit about the concept of unicode and the
distiction between a string object and encoded bytes. And once you get
that, then you kind know what "encode" and "decode" mean.

Hopefully, folks will get that "decoded to utf-8" is meaningless...

I suppose we could have used "decode_from" and "encode_to" in the first
place to be really clear, but I don't think it's worth making that change
now.

-CHB











> So if Python supported .from and .to, the part about encoding wouldn't feel
> like math.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/07bed17f/attachment.html>

From rosuav at gmail.com  Mon Mar 16 17:50:06 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 17 Mar 2015 03:50:06 +1100
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
Message-ID: <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>

On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>>  someone
>> who has little knowledge of Python's internals, and just wants to
>> build the latest version (either from hg or from a source tarball).
>
>
> That's the assumption I'm following -- the user wants the latest version --
> so probably already has easy access to an older version with which to run
> the build tools...

The tricky part here is defining the oldest supported version. "To
build Python X.Y, you must already have Python ??? or later installed"
- how do you pick the minimum? The older that minimum, the more warped
the build code has to be (especially if you want to support 2.x as
well as 3.x - imagine trying to make it possible to use the system
Python on RHEL, as well as any 3.[1-4]), but go too far the other way
and you end up with a multi-step build process ("first build Python
3.4, then use that to build Python 3.7") which is just going to be a
pain.

>> But in the second case, it'd be much safer to minimize the
>> requirements. If you can untar the source and just "./configure; make;
>> sudo make install" your way to having the latest Python, that's a LOT
>> easier than I've just been doing trying to get my Dad onto a newer
>> Mozilla Thunderbird. (I don't understand what it's doing all along
>> there, but it pulls a whole bunch of separate repositories when I say
>> update, and there's a separate bootstrap step, and ... I dunno.)
>
>
> requiring _some_, not very recent, python is not exactly
>
>  """a whole bunch of separate repositories when I say update, and there's a
> separate bootstrap step"""
>
> If there is an already build python (like OS-X, any general purpose linux
> distro, Windows if you don't mind point and clicking the python.org
> installer, etc, then it's not really a "bootstrap".

No, that's not what I'm talking about. I'm talking about trying to
build Thunderbird and having to do a bunch of steps where I run this
program, then run that, then let it chug its way through stuff, and
then eventually it'll start building. Without detailed knowledge of
the build internals I have no way of knowing what steps I can skip, so
every time I want to test the build again, I have to go through them
all. It took three parts of forever to figure out what was going on.

I much prefer projects that ship with the standard "./configure; make"
pattern. I don't care if the configure script is built by some arcane
incantation, as long as it's checked into source control (or at very
least is in the downloadable source tarball for any particular
version). If I want to build your program, I don't want to have to
hunt down your notes file, read it through in detail, and jump through
a dozen hoops; I just want to point my compiler at your source, and
then go make myself a hot chocolate while it does its work.

ChrisA

From rosuav at gmail.com  Mon Mar 16 17:55:00 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 17 Mar 2015 03:55:00 +1100
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CALGmxELSc7u6ozNDKEQtoNMWeCxwOur91hrnu4VikDP9y66ZRQ@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
 <CALGmxELSc7u6ozNDKEQtoNMWeCxwOur91hrnu4VikDP9y66ZRQ@mail.gmail.com>
Message-ID: <CAPTjJmrNPOeHuo6AGCEwAnUwr=uJbzOJNgRxU4WLxu1XTZ5MKw@mail.gmail.com>

On Tue, Mar 17, 2015 at 3:41 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> I suppose we could have used "decode_from" and "encode_to" in the first
> place to be really clear, but I don't think it's worth making that change
> now.

I don't think it'd help much. It'd just be noise words, at best. What
you're doing is encoding or decoding using a particular scheme. What
scheme? That's the parameter. You're encoding to bytes and decoding to
text; and in order to perform that transformation, you use a standard
technique by the name of UTF-8. The result of "spam".encode("UTF-8")
is not a UTF-8, it's a stream of bytes. So there'd be just as much
confusion the other way around.

*shrug*

ChrisA

From bcannon at gmail.com  Mon Mar 16 18:06:44 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Mon, 16 Mar 2015 17:06:44 +0000
Subject: [Python-ideas] Warn about comparing bytes to int for `python3
	-b`
In-Reply-To: <CAP7+vJ+5+BQ-kV2kxOV6PDy3mjyx3tWvO1c+zCb8OzkNZOZO9g@mail.gmail.com>
References: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
 <CAP7+vJ+5+BQ-kV2kxOV6PDy3mjyx3tWvO1c+zCb8OzkNZOZO9g@mail.gmail.com>
Message-ID: <CAP1=2W7HZ-p66o0jCOWTbjvVHNaELuDXNLON6RETXUju6n=Msg@mail.gmail.com>

http://bugs.python.org/issue23681 to track this for Python 3.5.

On Mon, Mar 16, 2015 at 11:16 AM Guido van Rossum <guido at python.org> wrote:

> As long as it's part of -b/-bb this sounds like a useful (though small)
> bit of help for people in the last throes of porting a big package to PY3.
> As for how many false positives it will trigger, who knows? The most likely
> case would be if people use dicts whose keys can be bytestrings or ints --
> I know that's a popular hobby when it comes to str/int, but I don't know if
> it's also common with bytes/int. I guess the only way to find out is to
> build and release it.
>
> On Mon, Mar 16, 2015 at 8:11 AM, Brett Cannon <bcannon at gmail.com> wrote:
>
>> One of the rather subtle issues with writing Python 2/3 code is that
>> indexing on bytes in Python 2 returns a length-1 bytes object while in
>> Python 3 it returns an int. Because ==/!= always returns True/False it can
>> be a very subtle failure and tough to track down.
>>
>> What do people think of extending -b/-bb in Python 3 to warn when
>> performing equality between an int and a bytes object of any length? I
>> don't want to restrict to length-1 bytes objects because people may be
>> doing comparisons where the result can be length-1 or any other length and
>> thus would still have a subtle bug to pick up. Do people think this would
>> raise a ton of false-positives? Would people find it useful?
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/e8a154c4/attachment-0001.html>

From rymg19 at gmail.com  Mon Mar 16 18:18:22 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 12:18:22 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
Message-ID: <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:50 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker <chris.barker at noaa.gov>
> wrote:
> >>  someone
> >> who has little knowledge of Python's internals, and just wants to
> >> build the latest version (either from hg or from a source tarball).
> >
> >
> > That's the assumption I'm following -- the user wants the latest version
> --
> > so probably already has easy access to an older version with which to run
> > the build tools...
>
> The tricky part here is defining the oldest supported version. "To
> build Python X.Y, you must already have Python ??? or later installed"
> - how do you pick the minimum? The older that minimum, the more warped
> the build code has to be (especially if you want to support 2.x as
> well as 3.x - imagine trying to make it possible to use the system
> Python on RHEL, as well as any 3.[1-4]), but go too far the other way
> and you end up with a multi-step build process ("first build Python
> 3.4, then use that to build Python 3.7") which is just going to be a
> pain.
>
>
Sounds like Go is now. It's a pain.


> >> But in the second case, it'd be much safer to minimize the
> >> requirements. If you can untar the source and just "./configure; make;
> >> sudo make install" your way to having the latest Python, that's a LOT
> >> easier than I've just been doing trying to get my Dad onto a newer
> >> Mozilla Thunderbird. (I don't understand what it's doing all along
> >> there, but it pulls a whole bunch of separate repositories when I say
> >> update, and there's a separate bootstrap step, and ... I dunno.)
> >
> >
> > requiring _some_, not very recent, python is not exactly
> >
> >  """a whole bunch of separate repositories when I say update, and
> there's a
> > separate bootstrap step"""
> >
> > If there is an already build python (like OS-X, any general purpose linux
> > distro, Windows if you don't mind point and clicking the python.org
> > installer, etc, then it's not really a "bootstrap".
>
> No, that's not what I'm talking about. I'm talking about trying to
> build Thunderbird and having to do a bunch of steps where I run this
> program, then run that, then let it chug its way through stuff, and
> then eventually it'll start building. Without detailed knowledge of
> the build internals I have no way of knowing what steps I can skip, so
> every time I want to test the build again, I have to go through them
> all. It took three parts of forever to figure out what was going on.
>
>
All of Mozilla's build systems need help. They have a build system genius
who wrote Tup working for them, and they somehow still have 20,000 stupid
mozbuild files with some weird, intensely slow make-based system that
everyone complains about.


> I much prefer projects that ship with the standard "./configure; make"
> pattern. I don't care if the configure script is built by some arcane
> incantation, as long as it's checked into source control (or at very
> least is in the downloadable source tarball for any particular
> version). If I want to build your program, I don't want to have to
> hunt down your notes file, read it through in detail, and jump through
> a dozen hoops; I just want to point my compiler at your source, and
> then go make myself a hot chocolate while it does its work.
>
>
Most of that is bad documentation. If there's no configure script but a
file labeled INSTALLING, then that should contain simple instructions to
build.

I hate most tools that ship with configure scripts because 70% of the time
some stupid test does some stupid thing wrong and gives the worst errors,
29% of the time the configure system doesn't pick up my options correctly,
and 1% of the time it actually *works*.

SCons also constantly gives me trouble.

Of course, you could always do it the plan9port way:

ryan at DevPC-LX:~/stuff/plan9port$ ./configure
read the README file.
ryan at DevPC-LX:~/stuff/plan9port$


> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/c6463fbd/attachment.html>

From bcannon at gmail.com  Mon Mar 16 18:25:58 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Mon, 16 Mar 2015 17:25:58 +0000
Subject: [Python-ideas] Warn about comparing bytes to int for `python3
	-b`
In-Reply-To: <CAMpsgwbW0NaZMpeSxJ4VibRAQ=R04RNrzXKSMY-31198PYWvkA@mail.gmail.com>
References: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
 <CAMpsgwbW0NaZMpeSxJ4VibRAQ=R04RNrzXKSMY-31198PYWvkA@mail.gmail.com>
Message-ID: <CAP1=2W4=ZO1Ouf7GwCfgTTzkONe1ULniW70rq2axi4k_=AF1ag@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:22 AM Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> 2015-03-16 16:11 GMT+01:00 Brett Cannon <bcannon at gmail.com>:
> > One of the rather subtle issues with writing Python 2/3 code is that
> > indexing on bytes in Python 2 returns a length-1 bytes object while in
> > Python 3 it returns an int. Because ==/!= always returns True/False it
> can
> > be a very subtle failure and tough to track down.
>
> I worked on such patch in the past, but I lost it :-) I can help to
> rewrite it if needed.
>

I filed http://bugs.python.org/issue23681 if you want to help (although I
don't expect the patch to be complicated; I might ask you for a code review
though =).


>
> So yes, it *is* very useful to port a large Python 2 project to Python 3.
>
> For example, you may not be able to run your application with Python 3
> because a third party library cannot be imported on Python 3, so it
> blocks the whole work on porting an application to Python 3. Until the
> module is ported, you may want to prepare the port. Checking
> bytes==str just by reading the source code is difficult.
>
> Other issues which can only be "seen" at runtime when running an
> application on Python 3 :
>
> - "x > 0" with x=None => TypeError is raised in Python 3
>

Testing can easily catch that *and* have a traceback to trace down where
the problem originated from, so I'm not worried.


> - x / 8 where x is an int => becomes a float in Python 3, it's hard to
> detect this issue in Python 2 just by reading the source code :-/
>

-Q in Python 2 and/or adding the division __future__ statement solves this
one.

Basically I'm just trying to plug the last few holes that can't be
automated by tools like Modernize or Futurize *and* won't surface where the
problem is easily during testing under Python 2 or 3 (e.g., a traceback
during testing makes it easy to find unless you swallow the exception, in
which case a warning won't help you either if you use -Werror).



>
> > What do people think of extending -b/-bb in Python 3 to warn when
> performing
> > equality between an int and a bytes object of any length? I don't want to
> > restrict to length-1 bytes objects because people may be doing
> comparisons
> > where the result can be length-1 or any other length and thus would still
> > have a subtle bug to pick up. Do people think this would raise a ton of
> > false-positives? Would people find it useful?
>
> First ensure that the stdlib doesn't raise any BytesWarning exception.
>

Yep, I always test with -Werror whenever I muck with something that can
raise a warning (side-effect from helping to write _warnings.c).

-Brett


>
> For example, os.get_exec_path() has to modify warnings filters
> temporary in Python 3 :-/
>
>     # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a
>     # BytesWarning when using python -b or python -bb: ignore the warning
>
> Victor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/a76db2cc/attachment.html>

From chris.barker at noaa.gov  Mon Mar 16 18:39:22 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 10:39:22 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
Message-ID: <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>

On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico <rosuav at gmail.com> wrote:

> The tricky part here is defining the oldest supported version. "To
> build Python X.Y, you must already have Python ??? or later installed"
> - how do you pick the minimum? The older that minimum, the more warped
> the build code has to be


sure -- but it seems what's on the table now is: must be buildable by an
old version of C -- not even C99 which is nominally 16 years old...

So yeah, the build system itself would need to support a "old" python --
though probably python2.7 would be OK at this point -- sorry, not py3 yet,
but what can you do?

> If there is an already build python (like OS-X, any general purpose linux
> > distro, Windows if you don't mind point and clicking the python.org
> > installer, etc, then it's not really a "bootstrap".
>
> No, that's not what I'm talking about. I'm talking about trying to
> build Thunderbird and having to do a bunch of steps where I run this
> program, then run that, then let it chug its way through stuff, and
> then eventually it'll start building.


It sounds like that build setup suck, yes -- but not sure why that's
relevant. Clearly we don't want something that ugly. We want something that
is both easy and robust to configure for new platfroms, etc, and easy on
the end user to "jsut build" the source. All I"m proposing that is
requireing a not-very-recent python as a first step is only a big hurdle
for folks poritn got new platforms -- your use case would be, in
INSTALL.txt:

"""
1) You need at least python 2.7 to build this software.

try:

python --version

if it starts up and reports a version greater than 2.7, then you are all
set. If not, then It can probably be installed with your system package
manager:

apt_get install python

or similar. On Windows, you can download an installer for Python2.7 from
python.org

2) Once a python is working, you can build the this latest version from
source with:

./buildme
"""

Of course, there may be options, etc, but that's the basic deal.

And heck, if people REALLY want the ./configure && make && make install dance,
then we could ship a little configure script that does nothing but call the
python-based one...

Of course, all this is predicated on there BEING some miraculous
as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use
python-based build system.....

One more note: so adding a python dependency is a pain. But the current
system requires separate stems for Windows and *nix-y systems, and who
knows what for hypothetical systems that don't support autotools. So maybe
a python-based build system that worked on all the common platforms, with
an additional makefile that you have to hand-mangle to bootstrap a new
platform isn't any more to maintain.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/6e1ab6af/attachment-0001.html>

From rymg19 at gmail.com  Mon Mar 16 18:49:21 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 12:49:21 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
Message-ID: <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>

On Mon, Mar 16, 2015 at 12:39 PM, Chris Barker <chris.barker at noaa.gov>
wrote:

> On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> The tricky part here is defining the oldest supported version. "To
>> build Python X.Y, you must already have Python ??? or later installed"
>> - how do you pick the minimum? The older that minimum, the more warped
>> the build code has to be
>
>
> sure -- but it seems what's on the table now is: must be buildable by an
> old version of C -- not even C99 which is nominally 16 years old...
>
> So yeah, the build system itself would need to support a "old" python --
> though probably python2.7 would be OK at this point -- sorry, not py3 yet,
> but what can you do?
>
> > If there is an already build python (like OS-X, any general purpose linux
>> > distro, Windows if you don't mind point and clicking the python.org
>> > installer, etc, then it's not really a "bootstrap".
>>
>> No, that's not what I'm talking about. I'm talking about trying to
>> build Thunderbird and having to do a bunch of steps where I run this
>> program, then run that, then let it chug its way through stuff, and
>> then eventually it'll start building.
>
>
> It sounds like that build setup suck, yes -- but not sure why that's
> relevant. Clearly we don't want something that ugly. We want something that
> is both easy and robust to configure for new platfroms, etc, and easy on
> the end user to "jsut build" the source. All I"m proposing that is
> requireing a not-very-recent python as a first step is only a big hurdle
> for folks poritn got new platforms -- your use case would be, in
> INSTALL.txt:
>
> """
> 1) You need at least python 2.7 to build this software.
>
> try:
>
> python --version
>
> if it starts up and reports a version greater than 2.7, then you are all
> set. If not, then It can probably be installed with your system package
> manager:
>
> apt_get install python
>
> or similar. On Windows, you can download an installer for Python2.7 from
> python.org
>
> 2) Once a python is working, you can build the this latest version from
> source with:
>
> ./buildme
> """
>
> Of course, there may be options, etc, but that's the basic deal.
>
> And heck, if people REALLY want the ./configure && make && make install dance,
> then we could ship a little configure script that does nothing but call the
> python-based one...
>
> Of course, all this is predicated on there BEING some miraculous
> as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use
> python-based build system.....
>
> One more note: so adding a python dependency is a pain. But the current
> system requires separate stems for Windows and *nix-y systems, and who
> knows what for hypothetical systems that don't support autotools. So maybe
> a python-based build system that worked on all the common platforms, with
> an additional makefile that you have to hand-mangle to bootstrap a new
> platform isn't any more to maintain.
>
>
The best build systems I know in Python are fbuild, Waf, and Meson. Meson
generates Ninja scripts, and Ninja's in C++, so that's out of the question.

I have used fbuild a LOT and can say that, after you get through the small
amount of command-line boilerplate, it works very well. It's also very easy
to extend with host/target compiler differences. Not sure about Waf, but I
do know it does about anything you'd guess.

I guess this narrows the result down to Boost.Build or a Python-based build
system.


> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/f85f84f0/attachment.html>

From abarnert at yahoo.com  Mon Mar 16 19:05:31 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Mar 2015 11:05:31 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
Message-ID: <B746894D-9BEA-4D93-A15B-0EB434C3B6DE@yahoo.com>

Something nobody's mentioned yet in this thread:

The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.).

Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems?


Sent from my iPhone

On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker at noaa.gov> wrote:

>> >> That written in Python, so Python would require Python to build which
>> >> would require Python to build which would require...
> 
> A bit more here:
> 
> It seems there are two cases:
> 
> 1) Development on and for a common, mature platform:
> 
> In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.
> 
> 2) Development for / porting to a new and/or obscure platform:
> 
> In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):
> 
>   a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...
> 
>   b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:
> 
>   i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools
> 
>  ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.
> 
>  iii) run make to do the actual build.
> 
> Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.
> 
> I have no idea if any of the known tools can do this, but if so, it would be pretty usable.
> 
> And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.
> 
> But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.
> 
>> Will this affect the commands required to build Python from source, or
>> is all this just for actual development?
> 
> well, it may or may not be both:
> 
> If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.
> 
>>  someone
>> who has little knowledge of Python's internals, and just wants to
>> build the latest version (either from hg or from a source tarball).
> 
> That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...
> 
>> But in the second case, it'd be much safer to minimize the
>> requirements. If you can untar the source and just "./configure; make;
>> sudo make install" your way to having the latest Python, that's a LOT
>> easier than I've just been doing trying to get my Dad onto a newer
>> Mozilla Thunderbird. (I don't understand what it's doing all along
>> there, but it pulls a whole bunch of separate repositories when I say
>> update, and there's a separate bootstrap step, and ... I dunno.)
> 
> requiring _some_, not very recent, python is not exactly
> 
>  """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""
> 
> If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".
> 
> In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.
> 
> -CHB
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/725f7aa4/attachment-0001.html>

From stefan at bytereef.org  Mon Mar 16 19:33:23 2015
From: stefan at bytereef.org (Stefan Krah)
Date: Mon, 16 Mar 2015 18:33:23 +0000 (UTC)
Subject: [Python-ideas] Rewriting the build system (part 2)
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
Message-ID: <loom.20150316T192520-981@post.gmane.org>

Ryan Gonzalez <rymg19 at ...> writes:
> I hate most tools that ship with configure scripts because 70% of the time
some stupid test does some stupid thing wrong and gives the worst errors,
29% of the time the configure system doesn't pick up my options correctly,
and 1% of the time it actually *works*.

That's just not true.  Also, the small build issue that triggered
this thread is already tracked here:

   http://bugs.python.org/issue22625


It seems more productive to me to fix that rather than rewrite the
build system.  Users don't have an unlimited tolerance for pointless
churn.



Stefan Krah 


From rymg19 at gmail.com  Mon Mar 16 20:19:36 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 14:19:36 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <B746894D-9BEA-4D93-A15B-0EB434C3B6DE@yahoo.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <B746894D-9BEA-4D93-A15B-0EB434C3B6DE@yahoo.com>
Message-ID: <CAO41-mO79pfRwppnTY9mRvJKiABkH+=wV=7S8v0teV3Fx+3W=A@mail.gmail.com>

Most due let you do that easily. For instance, fbuild lets you just trap
the raised error and move on.

On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> Something nobody's mentioned yet in this thread:
>
> The autotools stuff doesn't just handle different platforms, it also
> handles local differences (not every FooLinux x.y system has Tcl/Tk; if
> yours doesn't, you can still build Python, you just won't get tkinter) and
> user options (disabling features, using a specific OpenSSL instead of the
> older system one that would get autodetected, etc.).
>
> Autoconf makes this relatively easy (not that anything in autoconf is
> easy, but if you can write the OpenSSL detection, it's not significantly
> harder to write OpenSSL detection with a flag to override the path). Is the
> same thing true for other build configuration systems?
>
>
> Sent from my iPhone
>
> On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> >> That written in Python, so Python would require Python to build which
>> >> would require Python to build which would require...
>>
>
> A bit more here:
>
> It seems there are two cases:
>
> 1) Development on and for a common, mature platform:
>
> In this case finding a functional python with which to run the build tool
> sis trivial. We're all set, no need to exclude python-based tools.
>
> 2) Development for / porting to a new and/or obscure platform:
>
> In this case, then there is no existing python with which to run the build
> tools. And there are two ways you might go about this (in general):
>
>   a) cross-compiling -- run the compiler on a existing mature platform,
> and generate a compiled python for the new platform. IN this case, the
> build tools only need to run on the development platform, so we are back to
> (1), and all set. Note that the hard part of cross-compiling is still done
> by the C compiler, not really the build/configuration tools, The build
> tools need to be able to set up the cross-compile -- but presumable a new
> tool is chosen be cause it's easier to do things like that with it...
>
>   b) Native compiling -- compiling python on the new platform. This is the
> tricky one. In this case, the entire goal is to get python building/running
> on the new platform. Clearly you can't just use the nifty python-based
> build system to do that. But as that's the goal, you'll still need to do
> the tricky parts anyway. So you could cross-compile the first version --
> with maybe as few modules as possible to get started. Another option is a
> build system that needs python for the configuration stage, but not the
> compilation stage -- kind of like autotools: autotools is used in a
> three-step process:
>
>   i) run autoconf, etc, to generate a configure script -- this is often
> distributed with the source tarball, so that end users don't need the
> autoconf tools
>
>  ii) run the configure script to generate Makefiles, etc. configured for
> the platform at hand.
>
>  iii) run make to do the actual build.
>
> Usually the last two steps are run natively -- but it's not too much of a
> stretch to run i and ii on a known build machine, and then only ned to run
> make on the final target. That would require the build tools to have a
> cross-configure ability.
>
> I have no idea if any of the known tools can do this, but if so, it would
> be pretty usable.
>
> And also worth keeping in mind that truly new platforms are pretty rare --
> this kind of bootstrapping would not have to be done often --not a killer
> if it takes some hand-work.
>
> But the grain of salt : I have neer and am unlikely ever, to try to port a
> significant project to a new platform. Folks that actually need to do this
> should make these decisions.
>
> Will this affect the commands required to build Python from source, or
>> is all this just for actual development?
>>
>
> well, it may or may not be both:
>
> If the system supported a three-step process I outlined above, then
> makefiles (or the like) could be shipped for the common platforms, though
> probably easier to run the configure step on the final target. But the vast
> majority of final targets do have a python ready to run the configure/build
> machinery.
>
>  someone
>> who has little knowledge of Python's internals, and just wants to
>> build the latest version (either from hg or from a source tarball).
>>
>
> That's the assumption I'm following -- the user wants the latest version
> -- so probably already has easy access to an older version with which to
> run the build tools...
>
> But in the second case, it'd be much safer to minimize the
>>
> requirements. If you can untar the source and just "./configure; make;
>> sudo make install" your way to having the latest Python, that's a LOT
>> easier than I've just been doing trying to get my Dad onto a newer
>> Mozilla Thunderbird. (I don't understand what it's doing all along
>> there, but it pulls a whole bunch of separate repositories when I say
>> update, and there's a separate bootstrap step, and ... I dunno.)
>>
>
> requiring _some_, not very recent, python is not exactly
>
>  """a whole bunch of separate repositories when I say update, and there's
> a separate bootstrap step"""
>
> If there is an already build python (like OS-X, any general purpose linux
> distro, Windows if you don't mind point and clicking the python.org
> installer, etc, then it's not really a "bootstrap".
>
> In short -- using a python-based build system would make porting to a
> brand-new platform trickier -- but not really affect most other building.
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/19ea5690/attachment.html>

From rymg19 at gmail.com  Mon Mar 16 20:24:42 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 14:24:42 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <loom.20150316T192520-981@post.gmane.org>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
Message-ID: <CAO41-mPNxaveDELjVkgYz-J7yQJ6AYX8d8h+3pko3bAa1adYSA@mail.gmail.com>

On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah <stefan at bytereef.org> wrote:

> Ryan Gonzalez <rymg19 at ...> writes:
> > I hate most tools that ship with configure scripts because 70% of the
> time
> some stupid test does some stupid thing wrong and gives the worst errors,
> 29% of the time the configure system doesn't pick up my options correctly,
> and 1% of the time it actually *works*.
>
> That's just not true.  Also, the small build issue that triggered
> this thread is already tracked here:
>
>    http://bugs.python.org/issue22625
>
>
You've evidently had better luck than I usually have.


>
> It seems more productive to me to fix that rather than rewrite the
> build system.  Users don't have an unlimited tolerance for pointless
> churn.
>
>
Lots and lots of projects have moved from autotools to about everything
else.

It's slow, annoying, and generates huge files (the 5k-line configure.ac
generates a 17k-line configure script). And largely pointless, since
nowadays few like it anymore.


>
>
> Stefan Krah
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/c4831ba6/attachment-0001.html>

From breamoreboy at yahoo.co.uk  Mon Mar 16 21:04:59 2015
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Mon, 16 Mar 2015 20:04:59 +0000
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mPNxaveDELjVkgYz-J7yQJ6AYX8d8h+3pko3bAa1adYSA@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <CAO41-mPNxaveDELjVkgYz-J7yQJ6AYX8d8h+3pko3bAa1adYSA@mail.gmail.com>
Message-ID: <me7d1a$4bj$1@ger.gmane.org>

On 16/03/2015 19:24, Ryan Gonzalez wrote:
> On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah
> <stefan at bytereef.org
> <mailto:stefan at bytereef.org>> wrote:
>
>     Ryan Gonzalez <rymg19 at ...> writes:
>     > I hate most tools that ship with configure scripts because 70% of the time
>     some stupid test does some stupid thing wrong and gives the worst
>     errors,
>     29% of the time the configure system doesn't pick up my options
>     correctly,
>     and 1% of the time it actually *works*.
>
>     That's just not true.  Also, the small build issue that triggered
>     this thread is already tracked here:
>
>     http://bugs.python.org/issue22625
>
>
> You've evidently had better luck than I usually have.

Luck doesn't enter in to it.  More likely a bad workan always blames his 
tools?

>
>
>     It seems more productive to me to fix that rather than rewrite the
>     build system.  Users don't have an unlimited tolerance for pointless
>     churn.
>
>
> Lots and lots of projects have moved from autotools to about everything
> else.
>
> It's slow, annoying, and generates huge files (the 5k-line configure.ac
> <http://configure.ac> generates a 17k-line configure script). And
> largely pointless, since nowadays few like it anymore.
>

Perhaps everybody should move to MSVC, that way at least everything 
would be consistent :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From chris.barker at noaa.gov  Mon Mar 16 17:31:32 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 09:31:32 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
Message-ID: <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>

On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik at gmail.com>
wrote:

> "Autotools has reached the Chandrasekhar mass limit of software and
> become a black hole of suck that consumes endless hours of bright
> peoples? time without any good result. It is well past time for it to
> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
>
> I couldn't sum up this better. It looks even worse from Windows machine.
>

Is it even used for building Python on Windows (for the "normal" python.org
builds)?

But to miss-paraphrase Winston Churchill [1]:

"Autotools is the worst build system except for all the others."

Is there really something better out there?

-CHB

[1]
http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/1a5cf39a/attachment.html>

From rymg19 at gmail.com  Mon Mar 16 21:30:47 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 15:30:47 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <me7d1a$4bj$1@ger.gmane.org>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <CAO41-mPNxaveDELjVkgYz-J7yQJ6AYX8d8h+3pko3bAa1adYSA@mail.gmail.com>
 <me7d1a$4bj$1@ger.gmane.org>
Message-ID: <CAO41-mPzqacUQRtpQTZgHjG+joYfX69CjyMB8SrwL_SrtNdSng@mail.gmail.com>

On Mon, Mar 16, 2015 at 3:04 PM, Mark Lawrence <breamoreboy at yahoo.co.uk>
wrote:

> On 16/03/2015 19:24, Ryan Gonzalez wrote:
>
>> On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah
>> <stefan at bytereef.org
>> <mailto:stefan at bytereef.org>> wrote:
>>
>>     Ryan Gonzalez <rymg19 at ...> writes:
>>     > I hate most tools that ship with configure scripts because 70% of
>> the time
>>     some stupid test does some stupid thing wrong and gives the worst
>>     errors,
>>     29% of the time the configure system doesn't pick up my options
>>     correctly,
>>     and 1% of the time it actually *works*.
>>
>>     That's just not true.  Also, the small build issue that triggered
>>     this thread is already tracked here:
>>
>>     http://bugs.python.org/issue22625
>>
>>
>> You've evidently had better luck than I usually have.
>>
>
> Luck doesn't enter in to it.  More likely a bad workan always blames his
> tools?
>
>
:P

I never have that much trouble with other build tools, or even hand-written
makefiles/configure scripts. Just autotools.


>
>>
>>     It seems more productive to me to fix that rather than rewrite the
>>     build system.  Users don't have an unlimited tolerance for pointless
>>     churn.
>>
>>
>> Lots and lots of projects have moved from autotools to about everything
>> else.
>>
>> It's slow, annoying, and generates huge files (the 5k-line configure.ac
>> <http://configure.ac> generates a 17k-line configure script). And
>> largely pointless, since nowadays few like it anymore.
>>
>>
> Perhaps everybody should move to MSVC, that way at least everything would
> be consistent :)
>
>
And non-standards-compliant. :)


> --
> My fellow Pythonistas, ask not what our language can do for you, ask
> what you can do for our language.
>
> Mark Lawrence
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/88893e65/attachment.html>

From rymg19 at gmail.com  Mon Mar 16 21:33:11 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Mar 2015 15:33:11 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
Message-ID: <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:31 AM, Chris Barker <chris.barker at noaa.gov>
wrote:

> On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
>
>> "Autotools has reached the Chandrasekhar mass limit of software and
>> become a black hole of suck that consumes endless hours of bright
>> peoples? time without any good result. It is well past time for it to
>> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
>>
>> I couldn't sum up this better. It looks even worse from Windows machine.
>>
>
> Is it even used for building Python on Windows (for the "normal"
> python.org builds)?
>
>
No. Some random batch file is used. That's all I know.

However, when building without MSVC/VS2008, you need MinGW, which means
you'll end up using autotools.


> But to miss-paraphrase Winston Churchill [1]:
>
> "Autotools is the worst build system except for all the others."
>
> Is there really something better out there?
>
>
Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...


> -CHB
>
> [1]
> http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/d0f87b08/attachment-0001.html>

From chris.barker at noaa.gov  Mon Mar 16 16:51:37 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 08:51:37 -0700
Subject: [Python-ideas] math.tau
In-Reply-To: <CAMpsgwaPXmXsUjTd_ZiJ+ecAVMyhyFuDrjzAKd3oLfMdTsOAFQ@mail.gmail.com>
References: <CAPkN8xKA5weB=T1QOhpjanzO7u4FGajsUzFYkvKrOU2e8toRUQ@mail.gmail.com>
 <CAMpsgwaPXmXsUjTd_ZiJ+ecAVMyhyFuDrjzAKd3oLfMdTsOAFQ@mail.gmail.com>
Message-ID: <CALGmxEKuDSmHBOQddWaeLK94NBdf+oMQT64=oR1XvJ49uf00gw@mail.gmail.com>

On Mon, Mar 16, 2015 at 8:15 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Once a year, someone asks for math.tau and then disappears.


well, it's s small enough deal that overcoming the inertia is probably way
too much work for someone to carry it through -- which doesn't mean it's
not a good idea.


> The previous attempt to add math.tau was rejected:
> http://bugs.python.org/issue12345


well, toward teh end of that issue thread, we have fro Guido:

"""
Uh, not until I've seen a lot more evidence that people are habitually
writing "TAU = 2 * math.pi" in their programs...
"""

then a bit of evidence about how common the use is -- but no commentary on
that. I'd say "Habitually" is not well defined. The trick with "evidence"
is that many people compute a 2*pi constant, but not many people call it
"tau". I'm for one have some version of :

twopi = 2*math.pi

in a huge fraction of the code I use pi in -- though I'm can't say I'm
consistent as to what I call it, even in my own code.

So I'm going to equivocate:

+1 because while trivial, it's also useful and really commonly used

-1 because the name "tau" is unknown enough that even if it's there, a LOT
of people will compute themselves anyway :-( -- which, as pointed out is
not a big deal.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/5b7e8d92/attachment.html>

From nad at acm.org  Mon Mar 16 23:45:10 2015
From: nad at acm.org (Ned Deily)
Date: Mon, 16 Mar 2015 15:45:10 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
Message-ID: <nad-D58D3B.15451016032015@news.gmane.org>

In article <loom.20150316T192520-981 at post.gmane.org>,
 Stefan Krah <stefan at bytereef.org> wrote:
> That's just not true.  Also, the small build issue that triggered
> this thread is already tracked here:
> 
>    http://bugs.python.org/issue22625
>  
> It seems more productive to me to fix that rather than rewrite the
> build system.  Users don't have an unlimited tolerance for pointless
> churn.

+10

How many different platforms and configurations on each platform do we 
explicitly or implicitly support today for current CPython 2 and CPython 
3 releases?  I don't know (and I should since I help release them) but 
it's clearly at least in the dozens.  We do not currently have formal 
tests or test platforms (e.g. buildbots) for many of them like we should 
and it would be a monumental undertaking to try to migrate the current 
build system to something substantially different.  It's fine to let off 
steam about frustrations with build systems but talking about it here is 
not gonna cause it to change.  And it won't change unless someone (or, 
more likely, some big company) is willing to invest an enormous effort 
in people time and machine resources to do so.  Stefan's suggestion is 
much more practical.  Along with it, if someone is motivated, better 
documenting the current processes for cross-compilation and which pairs 
of build/target systems are supported would be a welcome and extremely 
useful improvement.  Much of what is there today for cross-compilation 
has slipped in over the years with little discussion or documentation.  
It's easy to break because it's not always clear how it is supposed to 
work and because it isn't regularly tested.

-- 
 Ned Deily,
 nad at acm.org


From chris.barker at noaa.gov  Tue Mar 17 00:45:17 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 16:45:17 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <nad-D58D3B.15451016032015@news.gmane.org>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <nad-D58D3B.15451016032015@news.gmane.org>
Message-ID: <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>

On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily <nad at acm.org> wrote:

>  it would be a monumental undertaking to try to migrate the current
> build system to something substantially different.  It's fine to let off
> steam about frustrations with build systems but talking about it here is
> not gonna cause it to change.  And it won't change unless someone (or,
> more likely, some big company) is willing to invest an enormous effort
> in people time and machine resources to do so.  Stefan's suggestion is
> much more practical.


This doesn't seem to me to be a winner takes all proposition -- there is no
reason someone couldn't set up a new build system for Python, get it all
working nicely on their platform(s) of choice, persuade others how
wonderful it is, get contributions, etc.

If it turns out to really be better/easier etc. than autoconf, then , and
only then, would it make any sense to talk about replacing the existing
official build system. And even then, the two could live in parallel for
quite some time...

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/3c84df49/attachment.html>

From nad at acm.org  Tue Mar 17 01:54:34 2015
From: nad at acm.org (Ned Deily)
Date: Mon, 16 Mar 2015 17:54:34 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <nad-D58D3B.15451016032015@news.gmane.org>
 <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
Message-ID: <nad-E7D2B5.17543316032015@news.gmane.org>

In article 
<CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg at mail.gmail.com>,
 Chris Barker <chris.barker at noaa.gov> wrote:
> This doesn't seem to me to be a winner takes all proposition -- there is no
> reason someone couldn't set up a new build system for Python, get it all
> working nicely on their platform(s) of choice, persuade others how
> wonderful it is, get contributions, etc.
> 
> If it turns out to really be better/easier etc. than autoconf, then , and
> only then, would it make any sense to talk about replacing the existing
> official build system. And even then, the two could live in parallel for
> quite some time...

Sorry, that's just not realistic.  We already have two different build 
systems, one for Windows and one for all other platforms, and it's 
difficult enough to keep both of them running.  Adding yet another 
system in parallel is not in the cards without a major infusion of 
resources.  There are frequent minor changes to the current build 
systems as part of new Python features or bug fixes and those run the 
risk of breaking things today and too often do.  Major changes to build 
systems impact most parts of the Python ecosphere, for example, 
development, automated testing, manual testing, release management, 
downstream distributors of CPython (many of them managing their own sets 
of local modifications to the current processes), etc etc.  To be even 
able to evaluate the merits of a major build system migration, we would 
first need to understand what we support today, what are the actual 
shortcomings of what we have today, and what we want to support in the 
future.  Then, given that, we'd need to decide what are the options and 
what are their costs.  One prerequisite to that is having a much more 
comprehensive test build environment so we could make even substantial 
incremental changes with a reasonable degree of confidence that we're 
not breaking support of a defined set of platforms and configurations.  
We don't have that today.  Working towards that would be a necessary and 
important step.

Also, note that, in all this talk about autoconf, be aware the CPython 
build processes don't even make full use of current automake/autoconf 
features.  Much of the current Python build process predates the 
addition of features to those tools.  Just bringing the current 
configure scripts and Makefiles up-to-date to use all the current 
features would be a big task that would imply a big testing effort on 
resources we don't currently have.

Also keep in mind that builds and installations are areas that few core 
CPython developers are all that interested in and with good reason.  
It's usually tedious and generally unrewarding work especially when 
operating in a vacuum.  Let's first work towards properly identifying 
and testing current important environments (for example, from typical 
installed Python configurations, not build directories and, as noted 
before, the almost totally undocumented and tested world of 
cross-building) and then we can start incrementally improving things 
with some degree of confidence that we're not going to break things for 
us and our users.

-- 
 Ned Deily,
 nad at acm.org


From stephen at xemacs.org  Tue Mar 17 03:27:21 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 17 Mar 2015 11:27:21 +0900
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
 <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
Message-ID: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>

Ryan Gonzalez writes:

 > I have used fbuild a LOT

Actually, I suspect that in the relevant sense, you haven't used it a
lot.  Have you used it for a single program on

1.  more than 3 CPUs, AND
2.  on each CPU, more than one (and preferably more than two) OSes, AND
3.  for each CPU-OS combination, more than two different configurations
    of the program being built, AND
4.  for each CPU-OS-configuration combination, at least 3 versions of
    the program source (preferably spanning a major version bump)?

The number of programs/versions you have built on a single (or small
selection of) CPU-OS combinations is irrelevant to the discussion.
The problem autotools helps to solve better than any other system I
know of is bootstrapping on a new system.  It's also quite good at
handling foibles (== regressions) across version bumps of the source
program.

I agree it's suboptimal (at best :-) to have a PyPI module that
depends on the autotools.  I've found it to be a real PITA for Windows
users who want to develop certain Emacs Lisp applications, for
example, which is a similar kind of development environment to Python.

But for bootstrapping Python itself, what else is there, really?
Every time I've seen a project move to something like SCons, I've also
seen their port rate to new platforms decrease dramatically.  (Hard to
interpret that trend, though -- if a project has the free cycles to
change build systems, maybe they have ported to almost all of the
plausible platforms already.)  Autotools alternatives just have way
fewer modules for dealing with new and/or old ;-) platforms.

Bright! Shiny! New! system         use crufty old system :-(
           +                  =>              +
   missing library :-(                     library

Caveat: Autotools is the only build system (well, Imake -- now you
know how old I am!) I have substantial experience with.  But I've
heard stories from colleagues about the limitations of Scons and
cmake in this respect, and I find it hard to believe that there are
systems out there with the accumulated knowledge of platforms that has
been assembled in Autotools over the decades.


From breamoreboy at yahoo.co.uk  Tue Mar 17 03:28:29 2015
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Tue, 17 Mar 2015 02:28:29 +0000
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <nad-D58D3B.15451016032015@news.gmane.org>
 <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
Message-ID: <me83gd$9p$1@ger.gmane.org>

On 16/03/2015 23:45, Chris Barker wrote:
> On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily
> <nad at acm.org
> <mailto:nad at acm.org>> wrote:
>
>       it would be a monumental undertaking to try to migrate the current
>     build system to something substantially different.  It's fine to let off
>     steam about frustrations with build systems but talking about it here is
>     not gonna cause it to change.  And it won't change unless someone (or,
>     more likely, some big company) is willing to invest an enormous effort
>     in people time and machine resources to do so.  Stefan's suggestion is
>     much more practical.
>
>
> This doesn't seem to me to be a winner takes all proposition -- there is
> no reason someone couldn't set up a new build system for Python, get it
> all working nicely on their platform(s) of choice, persuade others how
> wonderful it is, get contributions, etc.
>

This sounds really simple to me so when do you start work on it?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From abarnert at yahoo.com  Tue Mar 17 04:38:03 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Mar 2015 20:38:03 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mO79pfRwppnTY9mRvJKiABkH+=wV=7S8v0teV3Fx+3W=A@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <B746894D-9BEA-4D93-A15B-0EB434C3B6DE@yahoo.com>
 <CAO41-mO79pfRwppnTY9mRvJKiABkH+=wV=7S8v0teV3Fx+3W=A@mail.gmail.com>
Message-ID: <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com>

On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> 
> Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on.

How does trapping an error and moving on give you a user-configurable option to select which OpenSSL to use, whether he wants a fat i386/x86_64 build or just x86_64, whether he wants a framework bundle or a traditional layout...? (These are the first three flags that Homebrew passes to Python's configure aside from the install prefix, which I'd hope everything can handle.)

>> On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
>> Something nobody's mentioned yet in this thread:
>> 
>> The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.).
>> 
>> Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems?
>> 
>> 
>> Sent from my iPhone
>> 
>> On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>> 
>>>> >> That written in Python, so Python would require Python to build which
>>>> >> would require Python to build which would require...
>>> 
>>> A bit more here:
>>> 
>>> It seems there are two cases:
>>> 
>>> 1) Development on and for a common, mature platform:
>>> 
>>> In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools.
>>> 
>>> 2) Development for / porting to a new and/or obscure platform:
>>> 
>>> In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general):
>>> 
>>>   a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it...
>>> 
>>>   b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process:
>>> 
>>>   i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools
>>> 
>>>  ii) run the configure script to generate Makefiles, etc. configured for the platform at hand.
>>> 
>>>  iii) run make to do the actual build.
>>> 
>>> Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability.
>>> 
>>> I have no idea if any of the known tools can do this, but if so, it would be pretty usable.
>>> 
>>> And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work.
>>> 
>>> But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions.
>>> 
>>>> Will this affect the commands required to build Python from source, or
>>>> is all this just for actual development?
>>> 
>>> well, it may or may not be both:
>>> 
>>> If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery.
>>> 
>>>>  someone
>>>> who has little knowledge of Python's internals, and just wants to
>>>> build the latest version (either from hg or from a source tarball).
>>> 
>>> That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools...
>>> 
>>>> But in the second case, it'd be much safer to minimize the
>>>> requirements. If you can untar the source and just "./configure; make;
>>>> sudo make install" your way to having the latest Python, that's a LOT
>>>> easier than I've just been doing trying to get my Dad onto a newer
>>>> Mozilla Thunderbird. (I don't understand what it's doing all along
>>>> there, but it pulls a whole bunch of separate repositories when I say
>>>> update, and there's a separate bootstrap step, and ... I dunno.)
>>> 
>>> requiring _some_, not very recent, python is not exactly
>>> 
>>>  """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step"""
>>> 
>>> If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap".
>>> 
>>> In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building.
>>> 
>>> -CHB
>>> 
>>> 
>>> -- 
>>> 
>>> Christopher Barker, Ph.D.
>>> Oceanographer
>>> 
>>> Emergency Response Division
>>> NOAA/NOS/OR&R            (206) 526-6959   voice
>>> 7600 Sand Point Way NE   (206) 526-6329   fax
>>> Seattle, WA  98115       (206) 526-6317   main reception
>>> 
>>> Chris.Barker at noaa.gov
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong.
> http://kirbyfan64.github.io/ 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/109c5132/attachment-0001.html>

From chris.barker at noaa.gov  Tue Mar 17 05:13:05 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 16 Mar 2015 21:13:05 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <me83gd$9p$1@ger.gmane.org>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <nad-D58D3B.15451016032015@news.gmane.org>
 <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
 <me83gd$9p$1@ger.gmane.org>
Message-ID: <CALGmxEL_8yKtkjZEsq6t=GcXRA3sBhr4Xfx6C0epfRaUW0BopA@mail.gmail.com>

On Mon, Mar 16, 2015 at 7:28 PM, Mark Lawrence <breamoreboy at yahoo.co.uk>
wrote:

> On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily
>
>> This doesn't seem to me to be a winner takes all proposition -- there is
>> no reason someone couldn't set up a new build system for Python, get it
>> all working nicely on their platform(s) of choice, persuade others how
>> wonderful it is, get contributions, etc.
>>
>>
> This sounds really simple to me so when do you start work on it?


I think I was misunderstood -- my point was that there is no reason to
start all this with trying to persuade the python dev team to accept a new
build system up front.

The only possible way it could happen were if someone(s) did the work and
got a new system in place that worked well and proved to be better than
autoconf in various ways, while being just as capable. And that would be
LOT of work to get that far.

But it could be done without having to persuade anyone up front that it 's
a good idea.

Frankly, I doubt anyone's going to put that work in -- I know I won't --
remember that I'm one of the folks that's skeptical that this mythical
build system exists at all -- never mind the work to transition to it.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150316/f7ec1757/attachment.html>

From lkb.teichmann at gmail.com  Tue Mar 17 16:55:52 2015
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Tue, 17 Mar 2015 16:55:52 +0100
Subject: [Python-ideas] Simpler Customization of Class Creation, next round
Message-ID: <CAK9R32R2NX9n8DUGjENVhY4bDowqhd+VoddwZA6PBoyDZA99bg@mail.gmail.com>

Hello everybody,

recently I posted PEP 487, a simpler customization of class creation.
For newcomers: I propose the introduction of a __init_subclass__ classmethod
which initializes subclasses of a class, simplifying what metaclasses can
already do.

It took me a while to digest all the ideas from the list here, but well, we're
not in a hurry. So, I updated PEP 487, and pushed the changes to github
at https://github.com/tecki/peps/commits/master

I applied the following changes:

PEP 487 contained the possibility to set a namespace for subclasses.
The most important usecase for this feature would be to have an
OrderedDict as the class definition namespace. As Eric Snow pointed
out, that will be soon standard anyways, so I took out this feature from
the PEP. The implementation on PyPI now just uses an OrderedDict
as a namespace, anticipating the expected changes to CPython.

I also did some reading on possible usecases for PEP 487, so that it
actually may be used by someone. Some Traits-like thing is a standard
usecase, so I looked especially at IPython's traitlets, which are a simple
example of that usecase.

Currently traitlets use both __new__ and __init__ of a metaclass. So I
tried to also introduce a __new_subclass__ the same way I introduced
__init_subclass__. This turned out much harder than I thought, actually
impossible, because it is type.__new__ that sets the method resolution
order, so making super() work in a __new_subclass__ hook is a
chicken-egg problem: we need the MRO to find the next base class to
call, but the most basic base class is the one creating the MRO. Nick,
how did you solve that problem in PEP 422?

Anyhow, I think that traitlets can also be written just using
__init_subclass__. There is just this weird hint in the docs that you should
use __new__ for metaclasses, not __init__, a hint I never understood
as the reasons when to use __new__ or __init__ are precisely the same
for normal classes and metaclasses. So I think we don't miss out much
when not having __new_subclass__.

I also updated the implementation of PEP 487, it's still at
https://pypi.python.org/pypi/metaclass

Greetings

Martin

From luciano at ramalho.org  Tue Mar 17 17:10:32 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Tue, 17 Mar 2015 13:10:32 -0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
Message-ID: <CALxg4FWBQ2jDsHcwevAxb0EpiF8h5J1uma2MM93Ofsns1uUMng@mail.gmail.com>

On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> I use the analogy of "encoding abstract numbers into specific bytes" myself.

Your version of the analogy doesn't sound as helpful to me; "abastract
numbers" sounds even more cryptic than bytes.

My mnemonics are "decode from cryptic bytes to human text" and "encode
from human text to cryptic bytes".

A more objective problem is: "from" is a keyword, so it can't be a method name.

Best,

Luciano



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From luciano at ramalho.org  Tue Mar 17 17:52:04 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Tue, 17 Mar 2015 13:52:04 -0300
Subject: [Python-ideas] A mutable alternative to namedtuple
Message-ID: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>

Sometimes we need a simple class to hold some mutable attributes,
provide a nice repr, support == for testing, and support iterable
unpacking, so you can write:

>>> p = Point(3, 4)
>>> x, y = p

That's very much like the classes built by namedtuple, but mutable.

I propose we add to the collections module another class factory. I am
calling it plainclass, but perhaps we can think of a better name. Here
is how it would be used:

>>> import collections
>>> Point = collections.plainclass('Point', 'x y')

The signature of the plainclass function would be exactly the same as
namedtuple, supporting the same alternative ways of naming the
attributes.

The semantics of the generated Point class would be like this code:

https://gist.github.com/ramalho/fd3d367e9d3b2a659faf

What do you think?

Cheers,

Luciano

PS. I am aware that there are "Namespace" classes in the standard
library (e.g. [2]). They solve a different problem.

[2] https://docs.python.org/3/library/argparse.html#argparse.Namespace


-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From rymg19 at gmail.com  Tue Mar 17 18:01:24 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 17 Mar 2015 12:01:24 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
 <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
 <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAO41-mOSSXKZaScQvA5qESPWNaanSbTtJnZECCcofAEt+BO-+w@mail.gmail.com>

On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> Ryan Gonzalez writes:
>
>  > I have used fbuild a LOT
>
> Actually, I suspect that in the relevant sense, you haven't used it a
> lot.  Have you used it for a single program on
>
> 1.  more than 3 CPUs, AND
> 2.  on each CPU, more than one (and preferably more than two) OSes, AND
> 3.  for each CPU-OS combination, more than two different configurations
>     of the program being built, AND
> 4.  for each CPU-OS-configuration combination, at least 3 versions of
>     the program source (preferably spanning a major version bump)?
>
>
No...but I know that Felix uses it for iPhone, OSX, Linux, ... with
separation of host and target and multiple configurations.


> The number of programs/versions you have built on a single (or small
> selection of) CPU-OS combinations is irrelevant to the discussion.
> The problem autotools helps to solve better than any other system I
> know of is bootstrapping on a new system.  It's also quite good at
> handling foibles (== regressions) across version bumps of the source
> program.
>
> I agree it's suboptimal (at best :-) to have a PyPI module that
> depends on the autotools.  I've found it to be a real PITA for Windows
> users who want to develop certain Emacs Lisp applications, for
> example, which is a similar kind of development environment to Python.
>
> But for bootstrapping Python itself, what else is there, really?
> Every time I've seen a project move to something like SCons, I've also
> seen their port rate to new platforms decrease dramatically.  (Hard to
> interpret that trend, though -- if a project has the free cycles to
> change build systems, maybe they have ported to almost all of the
> plausible platforms already.)  Autotools alternatives just have way
> fewer modules for dealing with new and/or old ;-) platforms.
>
>
This is also SCons. Everybody loved it until they had to maintain it. :)


> Bright! Shiny! New! system         use crufty old system :-(
>            +                  =>              +
>    missing library :-(                     library
>
> Caveat: Autotools is the only build system (well, Imake -- now you
> know how old I am!) I have substantial experience with.  But I've
> heard stories from colleagues about the limitations of Scons and
> cmake in this respect, and I find it hard to believe that there are
> systems out there with the accumulated knowledge of platforms that has
> been assembled in Autotools over the decades.
>
>
SCons is very annoying. CMake is weird and only really useful for those
huge, 20,000 file C++ projects.

-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/a96b030f/attachment.html>

From rymg19 at gmail.com  Tue Mar 17 18:06:52 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 17 Mar 2015 12:06:52 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <B746894D-9BEA-4D93-A15B-0EB434C3B6DE@yahoo.com>
 <CAO41-mO79pfRwppnTY9mRvJKiABkH+=wV=7S8v0teV3Fx+3W=A@mail.gmail.com>
 <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com>
Message-ID: <CAO41-mNTpgrqopuimy9T0HO-Ght2NR_Ex6QKEuhf7qgFa4k8HQ@mail.gmail.com>

On Mon, Mar 16, 2015 at 10:38 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
> Most due let you do that easily. For instance, fbuild lets you just trap
> the raised error and move on.
>
>
> How does trapping an error and moving on give you a user-configurable
> option to select which OpenSSL to use, whether he wants a fat i386/x86_64
> build or just x86_64, whether he wants a framework bundle or a traditional
> layout...? (These are the first three flags that Homebrew passes to
> Python's configure aside from the install prefix, which I'd hope everything
> can handle.)
>
>
Just mix a command-line argument with a try-catch block. It's actually much
easier/less-hackish than it sounds.

The nice thing is that fbuild allows multiple builders, so separating
target/host stuff and command-line args is super-easy.


> On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert <
> abarnert at yahoo.com.dmarc.invalid> wrote:
>
>> Something nobody's mentioned yet in this thread:
>>
>> The autotools stuff doesn't just handle different platforms, it also
>> handles local differences (not every FooLinux x.y system has Tcl/Tk; if
>> yours doesn't, you can still build Python, you just won't get tkinter) and
>> user options (disabling features, using a specific OpenSSL instead of the
>> older system one that would get autodetected, etc.).
>>
>> Autoconf makes this relatively easy (not that anything in autoconf is
>> easy, but if you can write the OpenSSL detection, it's not significantly
>> harder to write OpenSSL detection with a flag to override the path). Is the
>> same thing true for other build configuration systems?
>>
>>
>> Sent from my iPhone
>>
>> On Mar 16, 2015, at 9:27 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>>
>> >> That written in Python, so Python would require Python to build which
>>> >> would require Python to build which would require...
>>>
>>
>> A bit more here:
>>
>> It seems there are two cases:
>>
>> 1) Development on and for a common, mature platform:
>>
>> In this case finding a functional python with which to run the build tool
>> sis trivial. We're all set, no need to exclude python-based tools.
>>
>> 2) Development for / porting to a new and/or obscure platform:
>>
>> In this case, then there is no existing python with which to run the
>> build tools. And there are two ways you might go about this (in general):
>>
>>   a) cross-compiling -- run the compiler on a existing mature platform,
>> and generate a compiled python for the new platform. IN this case, the
>> build tools only need to run on the development platform, so we are back to
>> (1), and all set. Note that the hard part of cross-compiling is still done
>> by the C compiler, not really the build/configuration tools, The build
>> tools need to be able to set up the cross-compile -- but presumable a new
>> tool is chosen be cause it's easier to do things like that with it...
>>
>>   b) Native compiling -- compiling python on the new platform. This is
>> the tricky one. In this case, the entire goal is to get python
>> building/running on the new platform. Clearly you can't just use the nifty
>> python-based build system to do that. But as that's the goal, you'll still
>> need to do the tricky parts anyway. So you could cross-compile the first
>> version -- with maybe as few modules as possible to get started. Another
>> option is a build system that needs python for the configuration stage, but
>> not the compilation stage -- kind of like autotools: autotools is used in a
>> three-step process:
>>
>>   i) run autoconf, etc, to generate a configure script -- this is often
>> distributed with the source tarball, so that end users don't need the
>> autoconf tools
>>
>>  ii) run the configure script to generate Makefiles, etc. configured for
>> the platform at hand.
>>
>>  iii) run make to do the actual build.
>>
>> Usually the last two steps are run natively -- but it's not too much of a
>> stretch to run i and ii on a known build machine, and then only ned to run
>> make on the final target. That would require the build tools to have a
>> cross-configure ability.
>>
>> I have no idea if any of the known tools can do this, but if so, it would
>> be pretty usable.
>>
>> And also worth keeping in mind that truly new platforms are pretty rare
>> -- this kind of bootstrapping would not have to be done often --not a
>> killer if it takes some hand-work.
>>
>> But the grain of salt : I have neer and am unlikely ever, to try to port
>> a significant project to a new platform. Folks that actually need to do
>> this should make these decisions.
>>
>> Will this affect the commands required to build Python from source, or
>>> is all this just for actual development?
>>>
>>
>> well, it may or may not be both:
>>
>> If the system supported a three-step process I outlined above, then
>> makefiles (or the like) could be shipped for the common platforms, though
>> probably easier to run the configure step on the final target. But the vast
>> majority of final targets do have a python ready to run the configure/build
>> machinery.
>>
>>  someone
>>> who has little knowledge of Python's internals, and just wants to
>>> build the latest version (either from hg or from a source tarball).
>>>
>>
>> That's the assumption I'm following -- the user wants the latest version
>> -- so probably already has easy access to an older version with which to
>> run the build tools...
>>
>> But in the second case, it'd be much safer to minimize the
>>>
>> requirements. If you can untar the source and just "./configure; make;
>>> sudo make install" your way to having the latest Python, that's a LOT
>>> easier than I've just been doing trying to get my Dad onto a newer
>>> Mozilla Thunderbird. (I don't understand what it's doing all along
>>> there, but it pulls a whole bunch of separate repositories when I say
>>> update, and there's a separate bootstrap step, and ... I dunno.)
>>>
>>
>> requiring _some_, not very recent, python is not exactly
>>
>>  """a whole bunch of separate repositories when I say update, and there's
>> a separate bootstrap step"""
>>
>> If there is an already build python (like OS-X, any general purpose linux
>> distro, Windows if you don't mind point and clicking the python.org
>> installer, etc, then it's not really a "bootstrap".
>>
>> In short -- using a python-based build system would make porting to a
>> brand-new platform trickier -- but not really affect most other building.
>>
>> -CHB
>>
>>
>> --
>>
>> Christopher Barker, Ph.D.
>> Oceanographer
>>
>> Emergency Response Division
>> NOAA/NOS/OR&R            (206) 526-6959   voice
>> 7600 Sand Point Way NE   (206) 526-6329   fax
>> Seattle, WA  98115       (206) 526-6317   main reception
>>
>> Chris.Barker at noaa.gov
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/92378334/attachment-0001.html>

From ericsnowcurrently at gmail.com  Tue Mar 17 18:14:33 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 17 Mar 2015 11:14:33 -0600
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
Message-ID: <CALFfu7BaWX87nPWbo30gVb7A8Eoi0v_yG34brTamuvFo6s=K1w@mail.gmail.com>

On Tue, Mar 17, 2015 at 10:52 AM, Luciano Ramalho <luciano at ramalho.org> wrote:
> Sometimes we need a simple class to hold some mutable attributes,
> provide a nice repr, support == for testing, and support iterable
> unpacking, so you can write:
>
>>>> p = Point(3, 4)
>>>> x, y = p
>
> That's very much like the classes built by namedtuple, but mutable.
>
> I propose we add to the collections module another class factory. I am
> calling it plainclass, but perhaps we can think of a better name. Here
> is how it would be used:
>
>>>> import collections
>>>> Point = collections.plainclass('Point', 'x y')
>
> The signature of the plainclass function would be exactly the same as
> namedtuple, supporting the same alternative ways of naming the
> attributes.
>
> The semantics of the generated Point class would be like this code:
>
> https://gist.github.com/ramalho/fd3d367e9d3b2a659faf

Should it also have all the same methods as a namedtuple class (e.g.
the tuple methods, _make, _replace)?  What about the other namedtuple
attrs (_fields, etc.)?

What's the motivation for parity with namedtuple (particularly
iteration)?  I suppose I see the desire for a fixed set of mutable
attributes specific to the generated type.  However, there have been
numerous discussions on this list about alternative approaches to
namedtuple which apply here.  Simply adapting namedtuple may not be
the right thing.

Regardless, this is the sort of thing that should bake outside the
stdlib for a while to prove it's approach and its worth, much as
namedtuple did.  It would also help if there were a concrete use case
in the stdlib that this new class (factory) would satisfy.

> PS. I am aware that there are "Namespace" classes in the standard
> library (e.g. [2]). They solve a different problem.
>
> [2] https://docs.python.org/3/library/argparse.html#argparse.Namespace

Don't forget types.SimpleNamespace. :)

-eric

From eric at trueblade.com  Tue Mar 17 18:20:08 2015
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 17 Mar 2015 13:20:08 -0400
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
Message-ID: <55086248.2010009@trueblade.com>

On 03/17/2015 12:52 PM, Luciano Ramalho wrote:
> Sometimes we need a simple class to hold some mutable attributes,
> provide a nice repr, support == for testing, and support iterable
> unpacking, so you can write:
> 
>>>> p = Point(3, 4)
>>>> x, y = p
> 
> That's very much like the classes built by namedtuple, but mutable.

https://pypi.python.org/pypi/namedlist

It also adds default values to the generated constructor, which may or
may not be desirable. But if used exactly like collections.namedtuple,
it ignores the default values.

Eric.

> 
> I propose we add to the collections module another class factory. I am
> calling it plainclass, but perhaps we can think of a better name. Here
> is how it would be used:
> 
>>>> import collections
>>>> Point = collections.plainclass('Point', 'x y')
> 
> The signature of the plainclass function would be exactly the same as
> namedtuple, supporting the same alternative ways of naming the
> attributes.
> 
> The semantics of the generated Point class would be like this code:
> 
> https://gist.github.com/ramalho/fd3d367e9d3b2a659faf
> 
> What do you think?
> 
> Cheers,
> 
> Luciano
> 
> PS. I am aware that there are "Namespace" classes in the standard
> library (e.g. [2]). They solve a different problem.
> 
> [2] https://docs.python.org/3/library/argparse.html#argparse.Namespace
> 
> 


From nicholas.chammas at gmail.com  Tue Mar 17 19:39:59 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Tue, 17 Mar 2015 11:39:59 -0700 (PDT)
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <a107b716-4b25-40eb-8d16-fce0106be877@googlegroups.com>
References: <a107b716-4b25-40eb-8d16-fce0106be877@googlegroups.com>
Message-ID: <92ac0f1b-617e-4da5-a4d0-47491ea446ad@googlegroups.com>

I posted this via Google Groups and got a response back by email saying my 
post was rejected because I wasn't a list member. So I signed up via 
Mailman and resubmitted this post by mail. Not sure if I should expect a 
duplicate post to show up here on not... Apologies.

On Tuesday, March 17, 2015 at 2:13:31 PM UTC-4, Nicholas Chammas wrote:
>
> The Zen of Python <https://www.python.org/dev/peps/pep-0020/> says:
>
> There should be one? and preferably only one ?obvious way to do it.
>
> Python already has a style guide in PEP 8 
> <https://www.python.org/dev/peps/pep-0008/>. More importantly, from a 
> practical standpoint, Python has automated third-party tools for checking 
> and enforcing PEP 8 guidelines:
>
>    - pep8 <https://pypi.python.org/pypi/pep8/>: Python style guide checker 
>    - autopep8 <https://pypi.python.org/pypi/autopep8/>: A tool that 
>    automatically formats Python code to conform to the PEP 8 style guide 
>
> I say ?more importantly? because these tools are what make the style guide 
> alive. If a Python project of any non-trivial size is PEP 8-compliant, you 
> can bet it?s because the project contributors automated the process of 
> enforcing compliance using these or similar tools. (Certainly, some parts 
> of PEP 8 are hard to automate. But those parts that *can* be automated 
> have been, to the benefit of Python programmers everywhere.)
>
> Having a style guide provides benefits that I think are well-understood. 
> Additionally having a standardized style that can be *applied 
> automatically* makes whatever benefits of the style guide that much more 
> prevalent in the greater community and actually effective.
>
> As many of you may already know, the Go-lang folks have such a feature for 
> their language (though they call it formatting as opposed to styling). They 
> note:
>
> go fmt your code <http://blog.golang.org/go-fmt-your-code>:
>
> Gofmt <http://golang.org/cmd/gofmt/> is a tool that automatically formats 
> Go source code.
>
> Gofmt?d code is:
>
>    - easier to *write*: never worry about minor formatting concerns while 
>    hacking away, 
>    - easier to *read*: when all code looks the same you need not mentally 
>    convert others? formatting style into something you can understand. 
>    - easier to *maintain*: mechanical changes to the source don?t cause 
>    unrelated changes to the file?s formatting; diffs show only the real 
>    changes. 
>    - *uncontroversial*: never have a debate about spacing or brace 
>    position ever again! 
>
>  Of course, we don?t worry about brace positions, but these benefits of a 
> standardized style carry over into any language. :)
>
> So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s 
> gofmt <http://blog.golang.org/go-fmt-your-code>, I propose we promote 
> standardized auto-styling to a first-class language feature.
>
> UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps 
> something like:
>
> python -m style script.py
>
> Don?t read too much into this UI suggestion, as I?m really not sure how it 
> would be implemented. The point is to expose some auto-styler as a standard 
> language tool?i.e. something like autopep8, but as a built-in utility.
>
> On running a command like that, script.py would automatically be updated 
> to conform to the standard style we want to promote. This is what gofmt 
> does for Go code.
>
> Promoting an auto-styler to be a first-class language feature builds on 
> the foundation laid in PEP 8 and in tools like pep8 and autopep8. It 
> helps reduce code style to more of a non-issue for Python projects than it 
> already is.
> ------------------------------
>
> *Side Note 1:* There was a previous python-ideas discussion about 
> introducing standardized formatting here 
> <https://groups.google.com/d/msg/python-ideas/ic-aPSvamS4/2lj53Em0VrsJ>, 
> but it was totally derailed by the OP?s suggestion that the formatter 
> introduce block delimiters. I am proposing no such thing here.
>
> *Side Note 2:* I found these additional comments 
> <http://blog.golang.org/go-fmt-your-code> about gofmt interesting:
>
> Gofmt also enables gofix <http://golang.org/cmd/fix/>, which can make 
> arbitrarily complex source transformations. Gofix was an invaluable tool 
> during the early days when we regularly made breaking changes to the 
> language and libraries.
>
> Introducing Gofix <http://blog.golang.org/introducing-gofix>:
>
> Gofix gives us the ability to fix mistakes or completely rethink package 
> APIs without worrying about the cost of converting existing code.
>
> Python has its own analog to gofix in 2to3 
> <https://docs.python.org/3/library/2to3.html>, though that was built for 
> a specific migration rather than as a regular language feature. I wonder if 
> having a gofmt analog in Python would similarly facilitate future 
> migrations, though I doubt we?ll ever have a big as migration as the 2-to-3 
> one.
> ?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/bf006c33/attachment-0001.html>

From nicholas.chammas at gmail.com  Tue Mar 17 19:31:10 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Tue, 17 Mar 2015 18:31:10 +0000
Subject: [Python-ideas] gofmt for Python: standardized styling as a language
	feature
Message-ID: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>

The Zen of Python <https://www.python.org/dev/peps/pep-0020/> says:

There should be one? and preferably only one ?obvious way to do it.

Python already has a style guide in PEP 8
<https://www.python.org/dev/peps/pep-0008/>. More importantly, from a
practical standpoint, Python has automated third-party tools for checking
and enforcing PEP 8 guidelines:

   - pep8 <https://pypi.python.org/pypi/pep8/>: Python style guide checker
   - autopep8 <https://pypi.python.org/pypi/autopep8/>: A tool that
   automatically formats Python code to conform to the PEP 8 style guide

I say ?more importantly? because these tools are what make the style guide
alive. If a Python project of any non-trivial size is PEP 8-compliant, you
can bet it?s because the project contributors automated the process of
enforcing compliance using these or similar tools. (Certainly, some parts
of PEP 8 are hard to automate. But those parts that *can* be automated have
been, to the benefit of Python programmers everywhere.)

Having a style guide provides benefits that I think are well-understood.
Additionally having a standardized style that can be *applied
automatically* makes
whatever benefits of the style guide that much more prevalent in the
greater community and actually effective.

As many of you may already know, the Go-lang folks have such a feature for
their language (though they call it formatting as opposed to styling). They
note:

go fmt your code <http://blog.golang.org/go-fmt-your-code>:

Gofmt <http://golang.org/cmd/gofmt/> is a tool that automatically formats
Go source code.

Gofmt?d code is:

   - easier to *write*: never worry about minor formatting concerns while
   hacking away,
   - easier to *read*: when all code looks the same you need not mentally
   convert others? formatting style into something you can understand.
   - easier to *maintain*: mechanical changes to the source don?t cause
   unrelated changes to the file?s formatting; diffs show only the real
   changes.
   - *uncontroversial*: never have a debate about spacing or brace position
   ever again!

Of course, we don?t worry about brace positions, but these benefits of a
standardized style carry over into any language. :)

So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s
gofmt <http://blog.golang.org/go-fmt-your-code>, I propose we promote
standardized auto-styling to a first-class language feature.

UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps
something like:

python -m style script.py

Don?t read too much into this UI suggestion, as I?m really not sure how it
would be implemented. The point is to expose some auto-styler as a standard
language tool?i.e. something like autopep8, but as a built-in utility.

On running a command like that, script.py would automatically be updated to
conform to the standard style we want to promote. This is what gofmt does
for Go code.

Promoting an auto-styler to be a first-class language feature builds on the
foundation laid in PEP 8 and in tools like pep8 and autopep8. It helps
reduce code style to more of a non-issue for Python projects than it
already is.
------------------------------

*Side Note 1:* There was a previous python-ideas discussion about
introducing standardized formatting here
<https://groups.google.com/d/msg/python-ideas/ic-aPSvamS4/2lj53Em0VrsJ>,
but it was totally derailed by the OP?s suggestion that the formatter
introduce block delimiters. I am proposing no such thing here.

*Side Note 2:* I found these additional comments
<http://blog.golang.org/go-fmt-your-code> about gofmt interesting:

Gofmt also enables gofix <http://golang.org/cmd/fix/>, which can make
arbitrarily complex source transformations. Gofix was an invaluable tool
during the early days when we regularly made breaking changes to the
language and libraries.

Introducing Gofix <http://blog.golang.org/introducing-gofix>:

Gofix gives us the ability to fix mistakes or completely rethink package
APIs without worrying about the cost of converting existing code.

Python has its own analog to gofix in 2to3
<https://docs.python.org/3/library/2to3.html>, though that was built for a
specific migration rather than as a regular language feature. I wonder if
having a gofmt analog in Python would similarly facilitate future
migrations, though I doubt we?ll ever have a big as migration as the 2-to-3
one.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/9b2f759b/attachment.html>

From nicholas.chammas at gmail.com  Tue Mar 17 21:51:30 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Tue, 17 Mar 2015 13:51:30 -0700 (PDT)
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <a107b716-4b25-40eb-8d16-fce0106be877@googlegroups.com>
References: <a107b716-4b25-40eb-8d16-fce0106be877@googlegroups.com>
Message-ID: <0592d0ad-ddea-4bcc-8139-930370cad143@googlegroups.com>

Please disregard this thread in favor of the one that went successfully 
through via Mailman 
here: https://groups.google.com/forum/#!topic/python-ideas/xute5NmXRps

Apologies again about the duplicate posting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/c98955a5/attachment.html>

From abarnert at yahoo.com  Tue Mar 17 23:16:33 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 17 Mar 2015 15:16:33 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
Message-ID: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com>

On Mar 17, 2015, at 11:31 AM, Nicholas Chammas <nicholas.chammas at gmail.com> wrote:
> The Zen of Python says:
> 
> There should be one? and preferably only one ?obvious way to do it.
> 
As someone who uses autopep8 regularly but has at least once had resistance to getting into a team's workflow, +1 on the idea. The analogy with gofmt isn't perfect because gofmt only deals with some simple issues where there's an unambiguous 100%-clear standard, while PEP 8 includes a wide range of guidelines that are usually but not always consistent and appropriate, but that (obviously) doesn't mean autopep8 is an impossible or pointless non-starter.

So, is your proposal that autopep8 (and therefore pep8, on which it depends) be added to the stdlib with only minor changes? If so:

 * You'd probably want to ask whether Hideo Hattori and Johann C. Rocholl are willing to relicense and contribute their code, and maintain it for years to come, before asking whether Python will accept the result.
 * Will it be easy to transform a new version of the PyPI libraries into a new version of the stdlib module in some mostly-automated way for each new Python release?
 * Given that Python updates less often than pep8 and autopep8, will it generally be "good enough" to use the autopep8 1.1.3 that came with your Python 3.5 instead of installing the 1.1.7 that's on PyPI? What if you're still using Python 3.5 in another two years, and Python 3.6 came with 1.2.1, and 1.2.5 is on PyPI; is using 1.1.3 still a good idea? (Compare with pysqlite; I've used sqlite3 many times, and only wanted to get the newer version twice--and once just because it was the easiest way to build with a local variant of the C library on Windows.)
 * Are all of the current features and bugs (see issues on GitHub) acceptable as-is for the stdlib today?

If the answers to those questions is "yes", this seems like the way to go. Otherwise, you're probably suggesting someone either fork the existing projects or write a brand new one (and as the one with the itch and the ideas, who better than you?).
> Python already has a style guide in PEP 8. More importantly, from a practical standpoint, Python has automated third-party tools for checking and enforcing PEP 8 guidelines:
> 
> pep8: Python style guide checker
> autopep8: A tool that automatically formats Python code to conform to the PEP 8 style guide
> I say ?more importantly? because these tools are what make the style guide alive. If a Python project of any non-trivial size is PEP 8-compliant, you can bet it?s because the project contributors automated the process of enforcing compliance using these or similar tools. (Certainly, some parts of PEP 8 are hard to automate. But those parts that can be automated have been, to the benefit of Python programmers everywhere.)
> 
> Having a style guide provides benefits that I think are well-understood. Additionally having a standardized style that can be applied automatically makes whatever benefits of the style guide that much more prevalent in the greater community and actually effective.
> 
> As many of you may already know, the Go-lang folks have such a feature for their language (though they call it formatting as opposed to styling). They note:
> 
> go fmt your code:
> 
> Gofmt is a tool that automatically formats Go source code.
> 
> Gofmt?d code is:
> 
> easier to write: never worry about minor formatting concerns while hacking away,
> easier to read: when all code looks the same you need not mentally convert others? formatting style into something you can understand.
> easier to maintain: mechanical changes to the source don?t cause unrelated changes to the file?s formatting; diffs show only the real changes.
> uncontroversial: never have a debate about spacing or brace position ever again!
> Of course, we don?t worry about brace positions, but these benefits of a standardized style carry over into any language. :)
> 
> So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s gofmt, I propose we promote standardized auto-styling to a first-class language feature.
> 
> UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps something like:
> 
> python -m style script.py
> Don?t read too much into this UI suggestion, as I?m really not sure how it would be implemented. The point is to expose some auto-styler as a standard language tool?i.e. something like autopep8, but as a built-in utility.
> 
> On running a command like that, script.py would automatically be updated to conform to the standard style we want to promote. This is what gofmt does for Go code.
> 
> Promoting an auto-styler to be a first-class language feature builds on the foundation laid in PEP 8 and in tools like pep8 and autopep8. It helps reduce code style to more of a non-issue for Python projects than it already is.
> 
> Side Note 1: There was a previous python-ideas discussion about introducing standardized formatting here, but it was totally derailed by the OP?s suggestion that the formatter introduce block delimiters. I am proposing no such thing here.
> 
> Side Note 2: I found these additional comments about gofmt interesting:
> 
> Gofmt also enables gofix, which can make arbitrarily complex source transformations. Gofix was an invaluable tool during the early days when we regularly made breaking changes to the language and libraries.
> 
> Introducing Gofix:
> 
> Gofix gives us the ability to fix mistakes or completely rethink package APIs without worrying about the cost of converting existing code.
> 
> Python has its own analog to gofix in 2to3, though that was built for a specific migration rather than as a regular language feature. I wonder if having a gofmt analog in Python would similarly facilitate future migrations, though I doubt we?ll ever have a big as migration as the 2-to-3 one.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/78b25925/attachment-0001.html>

From steve at pearwood.info  Tue Mar 17 23:28:52 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 18 Mar 2015 09:28:52 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
Message-ID: <20150317222851.GS7655@ando.pearwood.info>

On Tue, Mar 17, 2015 at 06:31:10PM +0000, Nicholas Chammas wrote:

> Additionally having a standardized style that can be *applied 
> automatically* makes whatever benefits of the style guide that much 
> more prevalent in the greater community and actually effective.

If you're talking about Gofmt and Gofix and enforcing PEP-8 rules for 
variable names, you're basically talking about refactoring tools, not 
merely source code reformatters. Do you have such tools you would like 
to propose for the standard library?

If you do have such tools, or would like to volunteer to write them, 
then they should spend some time as a third party project to knock the 
sharp edges off them. Once their usefulness is well-established and 
proven, and their API and featureset stable, then they can be proposed 
for the standard library.

Unlike Go-lang, which has the resources of Google behind it and can just 
pay some person or persons to work on this with as high a priority as 
Google wishes to give it, Python is reliant on volunteers. For major 
proposals like this, you're likely to get much more traction if you are 
volunteering yourself, rather than just hoping to convince somebody else 
to volunteer.

I think good refactoring tools for Python would be useful. I'm not 
volunteering to write them.



-- 
Steve

From tjreedy at udel.edu  Wed Mar 18 01:04:53 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 17 Mar 2015 20:04:53 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
Message-ID: <meafff$tjv$1@ger.gmane.org>

On 3/17/2015 2:31 PM, Nicholas Chammas wrote:
> The Zen of Python <https://www.python.org/dev/peps/pep-0020/> says:
>
>     There should be one? and preferably only one ?obvious way to do it.
>
> Python already has a style guide in PEP 8
> <https://www.python.org/dev/peps/pep-0008/>. More importantly, from a
> practical standpoint, Python has automated third-party tools for
> checking and enforcing PEP 8 guidelines:
>
>   * |pep8| <https://pypi.python.org/pypi/pep8/>: Python style guide checker
>   * |autopep8| <https://pypi.python.org/pypi/autopep8/>: A tool that
>     automatically formats Python code to conform to the PEP 8 style guide

Except for idlelib and more recently turtledemo, the stdlib consists of 
*library* modules that define constants, mostly functions and classes, 
for import into and use by other libraries and applications.  Notice 
that the stdlib contains lib2to3, while the 2to3 application that uses 
lib2to3 is in /Tools/Scripts, along with other application scripts, 
mostly short, written by core developers.

pep8, autopep8, and similar applications do not belong in the stdlib. 
They are easily installed with, for instance, 'pip install autopep8'. 
The install process puts a startup file in /Scripts, which (for Windows 
at least) is on PATH (unlike /Tools/Scripts).  After installing, I 
entered 'autopep8 file.py' and it runs.

There is an issue on the tracker to add to the Idle IDE a gui frontend 
to pip, so beginners who use Idle will be able to manage (install, 
update, delete) 3rd-party apps like these without knowing about 'command 
lines' or having to remember pip options.  (There is also an issue to 
apply such apps to the contents of an Editor window.)

-- 
Terry Jan Reedy



From nicholas.chammas at gmail.com  Wed Mar 18 01:55:17 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 18 Mar 2015 00:55:17 +0000
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com>
Message-ID: <CAOhmDzdVODThKHAbjSZFqqcUZjRPwiVW12jDJR4rrww_nX9skg@mail.gmail.com>

On Tue, Mar 17, 2015 at 6:16 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> So, is your proposal that autopep8 (and therefore pep8, on which it
> depends) be added to the stdlib with only minor changes?


Not necessarily, though that is certainly a possible approach. (In fact,
it's probably the approach most likely to succeed, since it builds on
existing and tested tools.)

I was more interested in first testing the _idea_ of an included
auto-styler before proposing a specific path to implementing it.

On Tue, Mar 17, 2015 at 6:28 PM, Steven D'Aprano <steve at pearwood.info>
 wrote:

Unlike Go-lang, which has the resources of Google behind it and can just
> pay some person or persons to work on this with as high a priority as
> Google wishes to give it, Python is reliant on volunteers. For major
> proposals like this, you're likely to get much more traction if you are
> volunteering yourself, rather than just hoping to convince somebody else
> to volunteer.
>

Totally understood and agreed. To be upfront, I am not volunteering myself
for this effort just yet.

This is my first time posting to python-ideas. (I subscribed today.) I
gather from Andrew and Steven's responses that ideas discussed on this list
are supposed to be more "fully baked" than my proposal here. My apologies.

For what it's worth, I certainly wouldn't expect myself or anyone else to
invest any effort in an initiative like this if there is a chance that it
will be rejected for reasons that can be explained upfront.

On the other hand, if there is some consensus that this would be good thing
to have at least in theory, then perhaps that would better set the stage
for it to happen in practice. Hence the motivation for my original post.

Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/17d70619/attachment.html>

From nicholas.chammas at gmail.com  Wed Mar 18 02:54:11 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Tue, 17 Mar 2015 21:54:11 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <meafff$tjv$1@ger.gmane.org>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
Message-ID: <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>

On Tue, Mar 17, 2015 at 8:04 PM, Terry Reedy <tjreedy at udel.edu> wrote:

pep8, autopep8, and similar applications do not belong in the stdlib. They
> are easily installed with, for instance, 'pip install autopep8'.

I am not proposing putting autopep8 in the standard library. I am proposing
that an auto-styler be included with Python as a first-class utility. The
emphasis here is on *included* and *first-class*.

That auto-styler may well be based somehow on autopep8, but that is a
separate discussion about *implementation*. I am first checking that the
*idea* is sound. I can?t tell if you had any objections to that idea. Yes,
I know that today I can install and use autopep8 as a third-party utility
using pip, but autopep8 today is neither included with Python nor a
first-class utility.

(FWIW, couldn?t autopep8 somehow be adapted to be an included module that
can be invoked via python -m <https://www.python.org/dev/peps/pep-0338/> ?
e.g. python -m compileall, python -m SimpleHTTPServer ? just as far as the
mechanics are concerned? As I noted in my original post, I?m not in a
position yet to specifically propose how an included auto-styler would
work, but let?s say for argument?s sake that 1) we decided we liked the
idea and 2) autopep8 fit the bill for promotion to an included utility, and
3) we addressed Andrew?s points.)

Nick
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/05fc9634/attachment-0001.html>

From ethan at stoneleaf.us  Wed Mar 18 02:58:08 2015
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 17 Mar 2015 18:58:08 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
Message-ID: <5508DBB0.2010209@stoneleaf.us>

On 03/17/2015 06:54 PM, Nicholas Chammas wrote:

> I am not proposing putting |autopep8| in the standard library. I am proposing that an auto-styler be included with
> Python as a first-class utility. The emphasis here is on /included/ and /first-class/.

Perhaps you could explain what you mean by a /first-class/ utility?

--
~Ethan~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/0c5117f1/attachment.sig>

From nicholas.chammas at gmail.com  Wed Mar 18 03:17:28 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 18 Mar 2015 02:17:28 +0000
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <5508DBB0.2010209@stoneleaf.us>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
Message-ID: <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>

First class means it's like a language feature. Actually, in my case I am
using "first-class" and "included with the language" synonymously. Not sure
if that's appropriate terminology, but that's what I mean.

To explain by rough analogy, Twisted is to asyncio what autopep8 would be
to a first-class auto-styler. (I say "rough" because I understand that
Twisted is not supposed to be replaced entirely by asyncio, but I hope my
point is clear.)

Nick

On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman <ethan at stoneleaf.us> wrote:

> On 03/17/2015 06:54 PM, Nicholas Chammas wrote:
>
> > I am not proposing putting |autopep8| in the standard library. I am
> proposing that an auto-styler be included with
> > Python as a first-class utility. The emphasis here is on /included/ and
> /first-class/.
>
> Perhaps you could explain what you mean by a /first-class/ utility?
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/c8987eb5/attachment.html>

From eliben at gmail.com  Wed Mar 18 04:29:43 2015
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 17 Mar 2015 20:29:43 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
Message-ID: <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>

Interesting coincidence :-)

We (at Google) are just on the verge of releasing an open-source tool for
auto-formatting of Python code. It should be posted to Github within the
next few days or weeks, under a permissive license (Apache 2.0 or something
similar).

The tool works like gofmt and clang-format (the C++/JS version provided as
part of the Clang toolchain) - it only acts on whitespace and performs
re-formatting of code (PEP-8 oriented for Python but it's configurable). It
does not rename variables/classes/functions to conform to any particular
style. The algorithm is based on clang-format, and lib2to3 is used for the
parsing side - so it's very solid. We use it on Python 2 code internally,
but Python 3 support is also planned and should be easy.

We intend to keep actively developing the tool in the open and will be
happy to accept test cases, fixes and patches in general.

So stay tuned!

P.S. I'm also against such a tool being a part of the Python distribution,
at least in the near term. A tool/library has to first prove itself outside
of core for a while, and only when/if it proves to be widely used and
stable, an incorporation into core should be considered. The stdlib is big
as it is, and any additional code is a burden on all core developers.
Incorporation into the stdlib also imposes a very slow pace of
development/release on said library/tool, and hence should only be
considered when it's been extremely stable for a while.

Eli



On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> First class means it's like a language feature. Actually, in my case I am
> using "first-class" and "included with the language" synonymously. Not sure
> if that's appropriate terminology, but that's what I mean.
>
> To explain by rough analogy, Twisted is to asyncio what autopep8 would be
> to a first-class auto-styler. (I say "rough" because I understand that
> Twisted is not supposed to be replaced entirely by asyncio, but I hope my
> point is clear.)
>
> Nick
>
> On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 03/17/2015 06:54 PM, Nicholas Chammas wrote:
>>
>> > I am not proposing putting |autopep8| in the standard library. I am
>> proposing that an auto-styler be included with
>> > Python as a first-class utility. The emphasis here is on /included/ and
>> /first-class/.
>>
>> Perhaps you could explain what you mean by a /first-class/ utility?
>>
>> --
>> ~Ethan~
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/91624c3a/attachment.html>

From tritium-list at sdamon.com  Wed Mar 18 04:42:39 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 17 Mar 2015 23:42:39 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
Message-ID: <5508F42F.8020604@sdamon.com>

What coding style would be used for such a tool?  PEP-8?  What about a 
tool for those of us who believe that PEP-8 got a few things wrong?

What about for the tribal knowledge that pythonistas develop over time 
that are not part of the standard?  Would such a tool include this?

And what of third party tools that do the same thing?  I have no 
evidence to support this, but I have the feeling that when the standard 
library includes something in a new space, third party work in that 
space slows (json and sqlite come to mind).

Good coding style really should not be something we ship in the library, 
it should be something we teach.

2?

On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas 
<nicholas.chammas at gmail.com <mailto:nicholas.chammas at gmail.com>> wrote:
>
>     First class means it's like a language feature. Actually, in my
>     case I am using "first-class" and "included with the language"
>     synonymously. Not sure if that's appropriate terminology, but
>     that's what I mean.
>
>     To explain by rough analogy, Twisted is to asyncio what autopep8
>     would be to a first-class auto-styler. (I say "rough" because I
>     understand that Twisted is not supposed to be replaced entirely by
>     asyncio, but I hope my point is clear.)
>
>     Nick
>
>     On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman <ethan at stoneleaf.us
>     <mailto:ethan at stoneleaf.us>> wrote:
>
>         On 03/17/2015 06:54 PM, Nicholas Chammas wrote:
>
>         > I am not proposing putting |autopep8| in the standard
>         library. I am proposing that an auto-styler be included with
>         > Python as a first-class utility. The emphasis here is on
>         /included/ and /first-class/.
>
>         Perhaps you could explain what you mean by a /first-class/
>         utility?
>
>         --
>         ~Ethan~
>
>         _______________________________________________
>         Python-ideas mailing list
>         Python-ideas at python.org <mailto:Python-ideas at python.org>
>         https://mail.python.org/mailman/listinfo/python-ideas
>         Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <mailto:Python-ideas at python.org>
>     https://mail.python.org/mailman/listinfo/python-ideas
>     Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/b865d415/attachment-0001.html>

From ianlee1521 at gmail.com  Wed Mar 18 07:33:32 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Tue, 17 Mar 2015 23:33:32 -0700
Subject: [Python-ideas] Evolution of Style Guides
Message-ID: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>

tl;dr -- How should style guides evolve?

So this is a post inspired largely by a thread that popped up today [1],
and specifically Alex's message [2]. I didn't want to derail that
conversation (specifically dealing with style checking / correcting as a
part of the standard library), but I did want to start a separate thread
for an idea that's been rattling around in my brain for a while now.
(Disclaimer, I am the current lead maintainer of the pep8 package, so part
of this comes from my work there.)

Namely, how should style guides, and here I'm particularly looking at
PEP-8, evolve?

I was influenced early on in my Python learning by a number of youtube
Python conference talks. In particular  "A Python ?sthetic: Beauty and Why
I Python" by Brandon Rhodes [3] and "Transforming Code into Beautiful,
Idiomatic Python" by Raymond Hettinger [4] had quite an impact on my
thinking while I was moving from C / C++ to Python a couple of years back,
and I highly recommend them.

Somewhat of an open question, I'm not sure I have a particularly hard and
fast opinion on this issue, and I'd love to get feedback from the
community.  In particular, a couple of points that are worth mentioning:

* There are a number of checks that are disabled in pep8 by default, due to
their being somewhat controversial for many users [5] which you can read
about what each of them are for in the docs [6] if you're interested.

* Even within the checker community, there is some discussion about whether
the tool should be rigidly set by the PEP-8 document (I tend to think it
should be) and what that means for folks that don't want to follow a rule
(I just disable it in my code) (see discussion  about the use of
``isinstance()`` at [7]).

* There are several large code bases, the Python stdlib and Twisted for
starters, that don't necessarily follow PEP-8 to the letter, due to
pre-dating PEP-8 or being highly specialized, or etc, and proposals to
bring them more in line have received pushback over their potential for
more harm than good (see [8] for an example) (which is completely valid).

[1] https://mail.python.org/pipermail/python-ideas/2015-March/032545.html
[2] https://mail.python.org/pipermail/python-ideas/2015-March/032556.html
[3] https://www.youtube.com/watch?v=x-kB2o8sd5c
[4] https://www.youtube.com/watch?v=OSGv2VnC0go
[5] https://github.com/jcrocholl/pep8/blob/master/pep8.py#L68
[6] http://pep8.readthedocs.org/en/latest/intro.html#error-codes
[7] https://github.com/jcrocholl/pep8/pull/313
[8] http://bugs.python.org/issue23061#msg233065

~ Ian Lee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/43a7fee6/attachment.html>

From ianlee1521 at gmail.com  Wed Mar 18 07:38:24 2015
From: ianlee1521 at gmail.com (Ian Lee)
Date: Tue, 17 Mar 2015 23:38:24 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <5508F42F.8020604@sdamon.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
Message-ID: <CA+GyjM=j0LLb4Sxtve2EKEio6v7X=umXoDb_hy5+v55J6=YnKQ@mail.gmail.com>

~ Ian Lee

On Tue, Mar 17, 2015 at 8:42 PM, Alexander Walters <tritium-list at sdamon.com>
wrote:

>  What coding style would be used for such a tool?  PEP-8?  What about a
> tool for those of us who believe that PEP-8 got a few things wrong?
>
> What about for the tribal knowledge that pythonistas develop over time
> that are not part of the standard?  Would such a tool include this?
>

And perhaps more to the point, should PEP-8 (and if so how), evolve over
time to incorporate newer ideas that the community has now moved towards
accepting as the "more Pythonic way to do things" now with the benefit of
additional hindsight.  I didn't want to derail this conversation, so I spun
off a separate thread to discuss:
https://mail.python.org/pipermail/python-ideas/2015-March/032557.html


>
> And what of third party tools that do the same thing?  I have no evidence
> to support this, but I have the feeling that when the standard library
> includes something in a new space, third party work in that space slows
> (json and sqlite come to mind).
>
> Good coding style really should not be something we ship in the library,
> it should be something we teach.
>
> 2?
>
>
> On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>     First class means it's like a language feature. Actually, in my case
>> I am using "first-class" and "included with the language" synonymously. Not
>> sure if that's appropriate terminology, but that's what I mean.
>>
>>  To explain by rough analogy, Twisted is to asyncio what autopep8 would
>> be to a first-class auto-styler. (I say "rough" because I understand that
>> Twisted is not supposed to be replaced entirely by asyncio, but I hope my
>> point is clear.)
>>
>>  Nick
>>
>>  On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>>  On 03/17/2015 06:54 PM, Nicholas Chammas wrote:
>>>
>>> > I am not proposing putting |autopep8| in the standard library. I am
>>> proposing that an auto-styler be included with
>>> > Python as a first-class utility. The emphasis here is on /included/
>>> and /first-class/.
>>>
>>> Perhaps you could explain what you mean by a /first-class/ utility?
>>>
>>> --
>>> ~Ethan~
>>>
>>>   _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> _______________________________________________
> Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150317/8cd1b333/attachment.html>

From abarnert at yahoo.com  Wed Mar 18 08:18:02 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 18 Mar 2015 00:18:02 -0700
Subject: [Python-ideas] Evolution of Style Guides
In-Reply-To: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
References: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
Message-ID: <F437CDFB-F6D8-4B32-9902-6902F9C71EF1@yahoo.com>

On Mar 17, 2015, at 11:33 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
> 
> 
> * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested.

I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented.

I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further).

It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)")
as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial.

What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)."

But how exactly can an automated checker "consider" doing something? Clearly always doing it is not what's intended. Even besides the examples, in a document that starts off by telling you it's only a guideline and should not be followed slavishly or rigidly, any part that goes out of its way to say "consider" and follow with "use your own judgment" must have a reason to do so.

So, your judgment can tell you that the No example has too much whitespace. But the pep8 code can't.* And this is, presumably, why the E226 code has no problem with it, which in turn is presumably why users have a problem with the E226 code.

So, I don't think the fact that some of the checks are disabled by default necessarily means there's any dissonance between PEP8 and the Python community, it just means there are things in PEP8 that, though worth following, are hard to codify as rigidly checkable rules.


* Maybe "can't" is a bit strong. But consider: Would your judgment be the same if the variables had longer names? If this were one line of a complicated multi-line expression? Even if you knew the rules that provided the answers to those questions, would you want to code them and turn pep8 into a massive 80s-style expert system?


From me at the-compiler.org  Wed Mar 18 08:30:04 2015
From: me at the-compiler.org (Florian Bruhin)
Date: Wed, 18 Mar 2015 08:30:04 +0100
Subject: [Python-ideas] Evolution of Style Guides
In-Reply-To: <F437CDFB-F6D8-4B32-9902-6902F9C71EF1@yahoo.com>
References: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
 <F437CDFB-F6D8-4B32-9902-6902F9C71EF1@yahoo.com>
Message-ID: <20150318073004.GJ10871@tonks>

* Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> [2015-03-18 00:18:02 -0700]:
> On Mar 17, 2015, at 11:33 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
> > 
> > 
> > * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested.
> 
> I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented.
> 
> I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further).
> 
> It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)")
> as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial.
> 
> What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)."

That was an addition made "recently" to pep8:
https://hg.python.org/peps/diff/37af28ad2972/pep-0008.txt

I guess pep8 adopted by disabling that check by default, rather than
completely removing it.

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/5675a823/attachment.sig>

From abarnert at yahoo.com  Wed Mar 18 09:48:29 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 18 Mar 2015 01:48:29 -0700
Subject: [Python-ideas] Evolution of Style Guides
In-Reply-To: <20150318073004.GJ10871@tonks>
References: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
 <F437CDFB-F6D8-4B32-9902-6902F9C71EF1@yahoo.com>
 <20150318073004.GJ10871@tonks>
Message-ID: <08381887-47D4-4B1D-A326-9F7EF5D7F703@yahoo.com>

On Mar 18, 2015, at 12:30 AM, Florian Bruhin <me at the-compiler.org> wrote:
> 
> * Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> [2015-03-18 00:18:02 -0700]:
>>> On Mar 17, 2015, at 11:33 PM, Ian Lee <ianlee1521 at gmail.com> wrote:
>>> 
>>> 
>>> * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested.
>> 
>> I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented.
>> 
>> I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further).
>> 
>> It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)")
>> as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial.
>> 
>> What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)."
> 
> That was an addition made "recently" to pep8:
> https://hg.python.org/peps/diff/37af28ad2972/pep-0008.txt
> 
> I guess pep8 adopted by disabling that check by default, rather than
> completely removing it.

So PEP 8 evolved to match changing community standards, maybe even in part because of community experience with the pep8 tool, and the tool adapted to those changes... So what needs to change here?

(If Ian was arguing that pep8, autopep8, etc. should remain outside the stdlib so they can continue to adapt at a reasonable pace, then I think I agree, but I think he was suggesting something different; I'll go re-read the post, and apologies in advance if I'm wrong.)

From steve at pearwood.info  Wed Mar 18 09:52:46 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 18 Mar 2015 19:52:46 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzdVODThKHAbjSZFqqcUZjRPwiVW12jDJR4rrww_nX9skg@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com>
 <CAOhmDzdVODThKHAbjSZFqqcUZjRPwiVW12jDJR4rrww_nX9skg@mail.gmail.com>
Message-ID: <20150318085246.GB23538@ando.pearwood.info>

On Wed, Mar 18, 2015 at 12:55:17AM +0000, Nicholas Chammas wrote:

> This is my first time posting to python-ideas. (I subscribed today.) I
> gather from Andrew and Steven's responses that ideas discussed on this list
> are supposed to be more "fully baked" than my proposal here. My apologies.

Not necessarily. This list frequently includes half-baked ideas, but the 
"bigger" the proposal, the more baked it should be to be taken 
seriously. I can proposal that Python should include a fully-functioning 
self-aware computer AI, but unless I actually have a fully-functioning 
self-aware AI written in Python to include, I might as well just wish 
for a flying pony. On the other hand, "Add math.zero through math.ten 
constants" doesn't need anything but a proposal to be discussed.

Still, occassionally its useful to make a really big proposal. Even if 
it doesn't end up being accepted, it might lead somewhere interesting.



-- 
Steve

From mal at egenix.com  Wed Mar 18 09:58:10 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 18 Mar 2015 09:58:10 +0100
Subject: [Python-ideas] Evolution of Style Guides
In-Reply-To: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
References: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
Message-ID: <55093E22.7000200@egenix.com>

On 18.03.2015 07:33, Ian Lee wrote:
> tl;dr -- How should style guides evolve?
> 
> So this is a post inspired largely by a thread that popped up today [1],
> and specifically Alex's message [2]. I didn't want to derail that
> conversation (specifically dealing with style checking / correcting as a
> part of the standard library), but I did want to start a separate thread
> for an idea that's been rattling around in my brain for a while now.
> (Disclaimer, I am the current lead maintainer of the pep8 package, so part
> of this comes from my work there.)
> 
> Namely, how should style guides, and here I'm particularly looking at
> PEP-8, evolve?

By revisiting them every now and them :-)

I think that people sometimes miss the "guide" aspect of PEP 8.

PEP 8 is a guideline for people to follow which results
in readable code. It's not a the only such guideline. Many
companies have their own which usually build on PEP 8 and then
add some extra rules or modify PEP 8 rules.

Whether or not to use a tool enforcing a guideline like PEP 8
is really up to the user or the programming context.

I'm -1 on having a go like feature in Python which mandates
one particular style, but absolutely nothing against tools
applying a formatting specification to some Python code on
demand :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 18 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-03-12: Released mxODBC 3.3.2 ...             http://egenix.com/go71

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From encukou at gmail.com  Wed Mar 18 11:53:52 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 18 Mar 2015 11:53:52 +0100
Subject: [Python-ideas] A macro for easier rich comparisons
Message-ID: <55095940.4010707@gmail.com>

Hello,
Rich comparison functions of many builtin types include a block of 
boilerplate code with a switch on the "op" argument, like this example 
from tupleobject:

     int cmp;
     PyObject *res;
     switch (op) {
     case Py_LT: cmp = vlen <  wlen; break;
     case Py_LE: cmp = vlen <= wlen; break;
     case Py_EQ: cmp = vlen == wlen; break;
     case Py_NE: cmp = vlen != wlen; break;
     case Py_GT: cmp = vlen >  wlen; break;
     case Py_GE: cmp = vlen >= wlen; break;
     default: return NULL; /* cannot happen */
     }
     if (cmp)
         res = Py_True;
     else
         res = Py_False;
     Py_INCREF(res);
     return res;

Each time it's implemented slightly differently: sometimes there are two 
values to compare, sometimes there's a cmp-style 0/1/-1 result to 
process; invalid "op" values are handled in various ways.

Such code is also required in third-party extensions.

I propose adding a public macro to ease this. My version takes two 
C-orderable values and the operation, similar to richcmpfunc:

#define Py_RICHCOMPARE(val1, val2, op) ( \
     ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \
     ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \
     ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \
     ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \
     ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \
     ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \
     (Py_INCREF(Py_NotImplemented), Py_NotImplemented))

(As for the behavior for unknown op: for most cases the best thing to do 
is setting an error and returning NULL, but that doesn't fit into a 
macro. A surprising number of richcmpfunc's in CPython either return 
NULL without error (e.g. tupleobject), or fall through and return 
True/False arbitrarily (as in bytearrayobject). Datetime does an assert.
I think Py_NotImplemented is a good value to return)


I've made a patch with the macro and its use in CPython itself.
Read it as examples of usage: I'm not saying that all these modules 
should start using it immediately (or ever, e.g. for speed reasons), and 
if they do, many could use some refactoring in other code paths.
The patch is here: 
https://github.com/encukou/cpython/compare/master...richcmp


Why I'm interested in this: When porting old code to Python 3, rich 
comparisons a major cause of added boilerplate ? especially when 
wrapping a C library that exposes either trivially comparable types or 
"cmp" functions. With this macro, a typical trivial rich comparison 
function reduces to:

     static PyObject* mytype_richcmp(PyObject *obj1, PyObject *obj2, int op)
     {
         if (mytype_Check(obj1) && mytype_Check(obj2)) {
             return PY_RICHCOMPARE(mytype_get_data(obj1), 
mytype_get_data(obj2), op);
         }
         Py_RETURN_NOTIMPLEMENTED;
     }

... or with `PY_RICHCOMPARE(mytype_cmp(obj1, obj2), 0, op)`.

(Note that even adding this to 3.5/3.6, would help in porting efforts: 
the macro itself is be easy to backport, but there are big benefits in 
having a standard name, known semantics, and official documentation.)


Is this a PEP-worthy idea?

From mal at egenix.com  Wed Mar 18 12:01:07 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 18 Mar 2015 12:01:07 +0100
Subject: [Python-ideas] A macro for easier rich comparisons
In-Reply-To: <55095940.4010707@gmail.com>
References: <55095940.4010707@gmail.com>
Message-ID: <55095AF3.6080201@egenix.com>

On 18.03.2015 11:53, Petr Viktorin wrote:
> ...
> I propose adding a public macro to ease this. My version takes two C-orderable values and the
> operation, similar to richcmpfunc:
> 
> #define Py_RICHCOMPARE(val1, val2, op) ( \
>     ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \
>     ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \
>     ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \
>     ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \
>     ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \
>     ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \
>     (Py_INCREF(Py_NotImplemented), Py_NotImplemented))
> 
> (As for the behavior for unknown op: for most cases the best thing to do is setting an error and
> returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython
> either return NULL without error (e.g. tupleobject), or fall through and return True/False
> arbitrarily (as in bytearrayobject). Datetime does an assert.
> I think Py_NotImplemented is a good value to return)
> ...
> Is this a PEP-worthy idea?

This is a great idea, no need for a PEP :-) Please submit the patch on
the bug tracker and also include a patch for the C API documentation.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 18 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-03-12: Released mxODBC 3.3.2 ...             http://egenix.com/go71

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From luciano at ramalho.org  Wed Mar 18 12:24:24 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Wed, 18 Mar 2015 08:24:24 -0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CAPkN8xKS0=vcyGRcSGEC2RdUA0Ob8wLqSsskK04RLv-d3FN7qA@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
 <CALxg4FWBQ2jDsHcwevAxb0EpiF8h5J1uma2MM93Ofsns1uUMng@mail.gmail.com>
 <CAPkN8xKS0=vcyGRcSGEC2RdUA0Ob8wLqSsskK04RLv-d3FN7qA@mail.gmail.com>
Message-ID: <CALxg4FXgOEJPwo=WLU1XzqVbEdixJDi_dgb-3Y7i2O+-f6LF_g@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:54 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> My mnemonics are "decode from cryptic bytes to human text" and "encode
>> from human text to cryptic bytes".
>
> Maybe I am thinking too low level, but there is no such thing as "human text" in
> computer memory. But I agree that using the words like "abstraction" is not a
> good idea, because Python is used by children of age 7, and things should be
> simpler.

It's a didactic mnemonic ;-).

Although str objects hold codepoints, they *are* used to represent
human text. On the other hand, we convert str to bytes when we need to
talk to the machine: writing to disk, sending over the network...

>
>> A more objective problem is: "from" is a keyword, so it can't be a method name.
>
> I don't see why "from" can not be used as a method name in Python. It can not
> be used as function name, because it will override the keyword, but method
> names are always namespaced with self. or class prefixes.

You can't write it's definition, "def from..." and you can't even
write "x.from" without a SyntaxError (instead of NameError or
AttributeError). Try it in the Python console.

The parser would need to be changed.

Best,

Luciano



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From luciano at ramalho.org  Wed Mar 18 12:36:23 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Wed, 18 Mar 2015 08:36:23 -0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CAPkN8x+VxAzS5D2+XCX5+pjniKAJwth6mFxJEWnqMk53s+jqYw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <CAPkN8x+VxAzS5D2+XCX5+pjniKAJwth6mFxJEWnqMk53s+jqYw@mail.gmail.com>
Message-ID: <CALxg4FXa+CoNi4pZPrcwVbry6J6Zzg6j0V+FF+r8baUXPNHmqw@mail.gmail.com>

On Wed, Mar 18, 2015 at 3:05 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
>> Sometimes we need a simple class to hold some mutable attributes,
>> provide a nice repr, support == for testing, and support iterable
>> unpacking
>
> +1, but I think that the core problem with such proposals is that they
> lack the use cases. The only reason for me to have such class is to
> work with tabular data.
>
> For example, for querying the capability of the system, I need to build
> an inmemory table of features, and then set parameters for each
> feature one by one. sqlite/SQL is an overkill for that, and dicts are
> just not enough to do readable lookups and updates to specific
> cells, so I'd more appreciate a full table class than just its "named
> row" model.

Thanks for your input and example, Anatoly. What would the full table
class offer that could not be easily done with a simple list of items
produced with a plainclass class?

> Practical example that I came up with:
> https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22

Oh, I see you have an update method, so that's the key value add of
the Table class, right?

I see the Table and plainclass as complementary ideas. You could use
plainclass to implement Table more easily. The Table.__iter__ could
return plainclass class instances instead of OrderedDict.

Cheers,

Luciano


-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From encukou at gmail.com  Wed Mar 18 14:48:46 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Wed, 18 Mar 2015 14:48:46 +0100
Subject: [Python-ideas] A macro for easier rich comparisons
In-Reply-To: <55095AF3.6080201@egenix.com>
References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com>
Message-ID: <CA+=+wqD6R5-vg9W250jbwUceCuizD8vJDJ=meYVr0Z3BmO4JGA@mail.gmail.com>

On Wed, Mar 18, 2015 at 12:01 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 18.03.2015 11:53, Petr Viktorin wrote:
>> ...
>> I propose adding a public macro to ease this. My version takes two C-orderable values and the
>> operation, similar to richcmpfunc:
>>
>> #define Py_RICHCOMPARE(val1, val2, op) ( \
>>     ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \
>>     ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \
>>     ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \
>>     ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \
>>     ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \
>>     ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \
>>     (Py_INCREF(Py_NotImplemented), Py_NotImplemented))
>>
>> (As for the behavior for unknown op: for most cases the best thing to do is setting an error and
>> returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython
>> either return NULL without error (e.g. tupleobject), or fall through and return True/False
>> arbitrarily (as in bytearrayobject). Datetime does an assert.
>> I think Py_NotImplemented is a good value to return)
>> ...
>> Is this a PEP-worthy idea?
>
> This is a great idea, no need for a PEP :-) Please submit the patch on
> the bug tracker and also include a patch for the C API documentation.

Issue with patches is here: http://bugs.python.org/issue23699

From techtonik at gmail.com  Wed Mar 18 06:11:46 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 08:11:46 +0300
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <5508F42F.8020604@sdamon.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
Message-ID: <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>

On Wed, Mar 18, 2015 at 6:42 AM, Alexander Walters
<tritium-list at sdamon.com> wrote:
> What coding style would be used for such a tool?  PEP-8?  What about a tool
> for those of us who believe that PEP-8 got a few things wrong?

+1 to make it optional. I use 2 space indent for most of my code.

I'd also add that policy to enforce rigid rules is harmful. The recent
normalization
of Python package versions made some of my apps that use semantic version
invalid for PyPI.

From techtonik at gmail.com  Wed Mar 18 06:33:52 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 08:33:52 +0300
Subject: [Python-ideas] math.tau
In-Reply-To: <CAMpsgwaPXmXsUjTd_ZiJ+ecAVMyhyFuDrjzAKd3oLfMdTsOAFQ@mail.gmail.com>
References: <CAPkN8xKA5weB=T1QOhpjanzO7u4FGajsUzFYkvKrOU2e8toRUQ@mail.gmail.com>
 <CAMpsgwaPXmXsUjTd_ZiJ+ecAVMyhyFuDrjzAKd3oLfMdTsOAFQ@mail.gmail.com>
Message-ID: <CAPkN8x+hbkALXEHEXqmiH7TBs0vJwP9DWd-Zg78Nq2T2vV9=Pg@mail.gmail.com>

On Mon, Mar 16, 2015 at 6:15 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> It looks like you forgot to write the body of your email, there is no
> text, only code?

There is video, which is a much better media for me.

For me the reason to add tau is that Python feels too boring now that
Monty Python
is passed away and stdlib doesn't shine in comparison with awesome-python too.
-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 18 06:54:51 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 08:54:51 +0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CALxg4FWBQ2jDsHcwevAxb0EpiF8h5J1uma2MM93Ofsns1uUMng@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
 <CALxg4FWBQ2jDsHcwevAxb0EpiF8h5J1uma2MM93Ofsns1uUMng@mail.gmail.com>
Message-ID: <CAPkN8xKS0=vcyGRcSGEC2RdUA0Ob8wLqSsskK04RLv-d3FN7qA@mail.gmail.com>

On Tue, Mar 17, 2015 at 7:10 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
> On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> I use the analogy of "encoding abstract numbers into specific bytes" myself.
>
> Your version of the analogy doesn't sound as helpful to me; "abastract
> numbers" sounds even more cryptic than bytes.
>
> My mnemonics are "decode from cryptic bytes to human text" and "encode
> from human text to cryptic bytes".

Maybe I am thinking too low level, but there is no such thing as "human text" in
computer memory. But I agree that using the words like "abstraction" is not a
good idea, because Python is used by children of age 7, and things should be
simpler.

> A more objective problem is: "from" is a keyword, so it can't be a method name.

I don't see why "from" can not be used as a method name in Python. It can not
be used as function name, because it will override the keyword, but method
names are always namespaced with self. or class prefixes.

-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 18 07:05:12 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 09:05:12 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
Message-ID: <CAPkN8x+VxAzS5D2+XCX5+pjniKAJwth6mFxJEWnqMk53s+jqYw@mail.gmail.com>

On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
> Sometimes we need a simple class to hold some mutable attributes,
> provide a nice repr, support == for testing, and support iterable
> unpacking

+1, but I think that the core problem with such proposals is that they
lack the use cases. The only reason for me to have such class is to
work with tabular data.

For example, for querying the capability of the system, I need to build
an inmemory table of features, and then set parameters for each
feature one by one. sqlite/SQL is an overkill for that, and dicts are
just not enough to do readable lookups and updates to specific
cells, so I'd more appreciate a full table class than just its "named
row" model.

Practical example that I came up with:
https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22

-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 18 07:19:27 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 09:19:27 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
Message-ID: <CAPkN8xLdizm+iFQcOq348m2k02=WteC5tRxAJg97+Nb-C7o9=Q@mail.gmail.com>

On Mon, Mar 16, 2015 at 8:18 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> They have a build system genius who wrote Tup working for them

I like the conceptual brevity of the Tup page http://gittup.org/tup/
It is the core of that I am try to bring from SCons internals to the
surface. It is very often that you need to process some DAG
and touch nodes based on their changed dependencies.

The problem is so common that it is even worth to spend time
on visualization for it. So far no build system has this stuff, and
I run of time to complete the SCons plumbing.
-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 18 07:21:23 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 09:21:23 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
Message-ID: <CAPkN8x+nWJ9yYqujoi1DGPwA5YbqvMiMrMnLjxZZ1o3GEOTTbA@mail.gmail.com>

On Mon, Mar 16, 2015 at 8:39 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> One more note: so adding a python dependency is a pain. But the current
> system requires separate stems for Windows and *nix-y systems, and who knows
> what for hypothetical systems that don't support autotools. So maybe a
> python-based build system that worked on all the common platforms, with an
> additional makefile that you have to hand-mangle to bootstrap a new platform
> isn't any more to maintain.

Manual instruction with lists of files that can be parsed out of it is
better than
Makefile IMO. Pumped up Markdown or something..
-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 18 08:18:26 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 10:18:26 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
Message-ID: <CAPkN8x+iw3f5-MR8wrWPF+PKAxeWgzBh+F8ikf6A9o8G4ooVjw@mail.gmail.com>

On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
>>
>> "Autotools has reached the Chandrasekhar mass limit of software and
>> become a black hole of suck that consumes endless hours of bright
>> peoples? time without any good result. It is well past time for it to
>> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
>>
>> I couldn't sum up this better. It looks even worse from Windows machine.
>
> Is it even used for building Python on Windows (for the "normal" python.org
> builds)?

There was some success with MinGW toolchain, but I am not tracking if it was
integrated into official source. The FUD was that compiled extensions are not
compatible between MSVC builds and MinGW builds, but I could not confirm
that. If I understand correctly, the incompatibility is only for C++ programs,
because of strange SEH strategy and proprietary name mangling.

> But to miss-paraphrase Winston Churchill [1]:
>
> "Autotools is the worst build system except for all the others."
>
> Is there really something better out there?

A lot of attempts are made, but reviewing them requires a lot of
time and effort.

From techtonik at gmail.com  Wed Mar 18 08:21:46 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 10:21:46 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
Message-ID: <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>

On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>> But to miss-paraphrase Winston Churchill [1]:
>>
>> "Autotools is the worst build system except for all the others."
>>
>> Is there really something better out there?
>>
>
> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...

Is there anything Javascriptey? Python needs to move to web. I heard
something about Grunt, which was replaced by Gulp, but no idea if they
really mature enough to include "system features detection" and "user
configured builds" (or work with C/C++ compilers at all).

From techtonik at gmail.com  Wed Mar 18 08:31:37 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 10:31:37 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CAO41-mO+RXkXV+35=buBgcY9accnPzH-sDjc4Uwmo_9n98Mgzw@mail.gmail.com>
 <loom.20150316T192520-981@post.gmane.org>
 <nad-D58D3B.15451016032015@news.gmane.org>
 <CALGmxEJLmU6Jy1LiRo9qbZs09WSsvK2yhGuWWQid3Q=UJC-8tg@mail.gmail.com>
Message-ID: <CAPkN8x+G=BvaqyfbEKBcZWYN936Ytnbx8sSzcHBrbS=0BrVudg@mail.gmail.com>

On Tue, Mar 17, 2015 at 2:45 AM, Chris Barker <chris.barker at noaa.gov> wrote:
>
> This doesn't seem to me to be a winner takes all proposition -- there is no
> reason someone couldn't set up a new build system for Python, get it all
> working nicely on their platform(s) of choice, persuade others how wonderful
> it is, get contributions, etc.
>
> If it turns out to really be better/easier etc. than autoconf, then , and
> only then, would it make any sense to talk about replacing the existing
> official build system. And even then, the two could live in parallel for
> quite some time...

Both Blender and Wesnoth have SCons and CMake running in parallel
for a long time. Both camps are just maintaining their own chains and it
works well.

From techtonik at gmail.com  Wed Mar 18 08:36:31 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 10:36:31 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
 <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
 <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAPkN8x+KgOCZLJOv7q1PyBr8bH840kFXr8+bBJPsJQox3uZ-3A@mail.gmail.com>

On Tue, Mar 17, 2015 at 5:27 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> systems out there with the accumulated knowledge of platforms that has
> been assembled in Autotools over the decades.

Given that Autotools still doesn't have native Windows support, the value of
that knowledge is overstated.

From techtonik at gmail.com  Wed Mar 18 08:44:04 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Mar 2015 10:44:04 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAO41-mOSSXKZaScQvA5qESPWNaanSbTtJnZECCcofAEt+BO-+w@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
 <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
 <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAO41-mOSSXKZaScQvA5qESPWNaanSbTtJnZECCcofAEt+BO-+w@mail.gmail.com>
Message-ID: <CAPkN8xKorUS+Lf79o4VKrKkPHjax8Habsf--swvN9LC88E_SRQ@mail.gmail.com>

On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <stephen at xemacs.org>
> wrote:
>>
>> Ryan Gonzalez writes:
>>
>>  > I have used fbuild a LOT
>>
>> Actually, I suspect that in the relevant sense, you haven't used it a
>> lot.  Have you used it for a single program on
>>
>> 1.  more than 3 CPUs, AND
>> 2.  on each CPU, more than one (and preferably more than two) OSes, AND
>> 3.  for each CPU-OS combination, more than two different configurations
>>     of the program being built, AND
>> 4.  for each CPU-OS-configuration combination, at least 3 versions of
>>     the program source (preferably spanning a major version bump)?
>>
>
> No...but I know that Felix uses it for iPhone, OSX, Linux, ... with
> separation of host and target and multiple configurations.

What or who is Felix?

> This is also SCons. Everybody loved it until they had to maintain it. :)

I know a blind girl, who made a voice synthesizer, which works on
Windows, Linux and Android, and she used SCons to build it. That
gave me another reason to stick with this build system and try to
enhance it.

> SCons is very annoying. CMake is weird and only really useful for those
> huge, 20,000 file C++ projects.

For me the advantage of SCons is that I already know its codebase.
There a lot of room for improvement if you can tag yourself as a Python
hacker it addition to inherent potential user property.

In any, I'd appreciate the feedback on most annoying SCons feature,
because it will give a criteria for comparing it with others and for future
development.
-- 
anatoly t.

From me at the-compiler.org  Wed Mar 18 16:21:50 2015
From: me at the-compiler.org (Florian Bruhin)
Date: Wed, 18 Mar 2015 16:21:50 +0100
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
 <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
Message-ID: <20150318152150.GN10871@tonks>

* anatoly techtonik <techtonik at gmail.com> [2015-03-18 10:21:46 +0300]:
> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> >>
> >> But to miss-paraphrase Winston Churchill [1]:
> >>
> >> "Autotools is the worst build system except for all the others."
> >>
> >> Is there really something better out there?
> >>
> >
> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
> 
> Is there anything Javascriptey? Python needs to move to web. I heard
> something about Grunt, which was replaced by Gulp, but no idea if they
> really mature enough to include "system features detection" and "user
> configured builds" (or work with C/C++ compilers at all).

What does "Python needs to move to web" to do with it's build system
being "Javascriptey"?

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/9faf0f5c/attachment.sig>

From rymg19 at gmail.com  Wed Mar 18 17:11:33 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 18 Mar 2015 11:11:33 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
 <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
Message-ID: <CAO41-mNj6yRKvKKLr=CWXCz+=aVrSbdVe36+SeajyRaT6PAsrw@mail.gmail.com>

Then you need Node.js installed, which is kind of big.

On Wed, Mar 18, 2015 at 2:21 AM, anatoly techtonik <techtonik at gmail.com>
wrote:

> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> >>
> >> But to miss-paraphrase Winston Churchill [1]:
> >>
> >> "Autotools is the worst build system except for all the others."
> >>
> >> Is there really something better out there?
> >>
> >
> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>
> Is there anything Javascriptey? Python needs to move to web. I heard
> something about Grunt, which was replaced by Gulp, but no idea if they
> really mature enough to include "system features detection" and "user
> configured builds" (or work with C/C++ compilers at all).
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/90e95d4a/attachment.html>

From rymg19 at gmail.com  Wed Mar 18 17:18:39 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 18 Mar 2015 11:18:39 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8xKorUS+Lf79o4VKrKkPHjax8Habsf--swvN9LC88E_SRQ@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CALGmxE+-MKdDG2OEZrkTJwXjJvOdmyRuH2_FiCX3rFv=LRSpxw@mail.gmail.com>
 <CAPTjJmp08SN1Z3hoEQ8E-Uo2F7uC2M3tCL=mDgbYtVyMf_Bcng@mail.gmail.com>
 <CALGmxELwHXWmJTNEPXn9nD648hARhZPA0gtS0vA5HRUyKKUeEA@mail.gmail.com>
 <CAPTjJmoqRhJ7BabGChhnyRygBJr0sEBVvtoDotkgUT_737_nCg@mail.gmail.com>
 <CALGmxELqViJT9sW9aBSEYyswS16uz-E_T0DNRqyt2QDPO2t-fg@mail.gmail.com>
 <CAO41-mNWNtRNBXWCrGgnU20K9Zy-n5A6_nbOkkp3QqKj+_=RiQ@mail.gmail.com>
 <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAO41-mOSSXKZaScQvA5qESPWNaanSbTtJnZECCcofAEt+BO-+w@mail.gmail.com>
 <CAPkN8xKorUS+Lf79o4VKrKkPHjax8Habsf--swvN9LC88E_SRQ@mail.gmail.com>
Message-ID: <CAO41-mOTbcHuCJh4uXx5tUh=nNfjp2fp_-SOsNtOX5ODwEe06Q@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:44 AM, anatoly techtonik <techtonik at gmail.com>
wrote:

> On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> > On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull <stephen at xemacs.org
> >
> > wrote:
> >>
> >> Ryan Gonzalez writes:
> >>
> >>  > I have used fbuild a LOT
> >>
> >> Actually, I suspect that in the relevant sense, you haven't used it a
> >> lot.  Have you used it for a single program on
> >>
> >> 1.  more than 3 CPUs, AND
> >> 2.  on each CPU, more than one (and preferably more than two) OSes, AND
> >> 3.  for each CPU-OS combination, more than two different configurations
> >>     of the program being built, AND
> >> 4.  for each CPU-OS-configuration combination, at least 3 versions of
> >>     the program source (preferably spanning a major version bump)?
> >>
> >
> > No...but I know that Felix uses it for iPhone, OSX, Linux, ... with
> > separation of host and target and multiple configurations.
>
> What or who is Felix?
>
>
A programming language. http://felix-lang.org/. The code base is pretty
huge; since it comes with most of its dependencies (re2, Judy, etc.), it
consists of about 1600 source files. Fbuild works pretty well, other than
the fact that the language's creator is 100% positive that dynamic
languages are cruel, evil, and unmaintainable.


> > This is also SCons. Everybody loved it until they had to maintain it. :)
>
> I know a blind girl, who made a voice synthesizer, which works on
> Windows, Linux and Android, and she used SCons to build it. That
> gave me another reason to stick with this build system and try to
> enhance it.
>
> > SCons is very annoying. CMake is weird and only really useful for those
> > huge, 20,000 file C++ projects.
>
> For me the advantage of SCons is that I already know its codebase.
> There a lot of room for improvement if you can tag yourself as a Python
> hacker it addition to inherent potential user property.
>
> In any, I'd appreciate the feedback on most annoying SCons feature,
> because it will give a criteria for comparing it with others and for future
> development.
>

Ah, right...you work on SCons...please don't take anything I said
personally. :)

It's just that 99% of the time I try to build a project with SCons, some
horribly stupid thing goes wrong because of some weird API call. Also,
variant directories are WAY to weird for me.

It's also kind of slow. I'm one of those people that gets impatient if I
have more than 1.01s for my build to complete, which is why I don't really
like make too much.


> --
> anatoly t.
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/7fe5665a/attachment-0001.html>

From nicholas.chammas at gmail.com  Wed Mar 18 17:52:07 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Wed, 18 Mar 2015 12:52:07 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
Message-ID: <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>

On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky <eliben at gmail.com> wrote:

> We (at Google) are just on the verge of releasing an open-source tool for
> auto-formatting of Python code.


Interesting. Are y'all familiar with `autopep8`? What are the differences
between the tool y'all are releasing and `autopep8`?


On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky <eliben at gmail.com> wrote:

> P.S. I'm also against such a tool being a part of the Python distribution,
> at least in the near term. A tool/library has to first prove itself outside
> of core for a while, and only when/if it proves to be widely used and
> stable, an incorporation into core should be considered. The stdlib is big
> as it is, and any additional code is a burden on all core developers.
> Incorporation into the stdlib also imposes a very slow pace of
> development/release on said library/tool, and hence should only be
> considered when it's been extremely stable for a while.


This makes sense. In fact, I recently watched a keynote that Guido gave and he
said something similar <https://youtu.be/EBRMq2Ioxsc?t=49m23s>.


On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <tritium-list at sdamon.com
> wrote:

> What coding style would be used for such a tool?  PEP-8?  What about a
> tool for those of us who believe that PEP-8 got a few things wrong?
>
> What about for the tribal knowledge that pythonistas develop over time
> that are not part of the standard?  Would such a tool include this?
>

I might be wrong, but I see PEP 8 as a living document. If the community
believes parts of it are out of date, I would think it's better to update
PEP 8 (or issue a replacement PEP) to reflect current thinking.

Any auto-styling tool included in the standard library would work best if
it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8
was meant to be a spec for programmatic implementation, but you get the
idea).

If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are
agreed to be "wrong", then PEP 8 should be updated accordingly. Any
auto-styling tool, I would hope, would just follow from PEP 8 (or a similar
document). The tool should not be the specification.


On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <tritium-list at sdamon.com
> wrote:
>
> And what of third party tools that do the same thing?  I have no evidence
> to support this, but I have the feeling that when the standard library
> includes something in a new space, third party work in that space slows
> (json and sqlite come to mind).
>

I see what you mean. Several people have made comments to this effect, and
this is a good point.

I guess this is a separate discussion, but is there some way to offer users
the advantages of having a library be included with Python without imposing
the 2-year release cycle and additional burden on the library's developers?

This seems to be the root of the concern about including, for example,
something like autopep8 with Python.


On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <tritium-list at sdamon.com
> wrote:

> Good coding style really should not be something we ship in the library,
> it should be something we teach.
>

This point is orthogonal to the discussion of whether or not to package an
auto-styler, but there are many style points that are so mechanical I see
little benefit in expending energy on teaching them. Best would be to
automate as much of it away so that people can focus on bigger problems
like, what is the right abstraction, did I design this API well, etc.

Put differently, consider the following straw man: "Good memory management
should not be something we ship in the language, it should be something we
teach."

There is always something to teach. But when we can automate part of the
problem away, I think we generally should.


On Wed, Mar 18, 2015 at 2:38 AM, Ian Lee <ianlee1521 at gmail.com> wrote:

> And perhaps more to the point, should PEP-8 (and if so how), evolve over
> time to incorporate newer ideas that the community has now moved towards
> accepting as the "more Pythonic way to do things" now with the benefit of
> additional hindsight.


I consider PEP 8 to be a living document. That is, it is already changing
with time. For example, I see that though PEP 8 was created in 2001,
it was updated
in 2013 <https://www.python.org/dev/peps/pep-0008/#id8>. I assume it will
continue to be updated as accepted standards change.


On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano <steve at pearwood.info>
 wrote:

> Not necessarily. This list frequently includes half-baked ideas, but the
> "bigger" the proposal, the more baked it should be to be taken
> seriously.
>

Duly noted!

Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/f3b25094/attachment.html>

From njs at pobox.com  Wed Mar 18 18:31:13 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 18 Mar 2015 10:31:13 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8x+iw3f5-MR8wrWPF+PKAxeWgzBh+F8ikf6A9o8G4ooVjw@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAPkN8x+iw3f5-MR8wrWPF+PKAxeWgzBh+F8ikf6A9o8G4ooVjw@mail.gmail.com>
Message-ID: <CAPJVwBkzw++=LAugaQBz6A-qFzr_XAhieMFR8vb+89oVXDWDPg@mail.gmail.com>

On Mar 18, 2015 8:14 AM, "anatoly techtonik" <techtonik at gmail.com> wrote:
>
> On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker at noaa.gov>
wrote:
> > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik at gmail.com>
> > wrote:
> >>
> >> "Autotools has reached the Chandrasekhar mass limit of software and
> >> become a black hole of suck that consumes endless hours of bright
> >> peoples? time without any good result. It is well past time for it to
> >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
> >>
> >> I couldn't sum up this better. It looks even worse from Windows
machine.
> >
> > Is it even used for building Python on Windows (for the "normal"
python.org
> > builds)?
>
> There was some success with MinGW toolchain, but I am not tracking if it
was
> integrated into official source. The FUD was that compiled extensions are
not
> compatible between MSVC builds and MinGW builds, but I could not confirm
> that. If I understand correctly, the incompatibility is only for C++
programs,
> because of strange SEH strategy and proprietary name mangling.

This isn't FUD, alas. mingw in almost all of its variants uses effectively
a different "libc" than MSVC, so code compiled with two compilers can't
even agree on the meaning of file numbers. It is mostly possible to set up
a mingw-w64 build that produces MSVC-compatible extensions, but it requires
using a configuration that's undocumented and untested upstream (I.e. you
will run into mingw bugs). NumPy *may* switch to this at some point --
we've been working on it for like a year now -- but it's extremely
non-trivial.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/744b1b5f/attachment.html>

From barry at python.org  Wed Mar 18 19:22:02 2015
From: barry at python.org (Barry Warsaw)
Date: Wed, 18 Mar 2015 14:22:02 -0400
Subject: [Python-ideas] Evolution of Style Guides
References: <CA+GyjMnZzgtQ=vFNZW1Mnk=G7V-fPK8Yx-J1OV-ZSqKRAxO=UQ@mail.gmail.com>
 <55093E22.7000200@egenix.com>
Message-ID: <20150318142202.0e2c302d@anarchist.wooz.org>

On Mar 18, 2015, at 09:58 AM, M.-A. Lemburg wrote:

>PEP 8 is a guideline for people to follow which results
>in readable code. It's not a the only such guideline. Many
>companies have their own which usually build on PEP 8 and then
>add some extra rules or modify PEP 8 rules.

Exactly.  Also PEP 8 is deliberately noncommittal about certain style points.
It shouldn't dot every t and cross every i <wink>.

Besides, PEP 8 was originally - and as its introduction still states -
intended for the coding conventions of the standard library, where consistency
is as important as it is lacking in practice. ;)  You're supposed to define
the coding standards for your own projects, and it's fine to include PEP 8 by
reference.  That's what we do for example with GNU Mailman (although it needs
an update).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/2cf4bd12/attachment-0001.sig>

From abarnert at yahoo.com  Wed Mar 18 21:38:58 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 18 Mar 2015 13:38:58 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
Message-ID: <5F528E38-D44A-4A7A-9FC2-25232F7D26AA@yahoo.com>

On Mar 18, 2015, at 9:52 AM, Nicholas Chammas <nicholas.chammas at gmail.com> wrote:
> 
> I might be wrong, but I see PEP 8 as a living document. If the community believes parts of it are out of date, I would think it's better to update PEP 8 (or issue a replacement PEP) to reflect current thinking.
> 
> Any auto-styling tool included in the standard library would work best if it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8 was meant to be a spec for programmatic implementation, but you get the idea).
> 
> If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are agreed to be "wrong", then PEP 8 should be updated accordingly. Any auto-styling tool, I would hope, would just follow from PEP 8 (or a similar document). The tool should not be the specification.

I agree with all of this.

But that implies that it's worth someone making a spec out of PEP 8. That implies going through PEP 8, getting a sense of what parts are (unambiguously and uncontroversially) mechanizable, and reorganizing and minimally rewriting it to get that sense across. Having numbered rules, and named anchors for each one in the document (or at least to narrower sections, so you can link to something more specific than "Programming Recommendations") would also be helpful. 

I'm not sure such a thing would be accepted if you did it. But it seems like if it were rejected, the standard styler would be no more "standard" than what already exists--and likely to be rejected for the same reasons.

> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <tritium-list at sdamon.com> wrote: 
>> 
>> And what of third party tools that do the same thing?  I have no evidence to support this, but I have the feeling that when the standard library includes something in a new space, third party work in that space slows (json and sqlite come to mind).
>  
> I see what you mean. Several people have made comments to this effect, and this is a good point.
> 
> I guess this is a separate discussion, but is there some way to offer users the advantages of having a library be included with Python without imposing the 2-year release cycle and additional burden on the library's developers?

It depends what you mean.

First, if pip comes with Python, and the library doesn't require C compilation, users already pretty much have that advantage. (And the really nice thing is that 2.7, 3.4, etc. users also have that advantage, which isn't true for something added to the stdlib in 3.5.)

It might even be possible to have language in the official docs and/or the Getting Started or other parts of the site that suggest (as strongly as seems appropriate) that users "pip install autopep8" and use it. (I'm saying "autopep8" as a shorthand for "autopep8, or whatever alternative you'd end up suggesting if you did".)

If the suggestion that IDLE add a graphical package manager wrapping pip goes forward, it could presumably add some kind of "recommended packages" feature, maybe even suggesting it at first run, but it seems like you'd have to jump-start the discussion among IDLE developers and users and then bring it back here, or wait and see if they add such a thing so you can suggest adding autopep8 to the list.

Beyond that, there's really no way to be "included without being included". If it's in the standard source and binary distributions, it's part of the release cycle. I suppose you could have something that came with Python but could be updated with pip, with users strongly suggested to update it--but that doesn't seem like a good idea; if you want to get everyone to pip install -U it, the best way to do that is not to give them an obsolete version, but to just not give it to them. And bootstrapping it the same way as pip doesn't seem reasonable (it's reasonable for pip because "just pip install pip" obviously doesn't work--sometimes, special cases _are_ special enough...). 

One thing that might be reasonable is to include an autopep8 script that runs autopep8 if present, and otherwise just tells you it's not present and suggests pip installing it (similar to what Apple does with Java and X11 binaries). Of course nothing else in Python works that way today, and I suspect a lot of people would resist it, but technically it's certainly doable, if you want to make a case for it.

Finally, remember that not everyone gets Python through an official Python.org installer. Almost every non-Windows, non-mobile platform comes with a "system python", and also has a package manager (possibly third-party, like Homebrew on Mac) that many people who want a different Python turn to. And some people build from source. And others (including on Windows) use third-party Python distributions like Enthought or PythonWin. To these users, what "comes with" Python is not the same thing that Python.org crams into the Windows and Mac installers--and can be much more up-to-date (but isn't automatically, of course). Apple's system Python comes with PyObjC. Some linux distros split off parts of the stdlib into packages named "python-spam" and then have non-stdlib packages in the same repo named "python-eggs". Enthought comes with the SciPy stack, and PythonWin with PyWin32. Anaconda has its own package management system, and a set of recommended extra packages. And so on.

> This seems to be the root of the concern about including, for example, something like autopep8 with Python.
> 
> 
>> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <tritium-list at sdamon.com> wrote:
>> Good coding style really should not be something we ship in the library, it should be something we teach.
> 
> This point is orthogonal to the discussion of whether or not to package an auto-styler, but there are many style points that are so mechanical I see little benefit in expending energy on teaching them. Best would be to automate as much of it away so that people can focus on bigger problems like, what is the right abstraction, did I design this API well, etc.
> 
> Put differently, consider the following straw man: "Good memory management should not be something we ship in the language, it should be something we teach."

That's a great example, because it's a little of both. Python automates memory management as far as is reasonable, and makes it easy for people who need to do more (clear recommendations on when to del/reassign variables, tools like weakrefdict, etc.), so some users don't need to be taught anything, and the more advanced users can be taught more easily than in, say, Objective C.

> There is always something to teach. But when we can automate part of the problem away, I think we generally should.
> 
>> On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> Not necessarily. This list frequently includes half-baked ideas, but the
>> "bigger" the proposal, the more baked it should be to be taken
>> seriously.
> 
> Duly noted!

Also keep in mind that, to whatever extent it isn't fully-baked, it should be clear that it is at least bakeable, and that you look willing and able to do the baking. That's a bit harder for a brand new list member than for, say, Guido (his initial proposal for asyncio, for example, wouldn't have gotten the same interest of it was your initial proposal), but it's doable, and necessary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/d4750934/attachment.html>

From tjreedy at udel.edu  Wed Mar 18 23:17:12 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 18 Mar 2015 18:17:12 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
Message-ID: <mecthj$jqe$1@ger.gmane.org>

On 3/18/2015 12:52 PM, Nicholas Chammas wrote:

> I might be wrong, but I see PEP 8 as a living document.

PEP 8 is Guido's view on how new CPython stdlib code should be written, 
with input from other core developers.  "This document gives coding 
conventions for the Python code comprising the standard library in the 
main Python distribution."  It changes as he and we change our views. It 
is against Guido's policy to blindly change old stdlib code to follow 
PEP 8.  (Note that autoPEP8 only changes whitespace, which is a small 
part of PEP 8 recommendations, and the safest aspect of code to change.)

 > If the community
> believes parts of it are out of date, I would think it's better to
> update PEP 8 (or issue a replacement PEP) to reflect current thinking.

The 'community' is free to follow it or not, but not to change it. Other 
people and organizations have written their own style guides.  Good for 
them.

> If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are
> agreed to be "wrong", then PEP 8 should be updated accordingly.

PEP 8 is by definition 'right' until Guido decides otherwise, with input 
from other core devs.

> I guess this is a separate discussion, but is there some way to offer
> users the advantages of having a library be included with Python without
> imposing the 2-year release cycle and additional burden on the library's
> developers?

Use pip.  It was added to make pypi (and other repositories) an 
easy-to-use extension of the stdlib, without imposing stdlib 
requirements and burdens.  We are working on making pip even easier to use.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Thu Mar 19 00:21:23 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 18 Mar 2015 19:21:23 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
Message-ID: <med19u$gni$1@ger.gmane.org>

On 3/17/2015 9:54 PM, Nicholas Chammas wrote:

> I am not proposing putting |autopep8| in the standard library.

You gave that as an example and I went with it the absence of there 
being anything else concrete, even a name.  Substitute a mythical 
'autostyle' if you wish.

> I am
> proposing that an auto-styler be included with Python as a first-class
> utility. The emphasis here is on /included/ and /first-class/.

We are wary about including anything more with the CPython distribution. 
  It is already too large to be managed by the core developer group as 
well as we would like.

I disagree than not being included in 'cpython' necessarily makes 
something less than first-class, as you seemed to say in another response.

> That auto-styler may well be based somehow on |autopep8|, but that is a
> separate discussion about /implementation/. I am first checking that the
> /idea/ is sound. I can?t tell if you had any objections to that idea.

I do.  We should be careful about adding anything.  When we do, we 
should add libraries, not applications.  We should especially not 
officially endorse one of many opinion-based competitors.

Pip is the intended way to include 3rd party modules, as one chooses, as 
virtual members of the stdlib or distribution.

FWIW, CPython already comes with a minimal autostyle script, 
Tools/Scripts/reindent.py, which is used to enforce minimal requirements 
for committing new or altered stdlib .py files to the cpython hg repository.


> couldn?t |autopep8| somehow be adapted to be an included module
> that can be invoked via |python -m|

Today it is invoked with 'autopep8'.  Why would you want to have to add 
'python -m'?  For Idle, I would like to move in the other direction and 
allow Idle to be started on a command line with 'idle' rather than 
'python -m idlelib'.  (This is possible today, but only if one adds 
'.../python/lib/idlelib' to PATH, or, on windows, moves an edited 
version of idle.bat to /Scripts.)

-- 
Terry Jan Reedy



From eliben at gmail.com  Thu Mar 19 00:30:10 2015
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 18 Mar 2015 16:30:10 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
Message-ID: <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>

On Wed, Mar 18, 2015 at 9:52 AM, Nicholas Chammas <
nicholas.chammas at gmail.com> wrote:

> On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky <eliben at gmail.com> wrote:
>
>> We (at Google) are just on the verge of releasing an open-source tool for
>> auto-formatting of Python code.
>
>
> Interesting. Are y'all familiar with `autopep8`?
>

Yes, very familiar.


> What are the differences between the tool y'all are releasing and
> `autopep8`?
>
>
autopep8's scope is both smaller and larger. It only detects PEP8
violations and attempts to fix them, not restricting itself to whitespace.
Our tool works more like the gofmt tool mentioned in the beginning of the
thread - it pretty much only cares about formatting, and it auto-reformats
the code to its own liking, covering more whitespace-related stuff than
PEP8. Basically the idea is "forget about formatting your code, just run it
through the tool", which is a similar philosophy as gofmt (and
clang-format) and has been hugely successful for those tools/languages.

Eli



>
> On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky <eliben at gmail.com> wrote:
>
>> P.S. I'm also against such a tool being a part of the Python
>> distribution, at least in the near term. A tool/library has to first prove
>> itself outside of core for a while, and only when/if it proves to be widely
>> used and stable, an incorporation into core should be considered. The
>> stdlib is big as it is, and any additional code is a burden on all core
>> developers. Incorporation into the stdlib also imposes a very slow pace of
>> development/release on said library/tool, and hence should only be
>> considered when it's been extremely stable for a while.
>
>
> This makes sense. In fact, I recently watched a keynote that Guido gave
> and he said something similar <https://youtu.be/EBRMq2Ioxsc?t=49m23s>.
>
>
> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <
> tritium-list at sdamon.com> wrote:
>
>> What coding style would be used for such a tool?  PEP-8?  What about a
>> tool for those of us who believe that PEP-8 got a few things wrong?
>>
>> What about for the tribal knowledge that pythonistas develop over time
>> that are not part of the standard?  Would such a tool include this?
>>
>
> I might be wrong, but I see PEP 8 as a living document. If the community
> believes parts of it are out of date, I would think it's better to update
> PEP 8 (or issue a replacement PEP) to reflect current thinking.
>
> Any auto-styling tool included in the standard library would work best if
> it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8
> was meant to be a spec for programmatic implementation, but you get the
> idea).
>
> If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are
> agreed to be "wrong", then PEP 8 should be updated accordingly. Any
> auto-styling tool, I would hope, would just follow from PEP 8 (or a similar
> document). The tool should not be the specification.
>
>
> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <
> tritium-list at sdamon.com> wrote:
>>
>> And what of third party tools that do the same thing?  I have no evidence
>> to support this, but I have the feeling that when the standard library
>> includes something in a new space, third party work in that space slows
>> (json and sqlite come to mind).
>>
>
> I see what you mean. Several people have made comments to this effect, and
> this is a good point.
>
> I guess this is a separate discussion, but is there some way to offer
> users the advantages of having a library be included with Python without
> imposing the 2-year release cycle and additional burden on the library's
> developers?
>
> This seems to be the root of the concern about including, for example,
> something like autopep8 with Python.
>
>
> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters <
> tritium-list at sdamon.com> wrote:
>
>> Good coding style really should not be something we ship in the library,
>> it should be something we teach.
>>
>
> This point is orthogonal to the discussion of whether or not to package an
> auto-styler, but there are many style points that are so mechanical I see
> little benefit in expending energy on teaching them. Best would be to
> automate as much of it away so that people can focus on bigger problems
> like, what is the right abstraction, did I design this API well, etc.
>
> Put differently, consider the following straw man: "Good memory management
> should not be something we ship in the language, it should be something we
> teach."
>
> There is always something to teach. But when we can automate part of the
> problem away, I think we generally should.
>
>
> On Wed, Mar 18, 2015 at 2:38 AM, Ian Lee <ianlee1521 at gmail.com> wrote:
>
>> And perhaps more to the point, should PEP-8 (and if so how), evolve over
>> time to incorporate newer ideas that the community has now moved towards
>> accepting as the "more Pythonic way to do things" now with the benefit of
>> additional hindsight.
>
>
> I consider PEP 8 to be a living document. That is, it is already changing
> with time. For example, I see that though PEP 8 was created in 2001, it was updated
> in 2013 <https://www.python.org/dev/peps/pep-0008/#id8>. I assume it will
> continue to be updated as accepted standards change.
>
>
> On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano <steve at pearwood.info>
>  wrote:
>
>> Not necessarily. This list frequently includes half-baked ideas, but the
>> "bigger" the proposal, the more baked it should be to be taken
>> seriously.
>>
>
> Duly noted!
>
> Nick
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150318/c1bfba78/attachment.html>

From szport at gmail.com  Thu Mar 19 08:04:38 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Thu, 19 Mar 2015 00:04:38 -0700 (PDT)
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <55086248.2010009@trueblade.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
Message-ID: <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>



???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith 
???????:
>
> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: 
> > Sometimes we need a simple class to hold some mutable attributes, 
> > provide a nice repr, support == for testing, and support iterable 
> > unpacking, so you can write: 
> > 
> >>>> p = Point(3, 4) 
> >>>> x, y = p 
> > 
> > That's very much like the classes built by namedtuple, but mutable. 
>
> https://pypi.python.org/pypi/namedlist 
>
> It also adds default values to the generated constructor, which may or 
> may not be desirable. But if used exactly like collections.namedtuple, 
> it ignores the default values. 
>
> Eric. 
>
> Since named tuple is considered as an object that is a tuple with 
attribute access. 
The mutable alternative could be considered as an array with attribute 
access.
Array in this context is tuple-like object that support assign operation.
Since python have not such object there are different approaches tomutable  
named tuple alternatives.

One should note that particular property of named tuple is memory saving.
So one can expect similar property of mutable named tuple too.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/8342986a/attachment-0001.html>

From abarnert at yahoo.com  Thu Mar 19 09:37:26 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 01:37:26 -0700
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
Message-ID: <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>

On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
> 
> 
> 
> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????:
>> 
>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: 
>> > Sometimes we need a simple class to hold some mutable attributes, 
>> > provide a nice repr, support == for testing, and support iterable 
>> > unpacking, so you can write: 
>> > 
>> >>>> p = Point(3, 4) 
>> >>>> x, y = p 
>> > 
>> > That's very much like the classes built by namedtuple, but mutable. 
>> 
>> https://pypi.python.org/pypi/namedlist 
>> 
>> It also adds default values to the generated constructor, which may or 
>> may not be desirable. But if used exactly like collections.namedtuple, 
>> it ignores the default values. 
>> 
>> Eric.
> Since named tuple is considered as an object that is a tuple with attribute access. 
> The mutable alternative could be considered as an array with attribute access.
> Array in this context is tuple-like object that support assign operation.
> Since python have not such object there are different approaches tomutable  named tuple alternatives.

Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two.

However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). 

(Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.)

> One should note that particular property of named tuple is memory saving.
> So one can expect similar property of mutable named tuple too.

If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea.

Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/64005f38/attachment.html>

From szport at gmail.com  Thu Mar 19 10:11:56 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Thu, 19 Mar 2015 12:11:56 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
Message-ID: <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>

That all right. But I want to note that `collections.namedtuple` has
several properties that make them exclusive:

1. Fast creation;
2. Minimal memory capacity;
3. Fast sequence interface;
4. Attribute access to elements via properties.

Different namedtuple alternatives has different sets of properties that
make them more ore less suitable depending on use cases.

So if someone search alternative of collections.namedtuple that support
assignment too then it could be constructed on top of array (actually
"tuple" but with assignment support, but python seems have not such array
type).

---
*Zaur Shibzukhov*


2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:

> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>
>
>
> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith
> ???????:
>>
>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote:
>> > Sometimes we need a simple class to hold some mutable attributes,
>> > provide a nice repr, support == for testing, and support iterable
>> > unpacking, so you can write:
>> >
>> >>>> p = Point(3, 4)
>> >>>> x, y = p
>> >
>> > That's very much like the classes built by namedtuple, but mutable.
>>
>> https://pypi.python.org/pypi/namedlist
>>
>> It also adds default values to the generated constructor, which may or
>> may not be desirable. But if used exactly like collections.namedtuple,
>> it ignores the default values.
>>
>> Eric.
>>
>> Since named tuple is considered as an object that is a tuple with
> attribute access.
> The mutable alternative could be considered as an array with attribute
> access.
> Array in this context is tuple-like object that support assign operation.
> Since python have not such object there are different approaches
> tomutable  named tuple alternatives.
>
>
> Python definitely does have such an object: list. A list is effectively
> the same as a tuple but mutable; it's the paradigm MutableSequence while
> tuple is the paradigm Sequence. Under the covers they have very similar
> headers that both use the same storage (a C array of pointers to Python
> objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM
> don't distinguish between the two.
>
> However, list is resizable, and presumably a "namedlist" would not be.
> That makes things more complicated for both the interface (there's no is-a
> relationship; a type without append is not a list--and, worse, a type that
> has __setitem__ but can't handle slice replacement is not a list but that's
> very hard to detect...) and the implementation (e.g., a list reserves extra
> space at the end to avoid having to reallocate on every append).
>
> (Python _also_ has an array type, which is for homogenous simple types
> (like 32-bit int) which can store the values directly, as opposed to tuple
> and list, which store (pointers to) heterogenous normal Python objects.)
>
> One should note that particular property of named tuple is memory saving.
> So one can expect similar property of mutable named tuple too.
>
>
> If you don't need to access the items by index for whatever reason, you
> don't need a namedtuple, and using one as a misguided misoptimization is a
> bad idea.
>
> Besides the fact that a normal class with __slots__ is also small, and
> even a normal class with a dict (in newer CPython versions and PyPy) not
> that much bigger, besides the fact that you can eliminate the row overhead
> rather than just slightly reducing it by using, e.g., a 2D array, you're
> optimizing the wrong thing in the first place--if your rows have 9
> elements, reducing the row overhead is focusing on fixing 10% of your
> overhead, while reducing or eliminating the element overhead by using,
> e.g., a 2D numpy array of low-level values fixes the 90% (along with the
> 10%).
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/8091e833/attachment.html>

From abarnert at yahoo.com  Thu Mar 19 11:07:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 03:07:20 -0700
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
 <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
Message-ID: <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>

On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
> 
> That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive:
> 
> 1. Fast creation;
> 2. Minimal memory capacity;
> 3. Fast sequence interface;
> 4. Attribute access to elements via properties.
> 
> Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases.
> 
> So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type).

Again, do you not know about list?

> 
> ---
> Zaur Shibzukhov
> 
> 
> 2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>> 
>>> 
>>> 
>>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????:
>>>> 
>>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: 
>>>> > Sometimes we need a simple class to hold some mutable attributes, 
>>>> > provide a nice repr, support == for testing, and support iterable 
>>>> > unpacking, so you can write: 
>>>> > 
>>>> >>>> p = Point(3, 4) 
>>>> >>>> x, y = p 
>>>> > 
>>>> > That's very much like the classes built by namedtuple, but mutable. 
>>>> 
>>>> https://pypi.python.org/pypi/namedlist 
>>>> 
>>>> It also adds default values to the generated constructor, which may or 
>>>> may not be desirable. But if used exactly like collections.namedtuple, 
>>>> it ignores the default values. 
>>>> 
>>>> Eric.
>>> Since named tuple is considered as an object that is a tuple with attribute access. 
>>> The mutable alternative could be considered as an array with attribute access.
>>> Array in this context is tuple-like object that support assign operation.
>>> Since python have not such object there are different approaches tomutable  named tuple alternatives.
>> 
>> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two.
>> 
>> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). 
>> 
>> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.)
>> 
>>> One should note that particular property of named tuple is memory saving.
>>> So one can expect similar property of mutable named tuple too.
>> 
>> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea.
>> 
>> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%).
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/8971293a/attachment-0001.html>

From szport at gmail.com  Thu Mar 19 11:43:32 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Thu, 19 Mar 2015 13:43:32 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
 <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
 <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>
Message-ID: <CAPOE21SbDg6cpyUNg8h_4L+=58E+AkN6wLFzr+h8am4eOVpovw@mail.gmail.com>

Yes. But:

First.

>>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([])))
64 48
print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2])))
104 64


Second.

Tuple object allocates it's memory 1 time, list object allocates it's
memory 2 time. That is why

>>>%time for i in range(100000): tuple(i for i in range(15))
CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms
Wall time: 243 ms

>>>%time for i in range(100000): list(i for i in range(15))
CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms
Wall time: 321 ms

Certainly this can have or not have a value depending on use case.

---
*Zaur Shibzukhov*


2015-03-19 13:07 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:

> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>
> That all right. But I want to note that `collections.namedtuple` has
> several properties that make them exclusive:
>
> 1. Fast creation;
> 2. Minimal memory capacity;
> 3. Fast sequence interface;
> 4. Attribute access to elements via properties.
>
> Different namedtuple alternatives has different sets of properties that
> make them more ore less suitable depending on use cases.
>
> So if someone search alternative of collections.namedtuple that support
> assignment too then it could be constructed on top of array (actually
> "tuple" but with assignment support, but python seems have not such array
> type).
>
>
> Again, do you not know about list?
>
>
> ---
> *Zaur Shibzukhov*
>
>
> 2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>
>>
>>
>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith
>> ???????:
>>>
>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote:
>>> > Sometimes we need a simple class to hold some mutable attributes,
>>> > provide a nice repr, support == for testing, and support iterable
>>> > unpacking, so you can write:
>>> >
>>> >>>> p = Point(3, 4)
>>> >>>> x, y = p
>>> >
>>> > That's very much like the classes built by namedtuple, but mutable.
>>>
>>> https://pypi.python.org/pypi/namedlist
>>>
>>> It also adds default values to the generated constructor, which may or
>>> may not be desirable. But if used exactly like collections.namedtuple,
>>> it ignores the default values.
>>>
>>> Eric.
>>>
>>> Since named tuple is considered as an object that is a tuple with
>> attribute access.
>> The mutable alternative could be considered as an array with attribute
>> access.
>> Array in this context is tuple-like object that support assign operation.
>> Since python have not such object there are different approaches
>> tomutable  named tuple alternatives.
>>
>>
>> Python definitely does have such an object: list. A list is effectively
>> the same as a tuple but mutable; it's the paradigm MutableSequence while
>> tuple is the paradigm Sequence. Under the covers they have very similar
>> headers that both use the same storage (a C array of pointers to Python
>> objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM
>> don't distinguish between the two.
>>
>> However, list is resizable, and presumably a "namedlist" would not be.
>> That makes things more complicated for both the interface (there's no is-a
>> relationship; a type without append is not a list--and, worse, a type that
>> has __setitem__ but can't handle slice replacement is not a list but that's
>> very hard to detect...) and the implementation (e.g., a list reserves extra
>> space at the end to avoid having to reallocate on every append).
>>
>> (Python _also_ has an array type, which is for homogenous simple types
>> (like 32-bit int) which can store the values directly, as opposed to tuple
>> and list, which store (pointers to) heterogenous normal Python objects.)
>>
>> One should note that particular property of named tuple is memory saving.
>> So one can expect similar property of mutable named tuple too.
>>
>>
>> If you don't need to access the items by index for whatever reason, you
>> don't need a namedtuple, and using one as a misguided misoptimization is a
>> bad idea.
>>
>> Besides the fact that a normal class with __slots__ is also small, and
>> even a normal class with a dict (in newer CPython versions and PyPy) not
>> that much bigger, besides the fact that you can eliminate the row overhead
>> rather than just slightly reducing it by using, e.g., a 2D array, you're
>> optimizing the wrong thing in the first place--if your rows have 9
>> elements, reducing the row overhead is focusing on fixing 10% of your
>> overhead, while reducing or eliminating the element overhead by using,
>> e.g., a 2D numpy array of low-level values fixes the 90% (along with the
>> 10%).
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/a3169834/attachment.html>

From abarnert at yahoo.com  Thu Mar 19 12:22:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 04:22:14 -0700
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CAPOE21SbDg6cpyUNg8h_4L+=58E+AkN6wLFzr+h8am4eOVpovw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
 <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
 <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>
 <CAPOE21SbDg6cpyUNg8h_4L+=58E+AkN6wLFzr+h8am4eOVpovw@mail.gmail.com>
Message-ID: <E1AF1097-C3D9-4098-8E52-F131B8B4D06B@yahoo.com>

On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
> 
> Yes. But:
> 
> First.
> 
> >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([])))
> 64 48
> print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2])))
> 104 64
> 
> 
> Second.
> 
> Tuple object allocates it's memory 1 time, list object allocates it's memory 2 time.

As I explained, list leaves room for expansion, so it doesn't have to keep reallocating with every append. A 2-element list actually has room for... off the top of my head, I think 8 elements; then it multiplies the capacity every time it runs out on an append.

In a C extension, you can create a list with whatever specific capacity you want, but that isn't exposed to Python (presumably because such a micro-optimization is rarely useful, and would complicate the API, as well as being an attractive nuisance to people who insist on optimizing the wrong things). So, if you really needed this optimization, you could implement your namedlist in C.

But what use case are you imagining where the extra 40 bytes per row matter, but the 64 bytes per row don't, and neither do the 56 or so bytes per element per row? I'm having a hard time imagining any case where using tuples would be necessary where it wouldn't be woefully insufficient.

> That is why
> 
> >>>%time for i in range(100000): tuple(i for i in range(15))
> CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms
> Wall time: 243 ms
> 
> >>>%time for i in range(100000): list(i for i in range(15))
> CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms
> Wall time: 321 ms
> 
> Certainly this can have or not have a value depending on use case. 

If you're going to show benchmarks, you really should use %timeit rather than %time, and also understand that "i for i in range(15)" is just going to give you a slower but equivalent iterable to just using "range(15)" in the first place.

But more importantly, what use case are you considering where this extra 0.8us for construction of each row object will matter, but the difference between a namedtuple (or, presumably, namedlist) vs. a tuple won't? 

And, even if you did, you're focusing on something that accounts for at worst 25% of the construction time, and 0% of the time for all the actual work you do with the object after construction.

Again, unless you have many millions of these, neither the memory not the construction time is going to matter--and if you do, a new type that's more like tuple isn't going to be anywhere near sufficient to make a noticeable difference, because you're optimizing the wrong part of both the memory and the time.

> 
> ---
> Zaur Shibzukhov
> 
> 
> 2015-03-19 13:07 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>> 
>>> That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive:
>>> 
>>> 1. Fast creation;
>>> 2. Minimal memory capacity;
>>> 3. Fast sequence interface;
>>> 4. Attribute access to elements via properties.
>>> 
>>> Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases.
>>> 
>>> So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type).
>> 
>> Again, do you not know about list?
>> 
>>> 
>>> ---
>>> Zaur Shibzukhov
>>> 
>>> 
>>> 2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>>>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????:
>>>>>> 
>>>>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: 
>>>>>> > Sometimes we need a simple class to hold some mutable attributes, 
>>>>>> > provide a nice repr, support == for testing, and support iterable 
>>>>>> > unpacking, so you can write: 
>>>>>> > 
>>>>>> >>>> p = Point(3, 4) 
>>>>>> >>>> x, y = p 
>>>>>> > 
>>>>>> > That's very much like the classes built by namedtuple, but mutable. 
>>>>>> 
>>>>>> https://pypi.python.org/pypi/namedlist 
>>>>>> 
>>>>>> It also adds default values to the generated constructor, which may or 
>>>>>> may not be desirable. But if used exactly like collections.namedtuple, 
>>>>>> it ignores the default values. 
>>>>>> 
>>>>>> Eric.
>>>>> Since named tuple is considered as an object that is a tuple with attribute access. 
>>>>> The mutable alternative could be considered as an array with attribute access.
>>>>> Array in this context is tuple-like object that support assign operation.
>>>>> Since python have not such object there are different approaches tomutable  named tuple alternatives.
>>>> 
>>>> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two.
>>>> 
>>>> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). 
>>>> 
>>>> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.)
>>>> 
>>>>> One should note that particular property of named tuple is memory saving.
>>>>> So one can expect similar property of mutable named tuple too.
>>>> 
>>>> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea.
>>>> 
>>>> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%).
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/3f1e1a52/attachment.html>

From nicholas.chammas at gmail.com  Thu Mar 19 15:32:02 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Thu, 19 Mar 2015 10:32:02 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
Message-ID: <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>

OK, I hereby abandon this proposal. :)

To summarize the conclusion I believe this thread has arrived at:

   - An auto-styler needs to have wide community adoption before it can be
   considered for inclusion into the standard library.
   - Even then, adding stuff to the standard library at this point in
   Python?s life is generally a bad thing. Putting an auto-styler in the
   standard library would be a big burden on its maintainers and slow down
   improvements that could be made to it.
   -

   Ignoring that difficulty, there is also the problem of what ?spec? to
   use for any such auto-styler. PEP 8 is an obvious choice, but it was
   written as a guide for humans, not a spec for programmatic implementation.
   Furthermore, there are doubts about PEP 8?s utility as the basis for a
   universal auto-styler, as it was intended for code specifically in the
   Python standard library.
    -

   Let a thousand auto-stylers bloom outside the standard library.

Some final responses to various people?s comments:

On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

But that implies that it's worth someone making a spec out of PEP 8. That
> implies going through PEP 8, getting a sense of what parts are
> (unambiguously and uncontroversially) mechanizable, and reorganizing and
> minimally rewriting it to get that sense across. Having numbered rules, and
> named anchors for each one in the document (or at least to narrower
> sections, so you can link to something more specific than "Programming
> Recommendations") would also be helpful.

This is an excellent point and one I did not consider.

On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:

PEP 8 is Guido's view on how new CPython stdlib code should be written,
> with input from other core developers.  "This document gives coding
> conventions for the Python code comprising the standard library in the main
> Python distribution."

Good point. For my all my talk about PEP 8, I missed this key point. Of
course, it should be noted that PEP 8 is being used by all kinds of people
for stuff outside of stdlib, but then again as you pointed out people are
also putting together their own style guides.

On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:

(Note that autoPEP8 only changes whitespace, which is a small part of PEP 8
> recommendations, and the safest aspect of code to change.)
>
Minor point: autopep8 can do more than just whitespace changes with the
--aggressive flag
<https://pypi.python.org/pypi/autopep8/#more-advanced-usage>.

On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy <tjreedy at udel.edu> wrote:

We should be careful about adding anything.  When we do, we should add
> libraries, not applications.  We should especially not officially endorse
> one of many opinion-based competitors.

I have come around to this view.

On Wed, Mar 18, 2015 at 7:30 PM, Eli Bendersky <eliben at gmail.com> wrote:

Basically the idea is "forget about formatting your code, just run it
> through the tool", which is a similar philosophy as gofmt (and
> clang-format) and has been hugely successful for those tools/languages.
>
Sounds good. I?m looking forward to seeing it released. As I said in an
earlier email, I think any part of the development process we automate away
is a good thing (stdlib inclusion or not notwithstanding).

Nick
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/163e78de/attachment-0001.html>

From slavek at redhat.com  Thu Mar 19 15:47:44 2015
From: slavek at redhat.com (Bohuslav Kabrda)
Date: Thu, 19 Mar 2015 10:47:44 -0400 (EDT)
Subject: [Python-ideas] Migration of /usr/bin/python to python3
In-Reply-To: <20150314115719.23ea9d75@anarchist.wooz.org>
References: <20150311194325.GA15390@python.ca>
 <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io>
 <CADiSq7dej2Ax6OcQ9tzwXXiu3tkuhG9Pcm-ksdEvbK1zY2sNjg@mail.gmail.com>
 <20150314115719.23ea9d75@anarchist.wooz.org>
Message-ID: <1073335302.7431680.1426776464310.JavaMail.zimbra@redhat.com>

----- Original Message -----
> On Mar 13, 2015, at 09:57 PM, Nick Coghlan wrote:
> 
> >Right, this most recently came up in the context of Fedora's plans to
> >transition to only having Python 3 in the base install
> >(https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the
> >moment the expected consequence is that there will come a Fedora
> >version (tentatively Fedora 23) where, out of the box,
> >"/usr/bin/python" and "/usr/bin/env python" will just stop working.
> >
> >Packages that are part of the distro aren't a problem - those are all
> >being dealt with as the transition progresses, and any which
> >specifically need Python 2 will depend on it accordingly.
> >
> >The problem arises with custom user packages and scripts, and third
> >party packages and scripts.
> 
> Sure, agreed so far.  But you're not going to be removing Python 2 from the
> Fedora archive right?  So any user with third party packages that require
> Python 2 could just install it and their stuff would continue to work.

Correct.

> At least that's the plan on Debian/Ubuntu.  I'd like to be at the point[*]
> where the default installs would not contain Python 2, but a simple `apt-get`
> would install it from the archives for anybody who needs it.  That seems like
> a safer option than changing the symlink.

Yes, this is precisely where we're heading with this; we won't change the symlink unless PEP 394 changes. We will also prefer Python 3 for *all* applications (assuming upstream is compatible), not only those that are on live media.
I think we'll have Python 2 in Fedora repositories as long as upstream will support it (and maybe even a bit longer).

> Cheers,
> -Barry
> 
> [*] I'm aiming for Debian Stretch and Ubuntu 16.04 Xylophonic X-ray fish LTS.

-- 
Regards,
Slavek Kabrda

From Steve.Dower at microsoft.com  Thu Mar 19 16:52:41 2015
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Thu, 19 Mar 2015 15:52:41 +0000
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPJVwBkzw++=LAugaQBz6A-qFzr_XAhieMFR8vb+89oVXDWDPg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAPkN8x+iw3f5-MR8wrWPF+PKAxeWgzBh+F8ikf6A9o8G4ooVjw@mail.gmail.com>,
 <CAPJVwBkzw++=LAugaQBz6A-qFzr_XAhieMFR8vb+89oVXDWDPg@mail.gmail.com>
Message-ID: <BY1PR03MB146686B7545613972B67CB9FF5010@BY1PR03MB1466.namprd03.prod.outlook.com>

And unfortunately, a lot of that work may need to be redone for 3.5 and VC14... Sorry about that, but once the other compilers have caught up with the new CRT interface then it will (should) all be better in the long run.

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Nathaniel Smith<mailto:njs at pobox.com>
Sent: ?3/?18/?2015 10:31
To: anatoly techtonik<mailto:techtonik at gmail.com>
Cc: python-ideas at python.org<mailto:python-ideas at python.org>
Subject: Re: [Python-ideas] Rewriting the build system (part 2)


On Mar 18, 2015 8:14 AM, "anatoly techtonik" <techtonik at gmail.com<mailto:techtonik at gmail.com>> wrote:
>
> On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker <chris.barker at noaa.gov<mailto:chris.barker at noaa.gov>> wrote:
> > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik <techtonik at gmail.com<mailto:techtonik at gmail.com>>
> > wrote:
> >>
> >> "Autotools has reached the Chandrasekhar mass limit of software and
> >> become a black hole of suck that consumes endless hours of bright
> >> peoples? time without any good result. It is well past time for it to
> >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877
> >>
> >> I couldn't sum up this better. It looks even worse from Windows machine.
> >
> > Is it even used for building Python on Windows (for the "normal" python.org<http://python.org>
> > builds)?
>
> There was some success with MinGW toolchain, but I am not tracking if it was
> integrated into official source. The FUD was that compiled extensions are not
> compatible between MSVC builds and MinGW builds, but I could not confirm
> that. If I understand correctly, the incompatibility is only for C++ programs,
> because of strange SEH strategy and proprietary name mangling.

This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/2f376a65/attachment.html>

From techtonik at gmail.com  Thu Mar 19 11:23:15 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 19 Mar 2015 13:23:15 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CALxg4FXa+CoNi4pZPrcwVbry6J6Zzg6j0V+FF+r8baUXPNHmqw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <CAPkN8x+VxAzS5D2+XCX5+pjniKAJwth6mFxJEWnqMk53s+jqYw@mail.gmail.com>
 <CALxg4FXa+CoNi4pZPrcwVbry6J6Zzg6j0V+FF+r8baUXPNHmqw@mail.gmail.com>
Message-ID: <CAPkN8x+A1-Jgr1Wg=swqKDTbey3UgoH1fd3kvmAk2RTnszsfpQ@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:36 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
> On Wed, Mar 18, 2015 at 3:05 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
>>> Sometimes we need a simple class to hold some mutable attributes,
>>> provide a nice repr, support == for testing, and support iterable
>>> unpacking
>>
>> +1, but I think that the core problem with such proposals is that they
>> lack the use cases. The only reason for me to have such class is to
>> work with tabular data.
>>
>> For example, for querying the capability of the system, I need to build
>> an inmemory table of features, and then set parameters for each
>> feature one by one. sqlite/SQL is an overkill for that, and dicts are
>> just not enough to do readable lookups and updates to specific
>> cells, so I'd more appreciate a full table class than just its "named
>> row" model.
>
> Thanks for your input and example, Anatoly. What would the full table
> class offer that could not be easily done with a simple list of items
> produced with a plainclass class?

Lookup operations. Attaching new columns to existing table or providing
a supplementary table with additional columns for existing ones. If
Python had first class tables, things right now could be very different.

We could have a better structured data handling.

Take logging for example. It is not extensible. You have message and
level. Then it got component. Then timestamp. But you can't add anything
yourself to that message - error code, binary dump, selfie or some read/
unread flags. If logging used a table, there could be an ability to add your
own data to its events. And this would be interoperable between
programs.

>> Practical example that I came up with:
>> https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22
>
> Oh, I see you have an update method, so that's the key value add of
> the Table class, right?

Right. The update method says "change the value of cell with column
name==name, in row where column name idname==idvalue".

It is basically lookup method with cell modification.

So it adds a second dimension to 1D "mutable namedtuple".

> I see the Table and plainclass as complementary ideas. You could use
> plainclass to implement Table more easily. The Table.__iter__ could
> return plainclass class instances instead of OrderedDict.

That's one of the uses. But I am concerned that it is the only example
where this "mutable namedtuple" is useful.

I don't like the dynamic class construction as with namedtuple. I believe
you can not serialize it reliably, and there are problems with static
analysis tools to deal with it (like locate the definition).
-- 
anatoly t.

From techtonik at gmail.com  Thu Mar 19 11:41:01 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 19 Mar 2015 13:41:01 +0300
Subject: [Python-ideas] .from and .to instead of .encode and .decode
In-Reply-To: <CALxg4FXgOEJPwo=WLU1XzqVbEdixJDi_dgb-3Y7i2O+-f6LF_g@mail.gmail.com>
References: <CAPkN8xKk5MW2TCDL7jqRgfzGdK+Ri_cBhqZHHe=wih1EChYdyQ@mail.gmail.com>
 <CAPTjJmqs=E1N7TNOerjnDhuB750VQUZyRCFGagOAOLgosKXiHA@mail.gmail.com>
 <CALxg4FUcds6AXykYhi_dR=gzjJKVc1ezNyRfmMCEqdbddcPyhg@mail.gmail.com>
 <CAFYqXL-fYHO+ZWmitbCA_T10nwqrM6-M68_633ooJvn7AntUHA@mail.gmail.com>
 <CAPkN8x+mjfiPBXdBC1LeHrtKA+Xghfa6DP0N+J45HqSdjU3u-w@mail.gmail.com>
 <CALxg4FWBQ2jDsHcwevAxb0EpiF8h5J1uma2MM93Ofsns1uUMng@mail.gmail.com>
 <CAPkN8xKS0=vcyGRcSGEC2RdUA0Ob8wLqSsskK04RLv-d3FN7qA@mail.gmail.com>
 <CALxg4FXgOEJPwo=WLU1XzqVbEdixJDi_dgb-3Y7i2O+-f6LF_g@mail.gmail.com>
Message-ID: <CAPkN8xJug4K-YWuxP1cJbSseDmqyXZ-w3b75PT7_5zGLf5uLoQ@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:24 PM, Luciano Ramalho <luciano at ramalho.org> wrote:
>>
>>> A more objective problem is: "from" is a keyword, so it can't be a method name.
>>
>> I don't see why "from" can not be used as a method name in Python. It can not
>> be used as function name, because it will override the keyword, but method
>> names are always namespaced with self. or class prefixes.
>
> You can't write it's definition, "def from..." and you can't even
> write "x.from" without a SyntaxError (instead of NameError or
> AttributeError). Try it in the Python console.
>
> The parser would need to be changed.

And that won't break anything.
-- 
anatoly t.

From techtonik at gmail.com  Thu Mar 19 12:12:36 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 19 Mar 2015 14:12:36 +0300
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <20150318152150.GN10871@tonks>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
 <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
 <20150318152150.GN10871@tonks>
Message-ID: <CAPkN8xK+82J_TY6tnCFOmm5VuAMsA=8uxY-v9OfGDRaYU54bhg@mail.gmail.com>

On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <me at the-compiler.org> wrote:
> * anatoly techtonik <techtonik at gmail.com> [2015-03-18 10:21:46 +0300]:
>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>> >>
>> >> But to miss-paraphrase Winston Churchill [1]:
>> >>
>> >> "Autotools is the worst build system except for all the others."
>> >>
>> >> Is there really something better out there?
>> >>
>> >
>> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>>
>> Is there anything Javascriptey? Python needs to move to web. I heard
>> something about Grunt, which was replaced by Gulp, but no idea if they
>> really mature enough to include "system features detection" and "user
>> configured builds" (or work with C/C++ compilers at all).
>
> What does "Python needs to move to web" to do with it's build system
> being "Javascriptey"?

Because things like Twisted are de-facto standard in JavaScript world
(Node.js), and Python will benefit more if people look for how things are
implemented in non-C worlds.
-- 
anatoly t.

From szport at gmail.com  Thu Mar 19 18:57:38 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Thu, 19 Mar 2015 20:57:38 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <E1AF1097-C3D9-4098-8E52-F131B8B4D06B@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
 <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
 <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>
 <CAPOE21SbDg6cpyUNg8h_4L+=58E+AkN6wLFzr+h8am4eOVpovw@mail.gmail.com>
 <E1AF1097-C3D9-4098-8E52-F131B8B4D06B@yahoo.com>
Message-ID: <CAPOE21RiTSGkdcxD5WyCzgQu5T5SsJuONcgjWi_jgk8S59hiew@mail.gmail.com>

2015-03-19 14:22 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:

> On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>
> Yes. But:
>
> First.
>
> >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([])))
> 64 48
> print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2])))
> 104 64
>
>
> Second.
>
> Tuple object allocates it's memory 1 time, list object allocates it's
> memory 2 time.
>
>
> As I explained, list leaves room for expansion, so it doesn't have to keep
> reallocating with every append. A 2-element list actually has room for...
> off the top of my head, I think 8 elements; then it multiplies the capacity
> every time it runs out on an append.
>

>>>for i in [4, 12, 28, 68]: print(sys.getsizeof(list(range(i))), end=' ')
120 216 360 720

> In a C extension, you can create a list with whatever specific capacity
you want, but that isn't exposed to Python
> (presumably because such a micro-optimization is rarely useful, and would
complicate the API, as well as being an
> attractive nuisance to people who insist on optimizing the wrong things).
So, if you really needed this optimization, you could
> implement your namedlist in C.

Yes you right. Actual optimization one could archive with the help of
Cython if necessary.

But what use case are you imagining where the extra 40 bytes per row
matter, but the 64 bytes per row don't, and neither do the 56 or so bytes
per element per row? I'm having a hard time imagining any case where using
tuples would be necessary where it wouldn't be woefully insufficient.


That is why

>>>%time for i in range(100000): tuple(i for i in range(15))
CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms
Wall time: 243 ms

>>>%time for i in range(100000): list(i for i in range(15))
CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms
Wall time: 321 ms

Certainly this can have or not have a value depending on use case.
>>>import namedlist

> If you're going to show benchmarks, you really should use %timeit rather
than %time, and also understand that "i for i in
> range(15)" is just going to give you a slower but equivalent iterable to
just using "range(15)" in the first place.

> But more importantly, what use case are you considering where this extra
0.8us for construction of each row object will
> matter, but the difference between a namedtuple (or, presumably,
namedlist) vs. a tuple won't?

>>>class Point:
    __slots__ = 'x', 'y', 'z'
    def __init__(self, x, y, z):
        self.x, self.y, self.z = x, y, z

>>>asizeof.asizeof(Point(1,2,3))
   232
>>>import collections
>>>Point2 = collections.namedtuple('Point', 'x y z')
>>>Point3 = namedlist.namedtuple('Point', 'x y z')
>>>Point4 = namedlist.namedlist('Point', 'x y z')
>>>asizeof.asizeof(Point2(1,2,3))
   120
>>>asizeof.asizeof(Point3(1,2,3))
   120
>>>asizeof.asizeof(Point4(1,2,3))
   1704

> And, even if you did, you're focusing on something that accounts for at
worst 25% of the construction time, and 0% of the
> time for all the actual work you do with the object after construction.

> Again, unless you have many millions of these, neither the memory not the
construction time is going to matter--and if you do,
> a new type that's more like tuple isn't going to be anywhere near
sufficient to make a noticeable difference, because you're
> optimizing the wrong part of both the memory and the time.

Possibly basic gain in the savings of memory, especially in the context of
long running processes, limitations on the memory on hostings, long loop +
memory fragmentation in python heap space, ...


---
*Zaur Shibzukhov*


2015-03-19 13:07 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:

> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>
> That all right. But I want to note that `collections.namedtuple` has
> several properties that make them exclusive:
>
> 1. Fast creation;
> 2. Minimal memory capacity;
> 3. Fast sequence interface;
> 4. Attribute access to elements via properties.
>
> Different namedtuple alternatives has different sets of properties that
> make them more ore less suitable depending on use cases.
>
> So if someone search alternative of collections.namedtuple that support
> assignment too then it could be constructed on top of array (actually
> "tuple" but with assignment support, but python seems have not such array
> type).
>
>
> Again, do you not know about list?
>
>
> ---
> *Zaur Shibzukhov*
>
>
> 2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>
>>
>>
>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith
>> ???????:
>>>
>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote:
>>> > Sometimes we need a simple class to hold some mutable attributes,
>>> > provide a nice repr, support == for testing, and support iterable
>>> > unpacking, so you can write:
>>> >
>>> >>>> p = Point(3, 4)
>>> >>>> x, y = p
>>> >
>>> > That's very much like the classes built by namedtuple, but mutable.
>>>
>>> https://pypi.python.org/pypi/namedlist
>>>
>>> It also adds default values to the generated constructor, which may or
>>> may not be desirable. But if used exactly like collections.namedtuple,
>>> it ignores the default values.
>>>
>>> Eric.
>>>
>>> Since named tuple is considered as an object that is a tuple with
>> attribute access.
>> The mutable alternative could be considered as an array with attribute
>> access.
>> Array in this context is tuple-like object that support assign operation.
>> Since python have not such object there are different approaches
>> tomutable  named tuple alternatives.
>>
>>
>> Python definitely does have such an object: list. A list is effectively
>> the same as a tuple but mutable; it's the paradigm MutableSequence while
>> tuple is the paradigm Sequence. Under the covers they have very similar
>> headers that both use the same storage (a C array of pointers to Python
>> objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM
>> don't distinguish between the two.
>>
>> However, list is resizable, and presumably a "namedlist" would not be.
>> That makes things more complicated for both the interface (there's no is-a
>> relationship; a type without append is not a list--and, worse, a type that
>> has __setitem__ but can't handle slice replacement is not a list but that's
>> very hard to detect...) and the implementation (e.g., a list reserves extra
>> space at the end to avoid having to reallocate on every append).
>>
>> (Python _also_ has an array type, which is for homogenous simple types
>> (like 32-bit int) which can store the values directly, as opposed to tuple
>> and list, which store (pointers to) heterogenous normal Python objects.)
>>
>> One should note that particular property of named tuple is memory saving.
>> So one can expect similar property of mutable named tuple too.
>>
>>
>> If you don't need to access the items by index for whatever reason, you
>> don't need a namedtuple, and using one as a misguided misoptimization is a
>> bad idea.
>>
>> Besides the fact that a normal class with __slots__ is also small, and
>> even a normal class with a dict (in newer CPython versions and PyPy) not
>> that much bigger, besides the fact that you can eliminate the row overhead
>> rather than just slightly reducing it by using, e.g., a 2D array, you're
>> optimizing the wrong thing in the first place--if your rows have 9
>> elements, reducing the row overhead is focusing on fixing 10% of your
>> overhead, while reducing or eliminating the element overhead by using,
>> e.g., a 2D numpy array of low-level values fixes the 90% (along with the
>> 10%).
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/f540a38b/attachment-0001.html>

From abarnert at yahoo.com  Thu Mar 19 19:42:55 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 11:42:55 -0700
Subject: [Python-ideas] A mutable alternative to namedtuple
In-Reply-To: <CAPOE21RiTSGkdcxD5WyCzgQu5T5SsJuONcgjWi_jgk8S59hiew@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <55086248.2010009@trueblade.com>
 <a2db66f1-9804-4ede-b622-e971c5156c45@googlegroups.com>
 <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com>
 <CAPOE21R1ktZJKx8OfAUBo1nQ6QhMf2CjY8mz+eYVuC5X+aV_pA@mail.gmail.com>
 <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com>
 <CAPOE21SbDg6cpyUNg8h_4L+=58E+AkN6wLFzr+h8am4eOVpovw@mail.gmail.com>
 <E1AF1097-C3D9-4098-8E52-F131B8B4D06B@yahoo.com>
 <CAPOE21RiTSGkdcxD5WyCzgQu5T5SsJuONcgjWi_jgk8S59hiew@mail.gmail.com>
Message-ID: <FC50A292-93F4-417C-84E7-1A5EDDA005C8@yahoo.com>

> On Mar 19, 2015, at 10:57 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
> 
> 
> 
> 2015-03-19 14:22 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>> On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>> 
>>> Yes. But:
>>> 
>>> First.
>>> 
>>> >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([])))
>>> 64 48
>>> print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2])))
>>> 104 64
>>> 
>>> 
>>> Second.
>>> 
>>> Tuple object allocates it's memory 1 time, list object allocates it's memory 2 time.
>> 
>> As I explained, list leaves room for expansion, so it doesn't have to keep reallocating with every append. A 2-element list actually has room for... off the top of my head, I think 8 elements; then it multiplies the capacity every time it runs out on an append.
> 
> >>>for i in [4, 12, 28, 68]: print(sys.getsizeof(list(range(i))), end=' ')
> 120 216 360 720

And what's your point? That's exactly what you should expect if lists keep extra slack space at the end to avoid reallocation on expansion.

> > In a C extension, you can create a list with whatever specific capacity you want, but that isn't exposed to Python
> > (presumably because such a micro-optimization is rarely useful, and would complicate the API, as well as being an 
> > attractive nuisance to people who insist on optimizing the wrong things). So, if you really needed this optimization, you could 
> > implement your namedlist in C.
> 
> Yes you right. Actual optimization one could archive with the help of Cython if necessary.

Not if you're trying to put this in the stdlib; you have to write the C extension. But if the person who thinks this is necessary (you) prefers to use Cython instead of C, that seems like another argument against putting it in the stdlib...

> But what use case are you imagining where the extra 40 bytes per row matter, but the 64 bytes per row don't, and neither do the 56 or so bytes per element per row? I'm having a hard time imagining any case where using tuples would be necessary where it wouldn't be woefully insufficient.

I've now asked this twice, and you've avoided answering it both times to present further irrelevancies. This is the key question. If you don't have any use case for which this matters, then it doesn't matter.

>> That is why
>> 
>> >>>%time for i in range(100000): tuple(i for i in range(15))
>> CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms
>> Wall time: 243 ms
>> 
>> >>>%time for i in range(100000): list(i for i in range(15))
>> CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms
>> Wall time: 321 ms
>> 
>> Certainly this can have or not have a value depending on use case. >>>import namedlist
> 
> > If you're going to show benchmarks, you really should use %timeit rather than %time, and also understand that "i for i in 
> > range(15)" is just going to give you a slower but equivalent iterable to just using "range(15)" in the first place.
> 
> > But more importantly, what use case are you considering where this extra 0.8us for construction of each row object will 
> > matter, but the difference between a namedtuple (or, presumably, namedlist) vs. a tuple won't? 
> 
> >>>class Point:
>     __slots__ = 'x', 'y', 'z'
>     def __init__(self, x, y, z):
>         self.x, self.y, self.z = x, y, z
> 
> >>>asizeof.asizeof(Point(1,2,3))
>    232
> >>>import collections
> >>>Point2 = collections.namedtuple('Point', 'x y z')
> >>>Point3 = namedlist.namedtuple('Point', 'x y z')
> >>>Point4 = namedlist.namedlist('Point', 'x y z')
> >>>asizeof.asizeof(Point2(1,2,3))
>    120
> >>>asizeof.asizeof(Point3(1,2,3))
>    120
> >>>asizeof.asizeof(Point4(1,2,3))
>    1704

Despite the name, the third-party module "namedlist" does not appear to make classes that inherit from list, or use a list for storage. You just get a normal class (using an OrderedDict __dict__ by default, or optionally __slots__) that provides a __getitem__ method implemented in terms of __getattr__. So I don't see how this is even vaguely relevant here.

And, even if that module were relevant, how would it have anything to do with answering "Where is the use case where this construction time matters?" You don't even _mention_ construction time here.

> > And, even if you did, you're focusing on something that accounts for at worst 25% of the construction time, and 0% of the 
> > time for all the actual work you do with the object after construction.
> 
> > Again, unless you have many millions of these, neither the memory not the construction time is going to matter--and if you do, 
> > a new type that's more like tuple isn't going to be anywhere near sufficient to make a noticeable difference, because you're 
> > optimizing the wrong part of both the memory and the time.
> 
> Possibly basic gain in the savings of memory, especially in the context of long running processes, limitations on the memory on hostings, long loop + memory fragmentation in python heap space, ...

No. This is basic computer science. Optimizing parts of your code that aren't actually relevant because they contribute only a tiny percentage to the waste doesn't actually improve anything. 


>> 
>> ---
>> Zaur Shibzukhov
>> 
>> 
>> 2015-03-19 13:07 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>>> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>>> 
>>>> That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive:
>>>> 
>>>> 1. Fast creation;
>>>> 2. Minimal memory capacity;
>>>> 3. Fast sequence interface;
>>>> 4. Attribute access to elements via properties.
>>>> 
>>>> Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases.
>>>> 
>>>> So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type).
>>> 
>>> Again, do you not know about list?
>>> 
>>>> 
>>>> ---
>>>> Zaur Shibzukhov
>>>> 
>>>> 
>>>> 2015-03-19 11:37 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>>>>>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov <szport at gmail.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????:
>>>>>>> 
>>>>>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: 
>>>>>>> > Sometimes we need a simple class to hold some mutable attributes, 
>>>>>>> > provide a nice repr, support == for testing, and support iterable 
>>>>>>> > unpacking, so you can write: 
>>>>>>> > 
>>>>>>> >>>> p = Point(3, 4) 
>>>>>>> >>>> x, y = p 
>>>>>>> > 
>>>>>>> > That's very much like the classes built by namedtuple, but mutable. 
>>>>>>> 
>>>>>>> https://pypi.python.org/pypi/namedlist 
>>>>>>> 
>>>>>>> It also adds default values to the generated constructor, which may or 
>>>>>>> may not be desirable. But if used exactly like collections.namedtuple, 
>>>>>>> it ignores the default values. 
>>>>>>> 
>>>>>>> Eric.
>>>>>> Since named tuple is considered as an object that is a tuple with attribute access. 
>>>>>> The mutable alternative could be considered as an array with attribute access.
>>>>>> Array in this context is tuple-like object that support assign operation.
>>>>>> Since python have not such object there are different approaches tomutable  named tuple alternatives.
>>>>> 
>>>>> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two.
>>>>> 
>>>>> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). 
>>>>> 
>>>>> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.)
>>>>> 
>>>>>> One should note that particular property of named tuple is memory saving.
>>>>>> So one can expect similar property of mutable named tuple too.
>>>>> 
>>>>> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea.
>>>>> 
>>>>> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%).
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/3c64927e/attachment-0001.html>

From abarnert at yahoo.com  Thu Mar 19 19:48:02 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 11:48:02 -0700
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <CAPkN8xK+82J_TY6tnCFOmm5VuAMsA=8uxY-v9OfGDRaYU54bhg@mail.gmail.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
 <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
 <20150318152150.GN10871@tonks>
 <CAPkN8xK+82J_TY6tnCFOmm5VuAMsA=8uxY-v9OfGDRaYU54bhg@mail.gmail.com>
Message-ID: <B00E6D5B-A412-4B64-A97D-A7E92A6A4E1C@yahoo.com>

On Mar 19, 2015, at 4:12 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
>> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <me at the-compiler.org> wrote:
>> * anatoly techtonik <techtonik at gmail.com> [2015-03-18 10:21:46 +0300]:
>>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>>>> 
>>>>> But to miss-paraphrase Winston Churchill [1]:
>>>>> 
>>>>> "Autotools is the worst build system except for all the others."
>>>>> 
>>>>> Is there really something better out there?
>>>> 
>>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
>>> 
>>> Is there anything Javascriptey? Python needs to move to web. I heard
>>> something about Grunt, which was replaced by Gulp, but no idea if they
>>> really mature enough to include "system features detection" and "user
>>> configured builds" (or work with C/C++ compilers at all).
>> 
>> What does "Python needs to move to web" to do with it's build system
>> being "Javascriptey"?
> 
> Because things like Twisted are de-facto standard in JavaScript world
> (Node.js), and Python will benefit more if people look for how things are
> implemented in non-C worlds.

Is this a sarcastic comment?

Or are you seriously suggesting that because people use things like the stdlib's asyncio, gevent, Tornado, etc. in Python, which have different paradigms than Twisted, the one Node copied, we need to force some other part of Python to be more like Node in some other way, because "move to the web" means "be more like the latest (but nowhere near most popular) framework for building web services"?

Also, correct me if I'm wrong, but isn't Node implemented in C, with a build system in C and Python? So if we really want to be more like Node for some reason, we should make the build system Pythony instead of Javascripty?

From rymg19 at gmail.com  Thu Mar 19 19:55:05 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 19 Mar 2015 13:55:05 -0500
Subject: [Python-ideas] Rewriting the build system (part 2)
In-Reply-To: <B00E6D5B-A412-4B64-A97D-A7E92A6A4E1C@yahoo.com>
References: <CAO41-mM_DGGDX5LhALvJHmsX2YL2t4vL4BSOTdsGcmMPfVUZAA@mail.gmail.com>
 <CACfEFw9PN2xrPV3SH50rzTPV=nZrOO10fWo+L2gpjiMCAqKeCQ@mail.gmail.com>
 <CAO41-mN2Dyv4CMjEk3gx8OPjn=GT9OSf9FCCtTsjBNVmpvC0rA@mail.gmail.com>
 <CACfEFw-KeAU1q_fDdWPPp7SkVQxoTEduNgnJcF1zgOoS6wfd2Q@mail.gmail.com>
 <CAPkN8xJULJVz4s4iyCCkj21eAGvkOUuTmqsosdxJUSz3s8zvYw@mail.gmail.com>
 <CALGmxEKg3kDHqcPsGbEc6aVs1_RmY9gHY37wu09puEoi2=UP=w@mail.gmail.com>
 <CAO41-mNygqyEoLURiRugiHMVw+3Bu0vWWuT=GBcaSSgob-aa6A@mail.gmail.com>
 <CAPkN8xJS=Nh=_Y-bbhkya62O8tCXhexC_vZuU2gWbSaxhON3rg@mail.gmail.com>
 <20150318152150.GN10871@tonks>
 <CAPkN8xK+82J_TY6tnCFOmm5VuAMsA=8uxY-v9OfGDRaYU54bhg@mail.gmail.com>
 <B00E6D5B-A412-4B64-A97D-A7E92A6A4E1C@yahoo.com>
Message-ID: <CAO41-mPB8j7kOWx7EoMenB1eKh-2H+X2YqwkrMiyZCPRkgLRaA@mail.gmail.com>

On Thu, Mar 19, 2015 at 1:48 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> On Mar 19, 2015, at 4:12 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> >
> >> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin <me at the-compiler.org>
> wrote:
> >> * anatoly techtonik <techtonik at gmail.com> [2015-03-18 10:21:46 +0300]:
> >>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez <rymg19 at gmail.com>
> wrote:
> >>>>>
> >>>>> But to miss-paraphrase Winston Churchill [1]:
> >>>>>
> >>>>> "Autotools is the worst build system except for all the others."
> >>>>>
> >>>>> Is there really something better out there?
> >>>>
> >>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ...
> >>>
> >>> Is there anything Javascriptey? Python needs to move to web. I heard
> >>> something about Grunt, which was replaced by Gulp, but no idea if they
> >>> really mature enough to include "system features detection" and "user
> >>> configured builds" (or work with C/C++ compilers at all).
> >>
> >> What does "Python needs to move to web" to do with it's build system
> >> being "Javascriptey"?
> >
> > Because things like Twisted are de-facto standard in JavaScript world
> > (Node.js), and Python will benefit more if people look for how things are
> > implemented in non-C worlds.
>
> Is this a sarcastic comment?
>
> Or are you seriously suggesting that because people use things like the
> stdlib's asyncio, gevent, Tornado, etc. in Python, which have different
> paradigms than Twisted, the one Node copied, we need to force some other
> part of Python to be more like Node in some other way, because "move to the
> web" means "be more like the latest (but nowhere near most popular)
> framework for building web services"?
>
> Also, correct me if I'm wrong, but isn't Node implemented in C, with a
> build system in C and Python? So if we really want to be more like Node for
> some reason, we should make the build system Pythony instead of Javascripty?
>

You're not wrong. Node uses Gyp, although, if Chromium switches to GN
(written in C++), Node might switch, too.


> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/2d260925/attachment.html>

From tjreedy at udel.edu  Thu Mar 19 21:40:52 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 19 Mar 2015 16:40:52 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
Message-ID: <mefc8v$4t1$1@ger.gmane.org>

On 3/19/2015 10:32 AM, Nicholas Chammas wrote:
> OK, I hereby abandon this proposal. :)

Thank you for being so sensible ;-)

Let me first quote (out of order) your conclusion:

 > I think any part of the development process we automate
 > away is a good thing (stdlib inclusion or not notwithstanding).>

I agree, which is why I expect to do some work making it easier to 
download 3rd party modules and apply them to python code, both in a gui 
IDE framework (Idle).  I think automation should respect and augment, 
not replace. the human programmer.

> To summarize the conclusion I believe this thread has arrived at:

...

>     Ignoring that difficulty, there is also the problem of what ?spec?
>     to use for any such auto-styler. PEP 8 is an obvious choice, but it
>     was written as a guide for humans, not a spec for programmatic
>     implementation.

A program that treats PEP 8 guidelines as rigid rules, that ignores the 
first guideline "A Foolish Consistency is the Hobgoblin of Little 
Minds", that removes the human choices intentionally included in PEP 8, 
is not PEP 8 compliant.  I believe one of the motivations for some 
recent PEP 8 amendments was some unhappiness with pep8.py or other checkers.

 >     Furthermore, there are doubts about PEP 8?s utility
>     as the basis for a universal auto-styler, as it was intended for
>     code specifically in the Python standard library.

Pydev is a relatively small group of volunteers who otherwise work 
either for themselves or for various organizations scattered around the 
world.  Having the freedom to put actual readability ahead of rigid 
consistency makes volunteering more attractive.  I regard code as a form 
of poetry, but prefer a flexible but defined structure to free-form 
mishmash.

Google, as an example, is a very large corporation that pays people well 
to adhere to its guidelines and rules.  This is a different situation 
and its best choice for Python styling may well be different from pydev.

> On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert
> <abarnert at yahoo.com
> <mailto:abarnert at yahoo.com>> wrote:
>
>     But that implies that it's worth someone making a spec out of PEP 8.
>     That implies going through PEP 8, getting a sense of what parts are
>     (unambiguously and uncontroversially) mechanizable, and reorganizing
>     and minimally rewriting it to get that sense across. Having numbered
>     rules, and named anchors for each one in the document (or at least
>     to narrower sections, so you can link to something more specific
>     than "Programming Recommendations") would also be helpful.
>
> This is an excellent point and one I did not consider.

Having the guidelines numbered (id'ed), even though still regarded as 
guidelines, could help communication. Checkers could then easily refer 
to specific guidelines.

> On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>     (Note that autoPEP8 only changes whitespace, which is a small part
>     of PEP 8 recommendations, and the safest aspect of code to change.)
>
> Minor point: autopep8 can do more than just whitespace changes with the
> |--aggressive| flag
> <https://pypi.python.org/pypi/autopep8/#more-advanced-usage>.

Thanks for the link.  I was specifically thinking of global renamings to 
satisfy PEP 8's Naming Conventions.  Idlelib has a mishmash of module 
and function/method name styles.  (I might look and see what can be done 
with the undocumented (except as 'unstable') libe2to3.)

> On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>     We should be careful about adding anything.  When we do, we should
>     add libraries, not applications.  We should especially not
>     officially endorse one of many opinion-based competitors.
>
> I have come around to this view.

Accepting this view, we can still ask if the stdlib should have more 
(policy-neutral) string or list of strings reforming components than it 
does now.  Textwrap has a few functions for strings in general.  Idle 
has about 10 formatting functions exposed on its editor Format menu, but 
they are not exposed for general use.  Tools/Scripts/reindent.py contain 
some of the same functions as does autopep8.  Should there be a code 
format module that all three, and other programs, could import?

Idle and Tools/Scrips/highlight.py both have code to analyze Python code 
tokens into categories and colorize by category.  (Idle uses tkinter 
tags, highlight.py uses ANSI or HTML tags).  Can and should both use a 
common code analysis function that could be imported?  This might 
actually ease maintainance.

Should a generalized 2to3 package be renamed 'codefix' and documented? 
As 2to3, it is already used by both Tools/Scripts/2to2 and optionally by 
autopep8, and perhaps by others.  I believe it could also be used for 
some non-2to3 pep8 fixes.

-- 
Terry Jan Reedy



From guido at python.org  Thu Mar 19 22:01:39 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Mar 2015 14:01:39 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <mefc8v$4t1$1@ger.gmane.org>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
Message-ID: <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>

On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
> A program that treats PEP 8 guidelines as rigid rules, that ignores the
> first guideline "A Foolish Consistency is the Hobgoblin of Little Minds",
> that removes the human choices intentionally included in PEP 8, is not PEP
> 8 compliant.  I believe one of the motivations for some recent PEP 8
> amendments was some unhappiness with pep8.py or other checkers.
>

I have always been skeptical of automatic style checkers. They encourage a
nit-pickish attitude that makes the development process less effective, by
encouraging critique of superficial style elements rather than of the
code's structure and meaning. In contrast, the intention of PEP 8 was to
make code more readable so that its structure and meaning are more easily
understood.


>  On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert
>> <abarnert at yahoo.com
>> <mailto:abarnert at yahoo.com>> wrote:
>>
>>     But that implies that it's worth someone making a spec out of PEP 8.
>>     That implies going through PEP 8, getting a sense of what parts are
>>     (unambiguously and uncontroversially) mechanizable, and reorganizing
>>     and minimally rewriting it to get that sense across. Having numbered
>>     rules, and named anchors for each one in the document (or at least
>>     to narrower sections, so you can link to something more specific
>>     than "Programming Recommendations") would also be helpful.
>>
>> This is an excellent point and one I did not consider.
>>
>
> Having the guidelines numbered (id'ed), even though still regarded as
> guidelines, could help communication. Checkers could then easily refer to
> specific guidelines.
>

I worry this will just encourage the nit-picky attitude more.

>
>  On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu
>> <mailto:tjreedy at udel.edu>> wrote:
>>
>>     (Note that autoPEP8 only changes whitespace, which is a small part
>>     of PEP 8 recommendations, and the safest aspect of code to change.)
>>
>> Minor point: autopep8 can do more than just whitespace changes with the
>> |--aggressive| flag
>> <https://pypi.python.org/pypi/autopep8/#more-advanced-usage>.
>>
>
> Thanks for the link.  I was specifically thinking of global renamings to
> satisfy PEP 8's Naming Conventions.  Idlelib has a mishmash of module and
> function/method name styles.  (I might look and see what can be done with
> the undocumented (except as 'unstable') libe2to3.)
>

This feels hard to automate, because you don't know what names are part of
an external specification.


>  On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy <tjreedy at udel.edu
>> <mailto:tjreedy at udel.edu>> wrote:
>>
>>     We should be careful about adding anything.  When we do, we should
>>     add libraries, not applications.  We should especially not
>>     officially endorse one of many opinion-based competitors.
>>
>> I have come around to this view.
>>
>
> Accepting this view, we can still ask if the stdlib should have more
> (policy-neutral) string or list of strings reforming components than it
> does now.  Textwrap has a few functions for strings in general.  Idle has
> about 10 formatting functions exposed on its editor Format menu, but they
> are not exposed for general use.  Tools/Scripts/reindent.py contain some of
> the same functions as does autopep8.  Should there be a code format module
> that all three, and other programs, could import?
>
> Idle and Tools/Scrips/highlight.py both have code to analyze Python code
> tokens into categories and colorize by category.  (Idle uses tkinter tags,
> highlight.py uses ANSI or HTML tags).  Can and should both use a common
> code analysis function that could be imported?  This might actually ease
> maintainance.
>
> Should a generalized 2to3 package be renamed 'codefix' and documented? As
> 2to3, it is already used by both Tools/Scripts/2to2 and optionally by
> autopep8, and perhaps by others.  I believe it could also be used for some
> non-2to3 pep8 fixes.


I think that a quick "clean up whitespace" feature would be a nice addition
to IDLE, provided it's a user-selectable menu items (similar to the
existing indent/dedent operations). Refactoring, however, is fiendishly
hard for Python -- give it a try in e.g. PyCharm and you will agree.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/cc6ca080/attachment.html>

From donald at stufft.io  Thu Mar 19 22:41:50 2015
From: donald at stufft.io (Donald Stufft)
Date: Thu, 19 Mar 2015 17:41:50 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
Message-ID: <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>


> On Mar 19, 2015, at 5:01 PM, Guido van Rossum <guido at python.org> wrote:
> 
> On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu>> wrote:
> A program that treats PEP 8 guidelines as rigid rules, that ignores the first guideline "A Foolish Consistency is the Hobgoblin of Little Minds", that removes the human choices intentionally included in PEP 8, is not PEP 8 compliant.  I believe one of the motivations for some recent PEP 8 amendments was some unhappiness with pep8.py or other checkers.
> 
> I have always been skeptical of automatic style checkers. They encourage a nit-pickish attitude that makes the development process less effective, by encouraging critique of superficial style elements rather than of the code's structure and meaning. In contrast, the intention of PEP 8 was to make code more readable so that its structure and meaning are more easily understood.

My experience is actually the exact opposite. When a machine is responsible for style enforcement people generally don?t feel as bad about being asked to adjust the formatting because it?s just a machine, it can?t make value judgements. When a person asks someone to adjust something to make it easier to read it often times ends up causing a distraction where people tend to feel like since it?s a person asking them to do it they are passing judgement on their code, or they don?t agree with it and since it?s a human saying it it feels more like there is room to budge so you end up with people more likely to be willing to ask for an exemption.

In my experience projects without mechnical enforcement ends up with massively inconsistent style all throughout the code base. The Python stdlib is a good example of this. Contrast this to projects like github.com/pyca/cryptography <http://github.com/pyca/cryptography> where there is an automated mechanical enforcement of style and it?s got some bare minimum of consistency throughout the entire code base.

> 
> On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert
> <abarnert at yahoo.com <mailto:abarnert at yahoo.com>
> <mailto:abarnert at yahoo.com <mailto:abarnert at yahoo.com>>> wrote:
> 
>     But that implies that it's worth someone making a spec out of PEP 8.
>     That implies going through PEP 8, getting a sense of what parts are
>     (unambiguously and uncontroversially) mechanizable, and reorganizing
>     and minimally rewriting it to get that sense across. Having numbered
>     rules, and named anchors for each one in the document (or at least
>     to narrower sections, so you can link to something more specific
>     than "Programming Recommendations") would also be helpful.
> 
> This is an excellent point and one I did not consider.
> 
> Having the guidelines numbered (id'ed), even though still regarded as guidelines, could help communication. Checkers could then easily refer to specific guidelines.
> 
> I worry this will just encourage the nit-picky attitude more.
> 
> On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu>
> <mailto:tjreedy at udel.edu <mailto:tjreedy at udel.edu>>> wrote:
> 
>     (Note that autoPEP8 only changes whitespace, which is a small part
>     of PEP 8 recommendations, and the safest aspect of code to change.)
> 
> Minor point: autopep8 can do more than just whitespace changes with the
> |--aggressive| flag
> <https://pypi.python.org/pypi/autopep8/#more-advanced-usage <https://pypi.python.org/pypi/autopep8/#more-advanced-usage>>.
> 
> Thanks for the link.  I was specifically thinking of global renamings to satisfy PEP 8's Naming Conventions.  Idlelib has a mishmash of module and function/method name styles.  (I might look and see what can be done with the undocumented (except as 'unstable') libe2to3.)
> 
> This feels hard to automate, because you don't know what names are part of an external specification.
> 
> On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu>
> <mailto:tjreedy at udel.edu <mailto:tjreedy at udel.edu>>> wrote:
> 
>     We should be careful about adding anything.  When we do, we should
>     add libraries, not applications.  We should especially not
>     officially endorse one of many opinion-based competitors.
> 
> I have come around to this view.
> 
> Accepting this view, we can still ask if the stdlib should have more (policy-neutral) string or list of strings reforming components than it does now.  Textwrap has a few functions for strings in general.  Idle has about 10 formatting functions exposed on its editor Format menu, but they are not exposed for general use.  Tools/Scripts/reindent.py contain some of the same functions as does autopep8.  Should there be a code format module that all three, and other programs, could import?
> 
> Idle and Tools/Scrips/highlight.py both have code to analyze Python code tokens into categories and colorize by category.  (Idle uses tkinter tags, highlight.py uses ANSI or HTML tags).  Can and should both use a common code analysis function that could be imported?  This might actually ease maintainance.
> 
> Should a generalized 2to3 package be renamed 'codefix' and documented? As 2to3, it is already used by both Tools/Scripts/2to2 and optionally by autopep8, and perhaps by others.  I believe it could also be used for some non-2to3 pep8 fixes.
> 
> I think that a quick "clean up whitespace" feature would be a nice addition to IDLE, provided it's a user-selectable menu items (similar to the existing indent/dedent operations). Refactoring, however, is fiendishly hard for Python -- give it a try in e.g. PyCharm and you will agree.
> 
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/0c4c5733/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/0c4c5733/attachment-0001.sig>

From nicholas.chammas at gmail.com  Thu Mar 19 23:13:53 2015
From: nicholas.chammas at gmail.com (Nicholas Chammas)
Date: Thu, 19 Mar 2015 22:13:53 +0000
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
Message-ID: <CAOhmDzdjq6esOHLB7xodoVXS40ujw0+5KObpRErE0EfdRRt72w@mail.gmail.com>

I second Donald?s remarks regarding the impact that automatic style
checkers have on the dev process.

The Apache Spark <https://github.com/apache/spark> project, for example,
makes a number of automated checks on all proposed patches. One of those
checks is a style check.

When a proposed patch fails a style check, an informative message
explaining why is posted in the CI build log for the author to review, and
the author typically just updates their patch accordingly.

Here?s an example
<https://github.com/apache/spark/pull/4916#issuecomment-77555917> from an
Apache Spark PR that touched some Python code. The author?s patch had some
blank lines that contained whitespace, so the patch failed the automated
style check. The author just updated their patch to pass the style check
and the discussion quickly returned to the meat of what the patch was about.

>From what I?ve seen so far, the process of a patch failing style checks
plays out almost exactly like it would if a contributor had submitted a
patch that failed unit tests. The contributor just updates their patch to
pass those checks and everyone moves on. There is no arguing or friction.

And the result in the case of the Spark project is that all the Python code
in the repo passes the pep8 checks we?ve enabled. Granted, this doesn?t
mean the code is readable or beautiful or correct, but it simply means that
there is a small but non-trivial set of style problems and inconsistencies
that we just don?t have to worry about.

For a large, open source project with hundreds of contributors from many
different organizations and backgrounds, that?s a solid win.

Nick
?

On Thu, Mar 19, 2015 at 5:42 PM Donald Stufft <donald at stufft.io> wrote:

> On Mar 19, 2015, at 5:01 PM, Guido van Rossum <guido at python.org> wrote:
>
> On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> A program that treats PEP 8 guidelines as rigid rules, that ignores the
>> first guideline "A Foolish Consistency is the Hobgoblin of Little Minds",
>> that removes the human choices intentionally included in PEP 8, is not PEP
>> 8 compliant.  I believe one of the motivations for some recent PEP 8
>> amendments was some unhappiness with pep8.py or other checkers.
>>
>
> I have always been skeptical of automatic style checkers. They encourage a
> nit-pickish attitude that makes the development process less effective, by
> encouraging critique of superficial style elements rather than of the
> code's structure and meaning. In contrast, the intention of PEP 8 was to
> make code more readable so that its structure and meaning are more easily
> understood.
>
>
> My experience is actually the exact opposite. When a machine is
> responsible for style enforcement people generally don?t feel as bad about
> being asked to adjust the formatting because it?s just a machine, it can?t
> make value judgements. When a person asks someone to adjust something to
> make it easier to read it often times ends up causing a distraction where
> people tend to feel like since it?s a person asking them to do it they are
> passing judgement on their code, or they don?t agree with it and since it?s
> a human saying it it feels more like there is room to budge so you end up
> with people more likely to be willing to ask for an exemption.
>
> In my experience projects without mechnical enforcement ends up with
> massively inconsistent style all throughout the code base. The Python
> stdlib is a good example of this. Contrast this to projects like
> github.com/pyca/cryptography where there is an automated mechanical
> enforcement of style and it?s got some bare minimum of consistency
> throughout the entire code base.
>
>
>
>>  On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert
>>> <abarnert at yahoo.com
>>> <mailto:abarnert at yahoo.com>> wrote:
>>>
>>>     But that implies that it's worth someone making a spec out of PEP 8.
>>>     That implies going through PEP 8, getting a sense of what parts are
>>>     (unambiguously and uncontroversially) mechanizable, and reorganizing
>>>     and minimally rewriting it to get that sense across. Having numbered
>>>     rules, and named anchors for each one in the document (or at least
>>>     to narrower sections, so you can link to something more specific
>>>     than "Programming Recommendations") would also be helpful.
>>>
>>> This is an excellent point and one I did not consider.
>>>
>>
>> Having the guidelines numbered (id'ed), even though still regarded as
>> guidelines, could help communication. Checkers could then easily refer to
>> specific guidelines.
>>
>
> I worry this will just encourage the nit-picky attitude more.
>
>>
>>  On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy <tjreedy at udel.edu
>>> <mailto:tjreedy at udel.edu>> wrote:
>>>
>>>     (Note that autoPEP8 only changes whitespace, which is a small part
>>>     of PEP 8 recommendations, and the safest aspect of code to change.)
>>>
>>> Minor point: autopep8 can do more than just whitespace changes with the
>>> |--aggressive| flag
>>> <https://pypi.python.org/pypi/autopep8/#more-advanced-usage>.
>>>
>>
>> Thanks for the link.  I was specifically thinking of global renamings to
>> satisfy PEP 8's Naming Conventions.  Idlelib has a mishmash of module and
>> function/method name styles.  (I might look and see what can be done with
>> the undocumented (except as 'unstable') libe2to3.)
>>
>
> This feels hard to automate, because you don't know what names are part of
> an external specification.
>
>
>>  On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy <tjreedy at udel.edu
>>> <mailto:tjreedy at udel.edu>> wrote:
>>>
>>>     We should be careful about adding anything.  When we do, we should
>>>     add libraries, not applications.  We should especially not
>>>     officially endorse one of many opinion-based competitors.
>>>
>>> I have come around to this view.
>>>
>>
>> Accepting this view, we can still ask if the stdlib should have more
>> (policy-neutral) string or list of strings reforming components than it
>> does now.  Textwrap has a few functions for strings in general.  Idle has
>> about 10 formatting functions exposed on its editor Format menu, but they
>> are not exposed for general use.  Tools/Scripts/reindent.py contain some of
>> the same functions as does autopep8.  Should there be a code format module
>> that all three, and other programs, could import?
>>
>> Idle and Tools/Scrips/highlight.py both have code to analyze Python code
>> tokens into categories and colorize by category.  (Idle uses tkinter tags,
>> highlight.py uses ANSI or HTML tags).  Can and should both use a common
>> code analysis function that could be imported?  This might actually ease
>> maintainance.
>>
>> Should a generalized 2to3 package be renamed 'codefix' and documented? As
>> 2to3, it is already used by both Tools/Scripts/2to2 and optionally by
>> autopep8, and perhaps by others.  I believe it could also be used for some
>> non-2to3 pep8 fixes.
>
>
> I think that a quick "clean up whitespace" feature would be a nice
> addition to IDLE, provided it's a user-selectable menu items (similar to
> the existing indent/dedent operations). Refactoring, however, is fiendishly
> hard for Python -- give it a try in e.g. PyCharm and you will agree.
>
> --
> --Guido van Rossum (python.org/~guido)
>  _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/2db40745/attachment.html>

From cs at zip.com.au  Thu Mar 19 23:29:01 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 20 Mar 2015 09:29:01 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <5508F42F.8020604@sdamon.com>
References: <5508F42F.8020604@sdamon.com>
Message-ID: <20150319222901.GA72481@cskk.homeip.net>

On 17Mar2015 23:42, Alexander Walters <tritium-list at sdamon.com> wrote:
>What coding style would be used for such a tool?  PEP-8?  What about a 
>tool for those of us who believe that PEP-8 got a few things wrong?

autopep8 at least lets you pick and choose specific issues to apply or ignore.  
As someone who now aims for a "mostly pep8" style, that is very handy.

Aside: I always wish "fix it" tools to have this kind of granularity. In a 
former life I got to use SGI's C compiler. Like GCC, it had a "lots of 
warnings" mode, somewhat like a builtin lint. _Unlike_ GCC, the SGI compiler 
let one suppress specific warnings, which was very very nice.

>What about for the tribal knowledge that pythonistas develop over time
>that are not part of the standard?  Would such a tool include this?

Such things get progressively harder to formalise into stuff a tool can 
implement. I imagine one could implement quite a lot of knobs, but I also 
imagine that some knobs would require huge code support, and perhaps be 
intrusive enough to make the default knobs (very?) hard to debug/maintain.

Using PEP8 as a guide at least gives a concrete baseline for the feature set.  
I'm not arguing that that is all one should ever want, just that it is well 
defined and very broadly appplicable.

>And what of third party tools that do the same thing?  I have no
>evidence to support this, but I have the feeling that when the
>standard library includes something in a new space, third party work
>in that space slows (json and sqlite come to mind).

Sure. But that isn't always a bad thing; life is too short.

In an ideal world the included battery is amenable to extension and then the 
supplied battery enables faster third party work, at least in the form of 
adding to the battery.

>Good coding style really should not be something we ship in the
>library, it should be something we teach.

Yes and no.

I wholely agree that good style comes from within, and that it should be taught 
"what and why"; blindly applied mechanical style does not itself improve code 
quality. Though having consistent style does at least usually improve code 
readability and maintainability. Not to mention diff noise.

That said, having a tool that will _apply_ a desirable style is very useful.  

Taking myself as an example: my personal library has gads of old code with bad 
style (and bad design choices!) I tend to mostly-PEP8 the old code as I come 
across it during development (committing separately from semantic changes 
unless I've rewritten something outright). That work is slow and incremental.

Having just downloaded autopep8 and given it a tiny trial run against a single 
module I found that a bare run produced a diff of more or less the entire file, 
but adding "--indent-size 2" (my personal major divergence from PEP8) got the 
diff down to a reviewable bunch of small changes, all of which I approve.

Performing that task by hand was an ongoing indefinite inconsistent minor 
improvement activity. Now I can branch and restyle vigorously, because I have a 
tool to do the picky editing. Yay!

Aside: why the bad style from the outset? When coming to a new language or 
system I find it useful to write batteries myself, sometimes because the 
supplied battery does not match my needs but usually because the battery is 
tractable and I learn quite a lot about the problem domain during this process.  
And the same applies to style; my old code has poor style, some leaked across 
from other languages. Having written plenty of code according to my whim, I can 
then look at a suggested style like PEP8 and evaluate it with context. Now I 
mostly aim for PEP8 because I've come to informed decisions about its choices, 
and mostly agree with them.

Metaaside: the above is also something of an excuse for the state of some of 
the code I've put out there:-)

Cheers,
Cameron Simpson <cs at zip.com.au>

Mac OS X. Because making Unix user-friendly is easier than debugging Windows.
- Mike Dawson, Macintosh Systems Administrator and Consultation.
  mdawson at mac.com http://herowars.onestop.net

From guido at python.org  Fri Mar 20 00:55:08 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 19 Mar 2015 16:55:08 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <550B3FED.9020200@udel.edu>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <550B3FED.9020200@udel.edu>
Message-ID: <CAP7+vJK6C_t1nih8H01MB2fNUC=-3chWi3ZwpumWgJdEBRVtPg@mail.gmail.com>

On Thu, Mar 19, 2015 at 2:30 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/19/2015 5:01 PM, Guido van Rossum wrote:
>
>> On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy at udel.edu
>>
>
>      Having the guidelines numbered (id'ed), even though still regarded
>>     as guidelines, could help communication. Checkers could then easily
>>     refer to specific guidelines.
>>
>> I worry this will just encourage the nit-picky attitude more.
>>
>
> I agree that this is a possible negative.
>

I'll leave the rest of the discussion to other participants. It seems all
possible POVs are well-represented.


>      Thanks for the link.  I was specifically thinking of global
>>     renamings to satisfy PEP 8's Naming Conventions.  Idlelib has a
>>     mishmash of module and function/method name styles.  (I might look
>>     and see what can be done with the undocumented (except as
>>     'unstable') libe2to3.)
>>
>> This feels hard to automate, because you don't know what names are part
>> of an external specification.
>>
>
> Modernizing idlelib names, in relation to PEP 434, is a separate
> discussion.
>
>  I think that a quick "clean up whitespace" feature would be a nice
>> addition to IDLE, provided it's a user-selectable menu items (similar to
>> the existing indent/dedent operations).
>>
>
> The Format menu now has 'Strip trailing whitespace', which may be new
> since you last looked.  It needs to strip trailing blank lines, as
> reindent.py does, in addition to trailing whitespace on each line, to make
> files ready to commit to the cpython repository.  Did you have in mind also
> doing within-line cleanups (as a separate menu entry)?
>

I probably spoke out of turn (I haven't really used IDLE in years). But I
could imagine another command that did something like autopep8 on a
selection, making sure the whitespace around various operators is
consistent with PEP 8, fixing indentation levels, and even
inserting/removing blank lines in certain cases. I'm not sure I would want
it to automatically break long lines, as I often have very specific ideas
about the best place to break a long line, which require understanding the
meaning of the code.

Have you used Emacs? The python-mode written by Tim Peters and maintained
by Barry Warsaw has some pretty amazing reformatting capabilities (though I
rarely use them, because my fingers automatically type correctly formatted
code :-).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150319/9315f5f5/attachment.html>

From ben+python at benfinney.id.au  Fri Mar 20 04:02:29 2015
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 20 Mar 2015 14:02:29 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
References: <5508F42F.8020604@sdamon.com>
 <20150319222901.GA72481@cskk.homeip.net>
Message-ID: <857fucqsl6.fsf@benfinney.id.au>

Cameron Simpson <cs at zip.com.au> writes:

> In a former life I got to use SGI's C compiler. Like GCC, it had a
> "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC,
> the SGI compiler let one suppress specific warnings, which was very
> very nice.

GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics
<URL:https://gcc.gnu.org/onlinedocs/gcc-4.3.3/gcc/Diagnostic-Pragmas.html#Diagnostic-Pragmas>,
is that what you mean?

-- 
 \       ?I cannot be angry at God, in whom I do not believe.? ?Simone |
  `\                                                       De Beauvoir |
_o__)                                                                  |
Ben Finney


From stephen at xemacs.org  Fri Mar 20 06:02:47 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 20 Mar 2015 14:02:47 +0900
Subject: [Python-ideas] gofmt for Python: standardized styling as
	a	language feature
In-Reply-To: <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
Message-ID: <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>

Donald Stufft writes:
>>>>> Guido van Rossum writes:

 > > I have always been skeptical of automatic style checkers. They
 > > encourage a nit-pickish attitude that makes the development
 > > process less effective, by encouraging critique of superficial
 > > style elements rather than of the code's structure and
 > > meaning. In contrast, the intention of PEP 8 was to make code
 > > more readable so that its structure and meaning are more easily
 > > understood.
 > 
 > My experience is actually the exact opposite.

This is one of those things where people differ.  I'm happy to follow
Guido on this, although personally I've been able to adapt to
mechanical rules where required.

 > In my experience projects without mechnical enforcement ends up
 > with massively inconsistent style all throughout the code base. The
 > Python stdlib is a good example of this. Contrast this to projects
 > like github.com/pyca/cryptography
 > <http://github.com/pyca/cryptography> where there is an automated
 > mechanical enforcement of style and it?s got some bare minimum of
 > consistency throughout the entire code base.

Which proves what?  That enforcing minimal style rules (and I think we
all agree that there's plenty of good and bad style that machines
can't yet reliably distinguish) results in a code base that follows
minimal style rules?


From tjreedy at udel.edu  Fri Mar 20 06:11:27 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 20 Mar 2015 01:11:27 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
Message-ID: <mega6a$pot$1@ger.gmane.org>

On 3/19/2015 5:01 PM, Guido van Rossum wrote:
 > On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy at udel.edu

 >     Having the guidelines numbered (id'ed), even though still regarded
 >     as guidelines, could help communication. Checkers could then easily
 >     refer to specific guidelines.
 >
 > I worry this will just encourage the nit-picky attitude more.

I agree that this is a possible negative.

 >     Thanks for the link.  I was specifically thinking of global
 >     renamings to satisfy PEP 8's Naming Conventions.  Idlelib has a
 >     mishmash of module and function/method name styles.  (I might look
 >     and see what can be done with the undocumented (except as
 >     'unstable') libe2to3.)
 >
 > This feels hard to automate, because you don't know what names are part
 > of an external specification.

Modernizing idlelib names, in relation to PEP 434, is a separate discussion.

 > I think that a quick "clean up whitespace" feature would be a nice
 > addition to IDLE, provided it's a user-selectable menu items (similar to
 > the existing indent/dedent operations).

The Format menu now has 'Strip trailing whitespace', which may be new 
since you last looked.  It needs to strip trailing blank lines, as 
reindent.py does, in addition to trailing whitespace on each line, to 
make files ready to commit to the cpython repository.  Did you have in 
mind also doing within-line cleanups (as a separate menu entry)?

--
Terry Jan Reedy



From donald at stufft.io  Fri Mar 20 06:12:11 2015
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Mar 2015 01:12:11 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
 <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>


> On Mar 20, 2015, at 1:02 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> 
> Donald Stufft writes:
>>>>>> Guido van Rossum writes:
> 
>>> I have always been skeptical of automatic style checkers. They
>>> encourage a nit-pickish attitude that makes the development
>>> process less effective, by encouraging critique of superficial
>>> style elements rather than of the code's structure and
>>> meaning. In contrast, the intention of PEP 8 was to make code
>>> more readable so that its structure and meaning are more easily
>>> understood.
>> 
>> My experience is actually the exact opposite.
> 
> This is one of those things where people differ.  I'm happy to follow
> Guido on this, although personally I've been able to adapt to
> mechanical rules where required.
> 
>> In my experience projects without mechnical enforcement ends up
>> with massively inconsistent style all throughout the code base. The
>> Python stdlib is a good example of this. Contrast this to projects
>> like github.com/pyca/cryptography
>> <http://github.com/pyca/cryptography> where there is an automated
>> mechanical enforcement of style and it?s got some bare minimum of
>> consistency throughout the entire code base.
> 
> Which proves what?  That enforcing minimal style rules (and I think we
> all agree that there's plenty of good and bad style that machines
> can't yet reliably distinguish) results in a code base that follows
> minimal style rules?
> 

Sure, just like tests can?t guarantee that a code base works in every
situation but that it works in some minimal set of tests (where minimal
is whatever tests were written).

Humans are inconsistent and emotional, people are more likely to nitpick
a patch that they don?t particularly like in an unconscious attempt to
discourage the contribution. They are also far more likely to let things
slip when it?s a contribution from someone they like or even from themselves.

Working on the Python stdlib is somewhat frustrating to me in this regard
because the code in the stdlib is often times wildly inconsistent even within
the same module.

Maybe having inconsistent style which is inconsistently applied doesn?t
bother you, that?s fine. I don?t particularly mind if the stdlib gains an
automated style checker or formatter because I?m happy using third party
modules. I was merely offering my experience that *anything* which relies
on a human to verify it is, without exception, going to be verified unevenly
and that using a human to verify it invites people to attempt to argue
against it more often than when a machine does it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/84bf3d6f/attachment-0001.sig>

From abarnert at yahoo.com  Fri Mar 20 06:32:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 19 Mar 2015 22:32:20 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
 <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>
 <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>
Message-ID: <94D070AA-D5F6-4FAA-91E4-250A1011A823@yahoo.com>

On Mar 19, 2015, at 10:12 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> Working on the Python stdlib is somewhat frustrating to me in this regard
> because the code in the stdlib is often times wildly inconsistent even within
> the same module.

The stdlib is full of milkmen^H^H^H^H^H^H^Hmodules. Some of them are very old. Often older than PEP 8--and presumably the inspiration for PEP 8 to be written in the first place.

At the time PEP 8 was written, obviously nobody went through and rewrote the whole stdlib to be compliant. Why not? Maybe no one volunteered--or maybe someone did, but it was rejected because the costs (risk of introducing new bugs, headaches every time you need to backport a fix across the rewrite, or even just look at the diffs to see what changed when, etc.) outweighed the benefits.

Over time, some parts that touched on the visible interface, that needed to be rewritten anyway, or that are intended to serve as sample code (as in, the source is linked from the docs) have been restyled, but most of the code hasn't. Since most of those updates were in 3.0, and it seems an obvious place for a flag-day restyling if ever there were to be one, I'll bet if you search the py3k archives you'll find some relevant discussions. (Unless Guido or someone else knows off the top of his head.)

From stephen at xemacs.org  Fri Mar 20 07:06:37 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 20 Mar 2015 15:06:37 +0900
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
 <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>
 <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>
Message-ID: <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp>

Donald Stufft writes:

 > Working on the Python stdlib is somewhat frustrating to me in this
 > regard because the code in the stdlib is often times wildly
 > inconsistent even within the same module.

Maybe you haven't paid attention to past discussions, but the Python
stdlib is a terrible example because fixing it up is a FAQ on this
list, on core-mentoriship, and (in the past) on python-dev.  It has
been repeatedly vetoed on the grounds that changes, even in
whitespace, are likely to introduce more bugs than consistent style is
worth.

If you're actually working *on* the stdlib, then improve the style as
you fix bugs or add features.  If you are working on something that
requires studying the stdlib, then you're out of luck.

None of the above means that improving the stdlib's coding style is
off the table as far as I'm concerned.  I personally wouldn't mind
seeing it happen, but I don't care that much.

What it means is simply that the stdlib is the worst possible example
to make your case -- everybody agrees with you about the inconsistent
style, but Those Who Make Decisions Around Here think there are more
important considerations than style.

 > I was merely offering my experience that *anything* which relies on
 > a human to verify it is, without exception, going to be verified
 > unevenly and that using a human to verify it invites people to
 > attempt to argue against it more often than when a machine does it.

But the stdlib is a terrible example for that purpose for the same
reason.

But I take your point that using a machine can smooth correction of
"mechanical" errors.  Mine is simply that I don't see that it makes
such a big difference in my experience; most people who have horrible
coding styles respond to human advice quite well, and the "minor"
differences are -- minor.

YMMV.


From encukou at gmail.com  Fri Mar 20 15:24:04 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 20 Mar 2015 15:24:04 +0100
Subject: [Python-ideas] A macro for easier rich comparisons
In-Reply-To: <CA+=+wqD6R5-vg9W250jbwUceCuizD8vJDJ=meYVr0Z3BmO4JGA@mail.gmail.com>
References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com>
 <CA+=+wqD6R5-vg9W250jbwUceCuizD8vJDJ=meYVr0Z3BmO4JGA@mail.gmail.com>
Message-ID: <CA+=+wqCfv=hhuFzA6EMF4n7ru++fOO0aXfgVdPwKT=RCSPuVwg@mail.gmail.com>

On Wed, Mar 18, 2015 at 2:48 PM, Petr Viktorin <encukou at gmail.com> wrote:
> On Wed, Mar 18, 2015 at 12:01 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 18.03.2015 11:53, Petr Viktorin wrote:
>>> ...
>>> I propose adding a public macro to ease this. My version takes two C-orderable values and the
>>> operation, similar to richcmpfunc:
>>>
>>> #define Py_RICHCOMPARE(val1, val2, op) ( \
>>>     ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \
>>>     ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \
>>>     ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \
>>>     ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \
>>>     ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \
>>>     ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \
>>>     (Py_INCREF(Py_NotImplemented), Py_NotImplemented))
>>>
>>> (As for the behavior for unknown op: for most cases the best thing to do is setting an error and
>>> returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython
>>> either return NULL without error (e.g. tupleobject), or fall through and return True/False
>>> arbitrarily (as in bytearrayobject). Datetime does an assert.
>>> I think Py_NotImplemented is a good value to return)
>>> ...
>>> Is this a PEP-worthy idea?
>>
>> This is a great idea, no need for a PEP :-) Please submit the patch on
>> the bug tracker and also include a patch for the C API documentation.
>
> Issue with patches is here: http://bugs.python.org/issue23699

Notes from the patch review: it is not so clear-cut after all.

A sequence of comparisons is less efficient than a switch statement.
Same with PyBool_FromLong vs. Py_RETURN_*
Also, it might be too complex for a macro.
These points could be solved by making it a function instead.

Another issue raised is that the use cases are too limited limited for
this to be a part of the Python API.
I don't agree with that; I think boilerplate-reducing features are
even if their usage is limited (see Py_RETURN_NOTIMPLEMENTED or
PyModule_AddIntConstant).

From donald at stufft.io  Fri Mar 20 16:50:44 2015
From: donald at stufft.io (Donald Stufft)
Date: Fri, 20 Mar 2015 11:50:44 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <A35FDAD9-00E2-497E-91AB-04BF8397BA45@stufft.io>
 <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp>
 <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io>
 <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <04C22794-C552-4DBF-A53F-E430D420F7AC@stufft.io>


> On Mar 20, 2015, at 2:06 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> 
> Donald Stufft writes:
> 
>> Working on the Python stdlib is somewhat frustrating to me in this
>> regard because the code in the stdlib is often times wildly
>> inconsistent even within the same module.
> 
> Maybe you haven't paid attention to past discussions, but the Python
> stdlib is a terrible example because fixing it up is a FAQ on this
> list, on core-mentoriship, and (in the past) on python-dev.  It has
> been repeatedly vetoed on the grounds that changes, even in
> whitespace, are likely to introduce more bugs than consistent style is
> worth.
> 
> If you're actually working *on* the stdlib, then improve the style as
> you fix bugs or add features.  If you are working on something that
> requires studying the stdlib, then you're out of luck.
> 
> None of the above means that improving the stdlib's coding style is
> off the table as far as I'm concerned.  I personally wouldn't mind
> seeing it happen, but I don't care that much.
> 
> What it means is simply that the stdlib is the worst possible example
> to make your case -- everybody agrees with you about the inconsistent
> style, but Those Who Make Decisions Around Here think there are more
> important considerations than style.
> 
>> I was merely offering my experience that *anything* which relies on
>> a human to verify it is, without exception, going to be verified
>> unevenly and that using a human to verify it invites people to
>> attempt to argue against it more often than when a machine does it.
> 
> But the stdlib is a terrible example for that purpose for the same
> reason.
> 
> But I take your point that using a machine can smooth correction of
> "mechanical" errors.  Mine is simply that I don't see that it makes
> such a big difference in my experience; most people who have horrible
> coding styles respond to human advice quite well, and the "minor"
> differences are -- minor.
> 
> YMMV.
> 

I used the stdlib primarily because it was an agreed upon fact that
the style there was inconsistent. However it?s not just in old
modules, it?s in any module that has had more than a handful of
contributors to it. I was picking on the stdlib to avoid calling out
other people?s software where it wasn?t an agreed upon thing that
it was bad.

If you want non stdlib examples, pip itself and virtualenv are both
great (terrible?) examples of wildly inconsistent style. Pip is better
now because we went through and fixed it all and are now running
automated style checks as part of our CI.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/5b0c43b2/attachment.sig>

From luoyonggang at gmail.com  Fri Mar 20 19:24:54 2015
From: luoyonggang at gmail.com (=?UTF-8?B?572X5YuH5YiaKFlvbmdnYW5nIEx1bykg?=)
Date: Sat, 21 Mar 2015 02:24:54 +0800
Subject: [Python-ideas] class os.DirEntry is confusing,
Message-ID: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>

Should be replaced with os.FsEntry or os.ScanEntry

-- 
         ??
?
???
Yours
    sincerely,
Yonggang Luo

From cs at zip.com.au  Fri Mar 20 23:00:25 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 21 Mar 2015 09:00:25 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <857fucqsl6.fsf@benfinney.id.au>
References: <857fucqsl6.fsf@benfinney.id.au>
Message-ID: <20150320220025.GA91756@cskk.homeip.net>

On 20Mar2015 14:02, Ben Finney <ben+python at benfinney.id.au> wrote:
>Cameron Simpson <cs at zip.com.au> writes:
>> In a former life I got to use SGI's C compiler. Like GCC, it had a
>> "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC,
>> the SGI compiler let one suppress specific warnings, which was very
>> very nice.
>
>GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics
><URL:https://gcc.gnu.org/onlinedocs/gcc-4.3.3/gcc/Diagnostic-Pragmas.html#Diagnostic-Pragmas>,
>is that what you mean?

In terms of function, yes, though it says "only warnings [...] can be 
controlled, and not all of them". In terms of usability, no.

I'd rather put compiler options to control noise in Makefiles (or whatever 
built system) rather than litter code with compiler specific junk which has no 
effect (one hopes) elsewhere (other platforms, other compilers).

Also, I don't think these #pragmas were around at the time:-)

Cheers,
Cameron Simpson <cs at zip.com.au>

Tim's Top 5:
 #5. Know when to give cages the bird.
 #4. I'm not hanging off, so you don't need to hang off either.
     (especially on the "up" side)
 #3. Always let me know when you're gonna get on and off.  And by God,
     don't let your boot scratch my polished exhaust canister.
 #2. Remain seated until the aircraft has safely stopped at the
     passenger terminal and the seat belt sign has been extinguished.
 #1. Shut the hell up, it'll all be over in a minute.
        - Timothy R. Redden <reddent at sequoia.csus.edu>

From barry at python.org  Fri Mar 20 23:07:47 2015
From: barry at python.org (Barry Warsaw)
Date: Fri, 20 Mar 2015 18:07:47 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <5508F42F.8020604@sdamon.com>
 <CAPkN8xLo+StFgyuSoR8BtNVDcKF6gGbsmTmuX1dh+k0C57kr3A@mail.gmail.com>
 <CAOhmDzfem2wDu3hyvaViWDVj4tLXkehQB-quYGZfT61ZHLLbpA@mail.gmail.com>
 <CAF-Rda_h5HW4WKbxt8xmBRifGuHAYx=ehZ48Z-3bN2VKxS5bpQ@mail.gmail.com>
 <CAOhmDzf=9iOBhBauDi+r0vBjPvrejg6dSZSP38iBOpmW1tE7VA@mail.gmail.com>
 <mefc8v$4t1$1@ger.gmane.org>
 <CAP7+vJL=c28SYXYsZeNiSdoz094WTUM7F5Y3i4mKkJ+VO_bhJg@mail.gmail.com>
 <550B3FED.9020200@udel.edu>
 <CAP7+vJK6C_t1nih8H01MB2fNUC=-3chWi3ZwpumWgJdEBRVtPg@mail.gmail.com>
Message-ID: <20150320180747.2507167e@anarchist.wooz.org>

On Mar 19, 2015, at 04:55 PM, Guido van Rossum wrote:

>Have you used Emacs? The python-mode written by Tim Peters and maintained
>by Barry Warsaw has some pretty amazing reformatting capabilities (though I
>rarely use them, because my fingers automatically type correctly formatted
>code :-).

I must admit to a long-standing lack of enthusiasm for Elisp hacking, so much
of the recent credit for python-mode's development goes to Andreas R?hler. :)

Simple additions of pyflakes (some folks like flake8) and whitespace-mode make
python-mode really fantastic for quickly spotting questionable code.  My
fingers too are well-trained but sometimes my brain gets in the way, and then
Emacs will clearly show me lines that need clean up.  I'd like that better
than something that automatically reformats my code.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/41a119ca/attachment-0001.sig>

From rymg19 at gmail.com  Fri Mar 20 23:06:53 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 20 Mar 2015 17:06:53 -0500
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <20150320220025.GA91756@cskk.homeip.net>
References: <857fucqsl6.fsf@benfinney.id.au>
 <20150320220025.GA91756@cskk.homeip.net>
Message-ID: <CAO41-mMzhfPjQLPxR5E_Vj8NrqtOMbBH_emE34=-LzHpFH38_A@mail.gmail.com>

Can't you do that witout pragmas?

gcc tst.c -Wall -Wno-parenthesis -Wno-unused-variable ...


On Fri, Mar 20, 2015 at 5:00 PM, Cameron Simpson <cs at zip.com.au> wrote:

> On 20Mar2015 14:02, Ben Finney <ben+python at benfinney.id.au> wrote:
>
>> Cameron Simpson <cs at zip.com.au> writes:
>>
>>> In a former life I got to use SGI's C compiler. Like GCC, it had a
>>> "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC,
>>> the SGI compiler let one suppress specific warnings, which was very
>>> very nice.
>>>
>>
>> GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics
>> <URL:https://gcc.gnu.org/onlinedocs/gcc-4.3.3/gcc/
>> Diagnostic-Pragmas.html#Diagnostic-Pragmas>,
>> is that what you mean?
>>
>
> In terms of function, yes, though it says "only warnings [...] can be
> controlled, and not all of them". In terms of usability, no.
>
> I'd rather put compiler options to control noise in Makefiles (or whatever
> built system) rather than litter code with compiler specific junk which has
> no effect (one hopes) elsewhere (other platforms, other compilers).
>
> Also, I don't think these #pragmas were around at the time:-)
>
> Cheers,
> Cameron Simpson <cs at zip.com.au>
>
> Tim's Top 5:
> #5. Know when to give cages the bird.
> #4. I'm not hanging off, so you don't need to hang off either.
>     (especially on the "up" side)
> #3. Always let me know when you're gonna get on and off.  And by God,
>     don't let your boot scratch my polished exhaust canister.
> #2. Remain seated until the aircraft has safely stopped at the
>     passenger terminal and the seat belt sign has been extinguished.
> #1. Shut the hell up, it'll all be over in a minute.
>        - Timothy R. Redden <reddent at sequoia.csus.edu>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/72bcf94f/attachment.html>

From cs at zip.com.au  Fri Mar 20 23:18:36 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Sat, 21 Mar 2015 09:18:36 +1100
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAO41-mMzhfPjQLPxR5E_Vj8NrqtOMbBH_emE34=-LzHpFH38_A@mail.gmail.com>
References: <CAO41-mMzhfPjQLPxR5E_Vj8NrqtOMbBH_emE34=-LzHpFH38_A@mail.gmail.com>
Message-ID: <20150320221836.GA52250@cskk.homeip.net>

On 20Mar2015 17:06, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>Can't you do that witout pragmas?
>
>gcc tst.c -Wall -Wno-parenthesis -Wno-unused-variable ...

Now, presumably yes. Then, definitely no.

So I presume the situation is improved a lot.

Cheers,
Cameron Simpson <cs at zip.com.au>

If you 'aint falling off, you ar'nt going hard enough.  - Fred Gassit

From abarnert at yahoo.com  Fri Mar 20 23:19:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 20 Mar 2015 15:19:09 -0700
Subject: [Python-ideas] A macro for easier rich comparisons
In-Reply-To: <CA+=+wqCfv=hhuFzA6EMF4n7ru++fOO0aXfgVdPwKT=RCSPuVwg@mail.gmail.com>
References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com>
 <CA+=+wqD6R5-vg9W250jbwUceCuizD8vJDJ=meYVr0Z3BmO4JGA@mail.gmail.com>
 <CA+=+wqCfv=hhuFzA6EMF4n7ru++fOO0aXfgVdPwKT=RCSPuVwg@mail.gmail.com>
Message-ID: <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com>

On Mar 20, 2015, at 7:24 AM, Petr Viktorin <encukou at gmail.com> wrote:
> 
> A sequence of comparisons is less efficient than a switch statement.
> Same with PyBool_FromLong vs. Py_RETURN_*
> Also, it might be too complex for a macro.
> These points could be solved by making it a function instead.

Or by just making the macro a statement instead of a function; then the code can be exactly the same code you're abstracting out, and instead of using it like this:

     return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op);

... you use it like this:

    PY_RETURNRICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op);


From paddy3118 at gmail.com  Sat Mar 21 05:41:03 2015
From: paddy3118 at gmail.com (Paddy3118)
Date: Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
Message-ID: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>

I had to add a comment to an RC entry 
<http://rosettacode.org/wiki/Longest_common_prefix#Python>that was using 
os.path.commonprefix to compute the longest common prefix which it does 
regardless of path separators, (which is what it should do).

It has been discussed before <http://bugs.python.org/issue10395>- it goes 
back to the nineties it seems, but we are still left with a misplaced 
function, (it should be str.commonprefix), with an awful hack of saying 
"yes we know its ballsed up" in the documentation 
<https://docs.python.org/3.5/library/os.path.html?highlight=commonprefix#os.path.commonprefix>
.

I guess we missed a great opportunity to fix this when we moved to Python 3 
too!?

The  fix seems clear: deprecate os.path.commonprefix whilst creating a true 
os.path.commonpath and str.commonprefix. The deprecated function should 
hang around and the standard libs modified to switch to the new function(s)

I've heard that some religious people put obvious faults in their work as 
only their god should get so close to perfection - maybe that's why we 
still have this wart :-)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/72c5f132/attachment.html>

From victor.stinner at gmail.com  Sat Mar 21 11:23:26 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 21 Mar 2015 11:23:26 +0100
Subject: [Python-ideas] class os.DirEntry is confusing,
In-Reply-To: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>
References: <CAE2XoE_YZCgZSQJqdQgZ1kkT_g9U2Dxqq2HwNAfotAXdHtJHfQ@mail.gmail.com>
Message-ID: <CAMpsgwbSdb+wCp6xYy3bet1QxAAKQz2WhEnYS1W9QfxkLY1YJw@mail.gmail.com>

Hi,

The "DirEntry" name comes from the POSIX structure "direntry". You should
not instanciate the DirEntry class nor store it, so not check its type
neither (isinstance(something, os.DirEntry)). I don't think that the exact
name of the class matters. I had prefer DirectoryEntry, but "dir"
shortcut is common in the Python os module (os.listdir) and in POSIX
strandard (opendir, struct direntry).

You are coming late. The PEP was discussed during 3 months (may-july 2014),
and Ben started to work on it since 2012! See the PEP for details:
https://www.python.org/dev/peps/pep-0471/#naming

I accepted the PEP and I don't plan to modify it even if you have a serious
rationale for the change. Bikeshedding is not :-) By the way, the PEP is
implemnted. os.scandir() is already in the latest python 3.5 alpha.
Optimized os.walk() will be in the next alpha.

Victor

Le vendredi 20 mars 2015, ???(Yonggang Luo) <luoyonggang at gmail.com> a
?crit :

> Should be replaced with os.FsEntry or os.ScanEntry
>
> --
>          ??
> ?
> ???
> Yours
>     sincerely,
> Yonggang Luo
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150321/41d56935/attachment-0001.html>

From solipsis at pitrou.net  Sat Mar 21 12:47:32 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 21 Mar 2015 12:47:32 +0100
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
Message-ID: <20150321124732.0b47730e@fsol>

On Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
Paddy3118 <paddy3118 at gmail.com> wrote:
> 
> The  fix seems clear: deprecate os.path.commonprefix whilst creating a true 
> os.path.commonpath and str.commonprefix. The deprecated function should 
> hang around and the standard libs modified to switch to the new function(s)

+1 from me.

> I've heard that some religious people put obvious faults in their work as 
> only their god should get so close to perfection - maybe that's why we 
> still have this wart :-)

If by chance their god is Guido, then at least /he/ can intervene to
fix his disciples' mess ;-)

Regards

Antoine.



From guido at python.org  Fri Mar 20 17:59:32 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 20 Mar 2015 09:59:32 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
Message-ID: <CAP7+vJL_XR7e+FUMYWYVhsy+ZsNg6xiOg8VVoew3_JBh86b2Rw@mail.gmail.com>

Here's an updated draft for PEP 484. There's more to do (especially generic
types need much more thought and writing) but I've done a bunch of editing
and thinking so I think this is ready for another round of review.
Remember, for technical questions it's often best to use the GitHub tracker
for this PEP at https://github.com/ambv/typehinting/issues .

Significant changes in this draft:
- Define stubs.
- Define `@overload`.
- Describe `cast()`.
- Fix description of `Any`.
- Describe `Callable[..., t]`.
- Explain why `List[t]` instead of `List<t>`.
- Add section on rejected alternatives.
- Various other edits for clarity.

Incomplete list of TODOs:
- Define and explain generics.
- Covariance vs. contravariance (
https://github.com/ambv/typehinting/issues/2)
- Other edits as indicated by FIXME comments.
- Other edits mentioned in
https://github.com/ambv/typehinting/blob/master/README.rst .

Here's the full text of the PEP (hopefully it will appear soon at
https://www.python.org/dev/peps/pep-0484/):

PEP: 484
Title: Type Hints
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo <
jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl>
Discussions-To: Python-Dev <python-dev at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Sep-2014
Post-History: 16-Jan-2015,20-Mar-2015
Resolution:


Abstract
========

This PEP introduces a standard syntax for type hints using annotations
(PEP 3107) on function definitions.  For example, here is a simple
function whose argument and return type are declared in the
annotations::

  def greeting(name: str) -> str:
      return 'Hello ' + name

While these annotations are available at runtime through the usual
``__annotations__`` attribute, *no type checking happens at runtime*.
Instead, the proposal assumes the existence of a separate off-line
type checker which users can run over their source code voluntarily.
Essentially, such a type checker acts as a very powerful linter.

The proposal is strongly inspired by mypy [mypy]_.  For example, the
type "sequence of integers" can be written as ``Sequence[int]``.  The
square brackets mean that no new syntax needs to be added to the
language.  The example here uses a custom class ``Sequence``, imported
from a pure-Python module ``typing.py``.  The ``Sequence[int]``
notation works by implementing ``__getitem__()`` in the metaclass.

The type system supports unions, generic types, and a special type
named ``Any`` which is consistent with (i.e. assignable to and from) all
types.  This latter feature is taken from the idea of gradual typing.
Gradual typing and the full type system are explained in PEP 483.

Other approaches from which we have borrowed or to which ours can be
compared and contrasted are described in PEP 482.


Rationale and Goals
===================

PEP 3107 added support for arbitrary annotations on parts of a function
definition.  Although no meaning was assigned to annotations then, there
has always been an implicit goal to use them for type hinting, which is
listed as the first possible use case in said PEP.

This PEP aims to provide a standard syntax for type annotations, opening
up Python code to easier static analysis and refactoring, potential
runtime type checking, and performance optimizations utilizing type
information.

Of these goals, static analysis is the most important.  This includes
support for off-line type checkers such as mypy, as well as providing
a standard notation that can be used by IDEs for code completion and
refactoring.

Non-goals
---------

While the proposed typing module will contain some building blocks for
runtime type checking -- in particular a useful ``isinstance()``
implementation -- third party packages would have to be developed to
implement specific runtime type checking functionality, for example
using decorators or metaclasses.  Using type hints for performance
optimizations is left as an exercise for the reader.

It should also be emphasized that Python will remain a dynamically
typed language, and the authors have no desire to ever make type hints
mandatory, even by convention.


Type Definition Syntax
======================

The syntax leverages PEP 3107-style annotations with a number of
extensions described in sections below.  In its basic form, type hinting
is used by filling function annotations with classes::

  def greeting(name: str) -> str:
      return 'Hello ' + name

This denotes that the expected type of the ``name`` argument is ``str``.
Analogically, the expected return type is ``str``.  Subclasses of
a specified argument type are also accepted as valid types for that
argument.

Abstract base classes, types available in the ``types`` module, and
user-defined classes may be used as type hints as well.  Annotations
must be valid expressions that evaluate without raising exceptions at
the time the function is defined.  In addition, the needs of static
analysis require that annotations must be simple enough to be
interpreted by static analysis tools.  (This is an intentionally
somewhat vague requirement.)

.. FIXME: Define rigorously what is/isn't supported.

When used as an annotation, the expression ``None`` is considered
equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting
purposes.

Type aliases are also valid type hints::

  integer = int

  def retry(url: str, retry_count: integer) -> None: ...

New names that are added to support features described in following
sections are available in the ``typing`` package.


Callbacks
---------

Frameworks expecting callback functions of specific signatures might be
type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``.
Examples::

  from typing import Callable

  def feeder(get_next_item: Callable[[], str]) -> None:
      # Body

  def async_query(on_success: Callable[[int], None],
                  on_error: Callable[[int, Exception], None]) -> None:
      # Body

It is possible to declare the return type of a callable without
specifying the call signature by substituting a literal ellipsis
(three dots) for the list of arguments::

  def partial(func: Callable[..., str], *args) -> Callable[..., str]:
      # Body

Note that there are no square brackets around the ellipsis.  The
arguments of the callback are completely unconstrained in this case
(and keyword arguments are acceptable).

Since using callbacks with keyword arguments is not perceived as a
common use case, there is currently no support for specifying keyword
arguments with ``Callable``.  Similarly, there is no support for
specifying callback signatures with a variable number of argument of a
specific type.


Generics
--------

Since type information about objects kept in containers cannot be
statically inferred in a generic way, abstract base classes have been
extended to support subscription to denote expected types for container
elements.  Example::

  from typing import Mapping, Set

  def notify_by_email(employees: Set[Employee], overrides: Mapping[str,
str]) -> None: ...

Generics can be parametrized by using a new factory available in
``typing`` called ``TypeVar``.  Example::

  from typing import Sequence, TypeVar

  T = TypeVar('T')      # Declare type variable

  def first(l: Sequence[T]) -> T:   # Generic function
      return l[0]

In this case the contract is that the returning value is consistent with
the elements held by the collection.

``TypeVar`` supports constraining parametric types to classes with any of
the specified bases.  Example::

  from typing import Iterable

  X = TypeVar('X')
  Y = TypeVar('Y', Iterable[X])

  def filter(rule: Callable[[X], bool], input: Y) -> Y:
      ...

.. FIXME: Add an example with multiple bases defined.

In the example above we specify that ``Y`` can be any subclass of
Iterable with elements of type ``X``, as long as the return type of
``filter()`` will be the same as the type of the ``input``
argument.

.. FIXME: Explain more about how this works.


Forward references
------------------

When a type hint contains names that have not been defined yet, that
definition may be expressed as a string, to be resolved later.  For
example, instead of writing::

  def notify_by_email(employees: Set[Employee]) -> None: ...

one might write::

  def notify_by_email(employees: 'Set[Employee]') -> None: ...

.. FIXME: Rigorously define this, and give a motivational example.


Union types
-----------

Since accepting a small, limited set of expected types for a single
argument is common, there is a new special factory called ``Union``.
Example::

  from typing import Union

  def handle_employees(e: Union[Employee, Sequence[Employee]]) -> None:
      if isinstance(e, Employee):
          e = [e]
      ...

A type factored by ``Union[T1, T2, ...]`` responds ``True`` to
``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and
any of its subclasses, and so on.

One common case of union types are *optional* types.  By default,
``None`` is an invalid value for any type, unless a default value of
``None`` has been provided in the function definition.  Examples::

  def handle_employee(e: Union[Employee, None]) -> None: ...

As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;
for example, the above is equivalent to::

  from typing import Optional

  def handle_employee(e: Optional[Employee]) -> None: ...

An optional type is also automatically assumed when the default value is
``None``, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None) -> None: ...

The ``Any`` type
----------------

A special kind of type is ``Any``.  Every class is a subclass of
``Any``.  This is also true for the builtin class ``object``.
However, to the static type checker these are completely different.

When the type of a value is ``object``, the type checker will reject
almost all operations on it, and assigning it to a variable (or using
it as a return value) of a more specialized type is a type error.  On
the other hand, when a value has type ``Any``, the type checker will
allow all operations on it, and a value of type `Any`` can be assigned
to a variable (or used as a return value) of a more constrained type.


Platform-specific type checking
-------------------------------

In some cases the typing information will depend on the platform that
the program is being executed on.  To enable specifying those
differences, simple conditionals can be used::

  from typing import PY2, WINDOWS

  if PY2:
      text = unicode
  else:
      text = str

  def f() -> text: ...

  if WINDOWS:
      loop = ProactorEventLoop
  else:
      loop = UnixSelectorEventLoop

.. FIXME: Also define PY3 and POSIX?

Arbitrary literals defined in the form of ``NAME = True`` will also be
accepted by the type checker to differentiate type resolution::

  DEBUG = False
  ...
  if DEBUG:
      class Tracer:
          <verbose implementation>
  else:
      class Tracer:
          <dummy implementation>

For the purposes of type hinting, the type checker assumes ``__debug__``
is set to ``True``, in other words the ``-O`` command-line option is not
used while type checking.


Compatibility with other uses of function annotations
-----------------------------------------------------

A number of existing or potential use cases for function annotations
exist, which are incompatible with type hinting.  These may confuse a
static type checker.  However, since type hinting annotations have no
runtime behavior (other than evaluation of the annotation expression
and storing annotations in the ``__annotations__`` attribute of the
function object), this does not make the program incorrect -- it just
makes it issue warnings when a static analyzer is used.

To mark portions of the program that should not be covered by type
hinting, use the following:

* a ``@no_type_check`` decorator on classes and functions

* a ``# type: ignore`` comment on arbitrary lines

.. FIXME: should we have a module-wide comment as well?

.. FIXME: suggest that other uses of annotations be replaced with decorators

.. FIXME: add reference to "rejected alternatives"


Type Hints on Local and Global Variables
========================================

No first-class syntax support for explicitly marking variables as being
of a specific type is added by this PEP.  To help with type inference in
complex cases, a comment of the following format may be used::

  x = []   # type: List[Employee]

In the case where type information for a local variable is needed before
it is declared, an ``Undefined`` placeholder might be used::

  from typing import Undefined

  x = Undefined   # type: List[Employee]
  y = Undefined(int)

If type hinting proves useful in general, a syntax for typing variables
may be provided in a future Python version.

Casts
=====

Occasionally the type checker may need a different kind of hint: the
programmer may know that an expression is of a more constrained type
than the type checker infers.  For example::

  from typing import List

  def find_first_str(a: List[object]) -> str:
      index = next(i for i, x in enumerate(a) if isinstance(x, str))
      # We only get here if there's at least one string in a
      return cast(str, a[index])

The type checker infers the type ``object`` for ``a[index]``, but we
know that (if the code gets to that point) it must be a string.  The
``cast(t, x)`` call tells the type checker that we are confident that
the type of ``x`` is ``t``.  At runtime a cast always returns the
expression unchanged -- it does not check the type, and it does not
convert or coerce the value.

Casts differ from type comments (see the previous section).  When
using a type comment, the type checker should still verify that the
inferred type is consistent with the stated type.  When using a cast,
the type checker trusts the programmer.  Also, casts can be used in
expressions, while type comments only apply to assignments.


Stub Files
==========

Stub files are files containing type hints that are only for use by
the type checker, not at runtime.  There are several use cases for
stub files:

* Extension modules

* 3rd party modules whose authors have not yet added type hints

* Standard library modules for which type hints have not yet been written

* Modules that must be compatible with Python 2 and 3

* Modules that use annotations for other purposes

Stub files have the same syntax as regular Python modules.  There is
one feature of the ``typing`` module that may only be used in stub
files: the ``@overload`` decorator described below.

The type checker should only check function signatures in stub files;
function bodies in stub files should just be a single ``pass`` statement.

The type checker should have a configurable search path for stub
files.  If a stub file is found the type checker should not read the
corresponding "real" module.

Stub files may use the ``.py`` extension or alternatively may use the
``.pyi`` extension.  The latter makes it possible to maintain stub
files in the same directory as the corresponding real module.

Function overloading
--------------------

The ``@overload`` decorator allows describing functions that support
multiple different combinations of argument types.  This pattern is
used frequently in builtin modules and types.  For example, the
``__getitem__()`` method of the ``bytes`` type can be described as
follows::

  from typing import overload

  class bytes:
    ...
    @overload
    def __getitem__(self, i: int) -> int: pass
    @overload
    def __getitem__(self, s: slice) -> bytes: pass

This description is more precise than would be possible using unions
(which cannot express the relationship between the argument and return
types)::

  from typing import Union
  class bytes:
    ...
    def __getitem__(self, a: Union[int, slice]) -> Union[int, bytes]: pass

Another example where ``@overload`` comes in handy is the type of the
builtin ``map()`` function, which takes a different number of
arguments depending on the type of the callable::

  from typing import Callable, Iterable, Iterator, Tuple, TypeVar, overload

  T1 = TypeVar('T1')
  T2 = TypeVar('T2)
  S = TypeVar('S')

  @overload
  def map(func: Callable[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass
  @overload
  def map(func: Callable[[T1, T2], S],
          iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterator[S]: pass
  # ... and we could add more items to support more than two iterables

Note that we could also easily add items to support ``map(None, ...)``::

  @overload
  def map(func: None, iter1: Iterable[T1]) -> Iterable[T1]: pass
  @overload
  def map(func: None,
          iter1: Iterable[T1],
          iter2: Iterable[T2]) -> Iterable[Tuple[T1, T2]]: pass

The ``@overload`` decorator may only be used in stub files.  While it
would be possible to provide a multiple dispatch implementation using
this syntax, its implementation would require using
``sys._getframe()``, which is frowned upon.  Also, designing and
implementing an efficient multiple dispatch mechanism is hard, which
is why previous attempts were abandoned in favor of
``functools.singledispatch()``.  (See PEP 443, especially its section
"Alternative approaches".)  In the future we may come up with a
satisfactory multiple dispatch design, but we don't want such a design
to be constrained by the overloading syntax defined for type hints in
stub files.


Exceptions
==========

No syntax for listing explicitly raised exceptions is proposed.
Currently the only known use case for this feature is documentational,
in which case the recommendation is to put this information in a
docstring.


The ``typing`` Package
======================

To open the usage of static type checking to Python 3.5 as well as older
versions, a uniform namespace is required.  For this purpose, a new
package in the standard library is introduced called ``typing``.  It
holds a set of classes representing builtin types with generics, namely:

* Dict, used as ``Dict[key_type, value_type]``

* List, used as ``List[element_type]``

* Set, used as ``Set[element_type]``. See remark for ``AbstractSet``
  below.

* FrozenSet, used as ``FrozenSet[element_type]``

* Tuple, used by listing the element types, for example
  ``Tuple[int, int, str]``.
  Arbitrary-length homogeneous tuples can be expressed
  using one type and ellipsis, for example ``Tuple[int, ...]``.
  (The ``...`` here are part of the syntax.)

The generic versions of concrete collection types (``Dict``, ``List``,
``Set``, ``FrozenSet``, and homogeneous arbitrary-length ``Tuple``)
are mainly useful for annotating return values.  For arguments, prefer
the abstract collection types defined below, e.g.  ``Mapping``,
``Sequence`` or ``AbstractSet``.

It also introduces factories and helper members needed to express
generics and union types:

* Any, used as ``def get(key: str) -> Any: ...``

* Union, used as ``Union[Type1, Type2, Type3]``

* TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply
  ``Y = TypeVar('Y')``

* Undefined, used as ``local_variable = Undefined # type: List[int]`` or
  ``local_variable = Undefined(List[int])`` (the latter being slower
  during runtime)

* Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]``

* AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)``

All abstract base classes available in ``collections.abc`` are
importable from the ``typing`` package, with added generics support:

* ByteString

* Callable

* Container

* Hashable

* ItemsView

* Iterable

* Iterator

* KeysView

* Mapping

* MappingView

* MutableMapping

* MutableSequence

* MutableSet

* Sequence

* Set as ``AbstractSet``. This name change was required because ``Set``
  in the ``typing`` module means ``set()`` with generics.

* Sized

* ValuesView

* Mapping

The library includes literals for platform-specific type hinting:

* PY2

* PY3, equivalent to ``not PY2``

* WINDOWS

* POSIX, equivalent to ``not WINDOWS``

The following types are available in the ``typing.io`` module:

* IO

* BinaryIO

* TextIO

The following types are provided by the ``typing.re`` module:

* Match and Pattern, types of ``re.match()`` and ``re.compile()``
  results

As a convenience measure, types from ``typing.io`` and ``typing.re`` are
also available in ``typing`` (quoting Guido, "There's a reason those
modules have two-letter names.").


The place of the ``typing`` module in the standard library
----------------------------------------------------------

.. FIXME: complete this section (or discard?)


Usage Patterns
==============

The main use case of type hinting is static analysis using an external
tool without executing the analyzed program.  Existing tools used for
that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_
might be extended to support type checking.  New tools, like mypy [mypy]_,
can be adopted specifically for this purpose.

Type checking based on type hints is understood as a best-effort
mechanism.  In other words, whenever types are not annotated and cannot
be inferred, the type checker considers such code valid.  Type errors
are only reported in case of explicit or inferred conflict.  Moreover,
as a mechanism that is not tied to execution of the code, it does not
affect runtime behaviour.  In other words, even in the case of a typing
error, the program will continue running.

The implementation of a type checker, whether linting source files or
enforcing type information during runtime, is out of scope for this PEP.

.. FIXME: This is somewhat redundant with the updated initial sections.

.. FIXME: Describe run-time behavior of generic types.


Rejected Alternatives
=====================

During discussion of earlier drafts of this PEP, various objections
were raised and alternatives were proposed.  We discuss some of these
here and explain why we reject them.

Several main objections were raised.

Which brackets for generic type parameters?
-------------------------------------------

Most people are familiar with the use of angular brackets
(e.g. ``List<int>``) in languages like C++, Java, C# and Swift to
express the parametrization of generic types.  The problem with these
is that they are really hard to parse, especially for a simple-minded
parser like Python.  In most languages the ambiguities are usually
dealy with by only allowing angular brackets in specific syntactic
positions, where general expressions aren't allowed.  (And also by
using very powerful parsing techniques that can backtrack over an
arbitrary section of code.)

But in Python, we'd like type expressions to be (syntactically) the
same as other expressions, so that we can use e.g. variable assignment
to create type aliases.  Consider this simple type expression::

    List<int>

>From the Python parser's perspective, the expression begins with the
same four tokens (NAME, LESS, NAME, GREATER) as a chained comparison::

    a < b > c  # I.e., (a < b) and (b > c)

We can even make up an example that could be parsed both ways::

    a < b > [ c ]

Assuming we had angular brackets in the language, this could be
interpreted as either of the following two::

    (a<b>)[c]      # I.e., (a<b>).__getitem__(c)
    a < b > ([c])  # I.e., (a < b) and (b > [c])

It would surely be possible to come up with a rule to disambiguate
such cases, but to most users the rules would feel arbitrary and
complex.  It would also require us to dramatically change the CPython
parser (and every other parser for Python).  It should be noted that
Python's current parser is intentionally "dumb" -- a simple grammar is
easier for users to reason about.

For all these reasons, square brackets (e.g. ``List[int]``) are (and
have long been) the preferred syntax for generic type parameters.
They can be implemented by defining the ``__getitem__()`` method on
the metaclass, and no new syntax is required at all.  This option
works in all recent versions of Python (starting with Python 2.2).
Python is not alone in this syntactic choice -- generic classes in
Scala also use square brackets.

What about existing uses of annotations?
----------------------------------------

One line of argument points out that PEP 3107 explicitly supports
the use of arbitrary expressions in function annotations.  The new
proposal is then considered incompatible with the specification of PEP
3107.

Our response to this is that, first of all, the current proposal does
not introduce any direct incompatibilities, so programs using
annotations in Python 3.4 will still work correctly and without
prejudice in Python 3.5.

We do hope that type hints will eventually become the sole use for
annotations, but this will require additional discussion and a
deprecation period after the initial roll-out of the typing module
with Python 3.5.  The current PEP will have provisional status (see
PEP 411) until Python 3.6 is released.  The fastest conceivable scheme
would introduce silent deprecation of non-type-hint annotations in
3.6, full deprecation in 3.7, and declare type hints as the only
allowed use of annotations in Python 3.8.  This should give authors of
packages that use annotations plenty of time to devise another
approach, even if type hints become an overnight success.

Another possible outcome would be that type hints will eventually
become the default meaning for annotations, but that there will always
remain an option to disable them.  For this purpose the current
proposal defines a decorator ``@no_type_check`` which disables the
default interpretation of annotations as type hints in a given class
or function.  It also defines a meta-decorator
``@no_type_check_decorator`` which can be used to decorate a decorator
(!), causing annotations in any function or class decorated with the
latter to be ignored by the type checker.

There are also ``# type: ignore`` comments, and static checkers should
support configuration options to disable type checking in selected
packages.

Despite all these options, proposals have been circulated to allow
type hints and other forms of annotations to coexist for individual
arguments.  One proposal suggests that if an annotation for a given
argument is a dictionary literal, each key represents a different form
of annotation, and the key ``'type'`` would be use for type hints.
The problem with this idea and its variants is that the notation
becomes very "noisy" and hard to read.  Also, in most cases where
existing libraries use annotations, there would be little need to
combine them with type hints.  So the simpler approach of selectively
disabling type hints appears sufficient.

The problem of forward declarations
-----------------------------------

The current proposal is admittedly sub-optimal when type hints must
contain forward references.  Python requires all names to be defined
by the time they are used.  Apart from circular imports this is rarely
a problem: "use" here means "look up at runtime", and with most
"forward" references there is no problem in ensuring that a name is
defined before the function using it is called.

The problem with type hints is that annotations (per PEP 3107, and
similar to default values) are evaluated at the time a function is
defined, and thus any names used in an annotation must be already
defined when the function is being defined.  A common scenario is a
class definition whose methods need to reference the class itself in
their annotations.  (More general, it can also occur with mutually
recursive classes.)  This is natural for container types, for
example::

  class Node:
      """Binary tree node."""

      def __init__(self, left: Node, right: None):
          self.left = left
          self.right = right

As written this will not work, because of the peculiarity in Python
that class names become defined once the entire body of the class has
been executed.  Our solution, which isn't particularly elegant, but
gets the job done, is to allow using string literals in annotations.
Most of the time you won't have to use this though -- most _uses_ of
type hints are expected to reference builtin types or types defined in
other modules.

A counterproposal would change the semantics of type hints so they
aren't evaluated at runtime at all (after all, type checking happens
off-line, so why would type hints need to be evaluated at runtime at
all).  This of course would run afoul of backwards compatibility,
since the Python interpreter doesn't actually know whether a
particular annotation is meant to be a type hint or something else.

The double colon
----------------

A few creative souls have tried to invent solutions for this problem.
For example, it was proposed to use a double colon (``::``) for type
hints, solving two problems at once: disambiguating between type hints
and other annotations, and changing the semantics to preclude runtime
evaluation.  There are several things wrong with this idea, however.

* It's ugly.  The single colon in Python has many uses, and all of
  them look familiar because they resemble the use of the colon in
  English text.  This is a general rule of thumb by which Python
  abides for most forms of punctuation; the exceptions are typically
  well known from other programming languages.  But this use of ``::``
  is unheard of in English, and in other languages (e.g. C++) it is
  used as a scoping operator, which is a very different beast.  In
  contrast, the single colon for type hints reads natural -- and no
  wonder, since it was carefully designed for this purpose (the idea
  long predates PEP 3107 [gvr-artima]_).  It is also used in the same
  fashion in other languages from Pascal to Swift.

* What would you do for return type annotations?

* It's actually a feature that type hints are evaluated at runtime.

  * Making type hints available at runtime allows runtime type
    checkers to be built on top of type hints.

  * It catches mistakes even when the type checker is not run.  Since
    it is a separate program, users may choose not to run it (or even
    install it), but might still want to use type hints as a concise
    form of documentation.  Broken type hints are no use even for
    documentation.

* Because it's new syntax, using the double colon for type hints would
  limit them to code that works with Python 3.5 only.  By using
  existing syntax, the current proposal can easily work for older
  versions of Python 3.  (And in fact mypy supports Python 3.2 and
  newer.)

* If type hints become successful we may well decide to add new syntax
  in the future to declare the type for variables, for example
  ``var age: int = 42``.  If we were to use a double colon for
  argument type hints, for consistency we'd have to use the same
  convention for future syntax, perpetuating the ugliness.

Other forms of new syntax
-------------------------

A few other forms of alternative syntax have been proposed, e.g. the
introduction of a ``where`` keyword [roberge]_, and Cobra-inspired
``requires`` clauses.  But these all share a problem with the double
colon: they won't work for earlier versions of Python 3.  The same
would apply to a new ``__future__`` import.

Other backwards compatible conventions
--------------------------------------

The ideas put forward include:

* A decorator, e.g. ``@typehints(name=str, returns=str)``.  This could
  work, but it's pretty verbose (an extra line, and the argument names
  must be repeated), and a far cry in elegance from the PEP 3107
  notation.

* Stub files.  We do want stub files, but they are primarily useful
  for adding type hints to existing code that doesn't lend itself to
  adding type hints, e.g. 3rd party packages, code that needs to
  support both Python 2 and Python 3, and especially extension
  modules.  For most situations, having the annotations in line with
  the function definitions makes them much more useful.

* Docstrings.  There is an existing convention for docstrings, based
  on the Sphinx notation (``:type arg1: description``).  This is
  pretty verbose (an extra line per parameter), and not very elegant.
  We could also make up something new, but the annotation syntax is
  hard to beat (because it was designed for this very purpose).

It's also been proposed to simply wait another release.  But what
problem would that solve?  It would just be procrastination.


Is Type Hinting Pythonic?
=========================

.. FIXME: Do we really need this section?

Type annotations provide important documentation for how a unit of code
should be used.  Programmers should therefore provide type hints on
public APIs, namely argument and return types on functions and methods
considered public.  However, because types of local and global variables
can be often inferred, they are rarely necessary.

The kind of information that type hints hold has always been possible to
achieve by means of docstrings.  In fact, a number of formalized
mini-languages for describing accepted arguments have evolved.  Moving
this information to the function declaration makes it more visible and
easier to access both at runtime and by static analysis.  Adding to that
the notion that ?explicit is better than implicit?, type hints are
indeed *Pythonic*.


Acknowledgements
================

This document could not be completed without valuable input,
encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson
Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski.

Influences include existing languages, libraries and frameworks
mentioned in PEP 482.  Many thanks to their creators, in alphabetical
order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings,
Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer,
Raoul-Gabriel Urma, and Julien Verlaguet.


References
==========

.. [mypy]
   http://mypy-lang.org

.. [pyflakes]
   https://github.com/pyflakes/pyflakes/

.. [pylint]
   http://www.pylint.org

.. [gvr-artima]
   http://www.artima.com/weblogs/viewpost.jsp?thread=85551

.. [roberge]
   http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150320/43edaa22/attachment-0001.html>

From bcannon at gmail.com  Sat Mar 21 14:57:56 2015
From: bcannon at gmail.com (Brett Cannon)
Date: Sat, 21 Mar 2015 13:57:56 +0000
Subject: [Python-ideas] Warn about comparing bytes to int for `python3
	-b`
In-Reply-To: <CAP1=2W4=ZO1Ouf7GwCfgTTzkONe1ULniW70rq2axi4k_=AF1ag@mail.gmail.com>
References: <CAP1=2W6kDKzB5CQYo36S7TD9PYpdStE2RALHD-5wu8JHWwvU2w@mail.gmail.com>
 <CAMpsgwbW0NaZMpeSxJ4VibRAQ=R04RNrzXKSMY-31198PYWvkA@mail.gmail.com>
 <CAP1=2W4=ZO1Ouf7GwCfgTTzkONe1ULniW70rq2axi4k_=AF1ag@mail.gmail.com>
Message-ID: <CAP1=2W6GvSm15NGOC9p1Moi+Dr=HsjHWWck5uO59q4q_f+MvMQ@mail.gmail.com>

Just to update this thread, Serhiy beat me to a patch and committed it.
Through the change he found a few latent bugs in the stdlib itself so this
idea has already proven/paid for itself.

On Mon, Mar 16, 2015 at 1:25 PM Brett Cannon <bcannon at gmail.com> wrote:

> On Mon, Mar 16, 2015 at 11:22 AM Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Hi,
>>
>> 2015-03-16 16:11 GMT+01:00 Brett Cannon <bcannon at gmail.com>:
>> > One of the rather subtle issues with writing Python 2/3 code is that
>> > indexing on bytes in Python 2 returns a length-1 bytes object while in
>> > Python 3 it returns an int. Because ==/!= always returns True/False it
>> can
>> > be a very subtle failure and tough to track down.
>>
>> I worked on such patch in the past, but I lost it :-) I can help to
>> rewrite it if needed.
>>
>
> I filed http://bugs.python.org/issue23681 if you want to help (although I
> don't expect the patch to be complicated; I might ask you for a code review
> though =).
>
>
>>
>> So yes, it *is* very useful to port a large Python 2 project to Python 3.
>>
>> For example, you may not be able to run your application with Python 3
>> because a third party library cannot be imported on Python 3, so it
>> blocks the whole work on porting an application to Python 3. Until the
>> module is ported, you may want to prepare the port. Checking
>> bytes==str just by reading the source code is difficult.
>>
>> Other issues which can only be "seen" at runtime when running an
>> application on Python 3 :
>>
>> - "x > 0" with x=None => TypeError is raised in Python 3
>>
>
> Testing can easily catch that *and* have a traceback to trace down where
> the problem originated from, so I'm not worried.
>
>
>> - x / 8 where x is an int => becomes a float in Python 3, it's hard to
>> detect this issue in Python 2 just by reading the source code :-/
>>
>
> -Q in Python 2 and/or adding the division __future__ statement solves this
> one.
>
> Basically I'm just trying to plug the last few holes that can't be
> automated by tools like Modernize or Futurize *and* won't surface where
> the problem is easily during testing under Python 2 or 3 (e.g., a traceback
> during testing makes it easy to find unless you swallow the exception, in
> which case a warning won't help you either if you use -Werror).
>
>
>
>>
>> > What do people think of extending -b/-bb in Python 3 to warn when
>> performing
>> > equality between an int and a bytes object of any length? I don't want
>> to
>> > restrict to length-1 bytes objects because people may be doing
>> comparisons
>> > where the result can be length-1 or any other length and thus would
>> still
>> > have a subtle bug to pick up. Do people think this would raise a ton of
>> > false-positives? Would people find it useful?
>>
>> First ensure that the stdlib doesn't raise any BytesWarning exception.
>>
>
> Yep, I always test with -Werror whenever I muck with something that can
> raise a warning (side-effect from helping to write _warnings.c).
>
> -Brett
>
>
>>
>> For example, os.get_exec_path() has to modify warnings filters
>> temporary in Python 3 :-/
>>
>>     # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a
>>     # BytesWarning when using python -b or python -bb: ignore the warning
>>
>> Victor
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150321/18339a03/attachment.html>

From steve at pearwood.info  Sat Mar 21 16:37:32 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 22 Mar 2015 02:37:32 +1100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAP7+vJL_XR7e+FUMYWYVhsy+ZsNg6xiOg8VVoew3_JBh86b2Rw@mail.gmail.com>
References: <CAP7+vJL_XR7e+FUMYWYVhsy+ZsNg6xiOg8VVoew3_JBh86b2Rw@mail.gmail.com>
Message-ID: <20150321153731.GN31862@ando.pearwood.info>

On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote:

> Union types
> -----------
[...]
> As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;

That only saves three characters. Is it worth it?


> An optional type is also automatically assumed when the default value is
> ``None``, for example::
> 
>   def handle_employee(e: Employee = None): ...

Should that apply to all default values or just None? E.g. if I have

    def spam(s: str = 23): ... 

should that be inferred as Union[str, int] or be flagged as a type 
error? I think that we want a type error here, and it's only None that 
actually should be treated as special. Perhaps that should be made 
explicit in the PEP.


[...]
> For the purposes of type hinting, the type checker assumes ``__debug__``
> is set to ``True``, in other words the ``-O`` command-line option is not
> used while type checking.

I'm afraid I don't understand what you are trying to say here. I would 
have expected that __debug__ and -O and the type checker would be 
independent of each other.


[...]
> To mark portions of the program that should not be covered by type
> hinting, use the following:
> 
> * a ``@no_type_check`` decorator on classes and functions
> 
> * a ``# type: ignore`` comment on arbitrary lines
> 
> .. FIXME: should we have a module-wide comment as well?

I think so, if for no other reason than it will reduce the fear of some 
people that type checks will be mandatory.



> Type Hints on Local and Global Variables
> ========================================
> 
> No first-class syntax support for explicitly marking variables as being
> of a specific type is added by this PEP.  To help with type inference in
> complex cases, a comment of the following format may be used::
> 
>   x = []   # type: List[Employee]
> 
> In the case where type information for a local variable is needed before
> it is declared, an ``Undefined`` placeholder might be used::
> 
>   from typing import Undefined
> 
>   x = Undefined   # type: List[Employee]
>   y = Undefined(int)

How is that better than just bringing forward the variable declaration?

    x = []   # type: List[Employee]
    y = 0



> Casts
> =====
> 
> Occasionally the type checker may need a different kind of hint: the
> programmer may know that an expression is of a more constrained type
> than the type checker infers.  For example::
>
>   from typing import List
> 
>   def find_first_str(a: List[object]) -> str:
>       index = next(i for i, x in enumerate(a) if isinstance(x, str))
>       # We only get here if there's at least one string in a
>       return cast(str, a[index])
> 
> The type checker infers the type ``object`` for ``a[index]``, but we
> know that (if the code gets to that point) it must be a string.  The
> ``cast(t, x)`` call tells the type checker that we are confident that
> the type of ``x`` is ``t``.

Is the type checker supposed to unconditionally believe the cast, or 
only if the cast is more constrained than the infered type (like str and 
object, or bool and int)?

E.g. if the type checker infers int, and the cast says list, I'm not 
entirely sure I would trust the programmer more than the type checker.

My feeling here is that some type checkers will unconditionally trust 
the cast, and some will flag the mismatch, or offer a config option to 
swap between the two, and that will be a feature for type checkers to 
compete on.


I'm also going to bike-shed the order of arguments. It seems to me that 
we want to say:

    cast(x, T)  # pronounced "cast x to T"

rather than Yoda-speak "cast T x to we shall" *wink*. That also matches 
the order of isinstance(obj, type) calls and makes it easier to 
remember.


> At runtime a cast always returns the
> expression unchanged -- it does not check the type, and it does not
> convert or coerce the value.

I'm a little concerned about cast() being a function. I know that it's a 
do-nothing function, but there's still the overhead of the name lookup 
and function call. It saddens me that giving a hint to the type checker 
has a runtime cost, small as it is.

(I know that *technically* annotations have a runtime cost too, but 
they're once-only, at function definition time, not every time you call 
the function.)

Your point below that cast() can be used inside expressions is a valid 
point, so there has to be a cast() function to support those cases, but 
for the example given here where the cast occurs in a return statement, 
wouldn't a type comment do?

    return some_expression  # type: T

hints that some_expression is to be treated as type T, regardless of 
what was infered.


> Casts differ from type comments (see the previous section).  When
> using a type comment, the type checker should still verify that the
> inferred type is consistent with the stated type.  When using a cast,
> the type checker trusts the programmer.  Also, casts can be used in
> expressions, while type comments only apply to assignments.


> Stub Files
> ==========
[...]
> Stub files may use the ``.py`` extension or alternatively may use the
> ``.pyi`` extension.  The latter makes it possible to maintain stub
> files in the same directory as the corresponding real module.

I don't like anything that could cause confusion between stub files and 
actual Python files. If we allow .py extension on stub files, I'm sure 
there will be confusing errors where people somehow manage to get the 
stub file imported instead of the actual module they want.

Is there any advantage to allowing stub files use a .py extension? If 
not, then don't allow it.


-- 
Steve

From luciano at ramalho.org  Sat Mar 21 19:30:27 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Sat, 21 Mar 2015 15:30:27 -0300
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150321153731.GN31862@ando.pearwood.info>
References: <CAP7+vJL_XR7e+FUMYWYVhsy+ZsNg6xiOg8VVoew3_JBh86b2Rw@mail.gmail.com>
 <20150321153731.GN31862@ando.pearwood.info>
Message-ID: <CALxg4FXCxN7115fJKW5xrcybiWFttVgKEhbW4_XscyU641OfFw@mail.gmail.com>

On Saturday, March 21, 2015, Steven D'Aprano <steve at pearwood.info> wrote:

> On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote:
>
> > Union types
> > -----------
> [...]
> > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;
>
> That only saves three characters. Is it worth it?


I think it is worth it because will be a very common case, and the shortcut
is more readable.

cheers,

Luciano


>
>
> > An optional type is also automatically assumed when the default value is
> > ``None``, for example::
> >
> >   def handle_employee(e: Employee = None): ...
>
> Should that apply to all default values or just None? E.g. if I have
>
>     def spam(s: str = 23): ...
>
> should that be inferred as Union[str, int] or be flagged as a type
> error? I think that we want a type error here, and it's only None that
> actually should be treated as special. Perhaps that should be made
> explicit in the PEP.
>
>
> [...]
> > For the purposes of type hinting, the type checker assumes ``__debug__``
> > is set to ``True``, in other words the ``-O`` command-line option is not
> > used while type checking.
>
> I'm afraid I don't understand what you are trying to say here. I would
> have expected that __debug__ and -O and the type checker would be
> independent of each other.
>
>
> [...]
> > To mark portions of the program that should not be covered by type
> > hinting, use the following:
> >
> > * a ``@no_type_check`` decorator on classes and functions
> >
> > * a ``# type: ignore`` comment on arbitrary lines
> >
> > .. FIXME: should we have a module-wide comment as well?
>
> I think so, if for no other reason than it will reduce the fear of some
> people that type checks will be mandatory.
>
>
>
> > Type Hints on Local and Global Variables
> > ========================================
> >
> > No first-class syntax support for explicitly marking variables as being
> > of a specific type is added by this PEP.  To help with type inference in
> > complex cases, a comment of the following format may be used::
> >
> >   x = []   # type: List[Employee]
> >
> > In the case where type information for a local variable is needed before
> > it is declared, an ``Undefined`` placeholder might be used::
> >
> >   from typing import Undefined
> >
> >   x = Undefined   # type: List[Employee]
> >   y = Undefined(int)
>
> How is that better than just bringing forward the variable declaration?
>
>     x = []   # type: List[Employee]
>     y = 0
>
>
>
> > Casts
> > =====
> >
> > Occasionally the type checker may need a different kind of hint: the
> > programmer may know that an expression is of a more constrained type
> > than the type checker infers.  For example::
> >
> >   from typing import List
> >
> >   def find_first_str(a: List[object]) -> str:
> >       index = next(i for i, x in enumerate(a) if isinstance(x, str))
> >       # We only get here if there's at least one string in a
> >       return cast(str, a[index])
> >
> > The type checker infers the type ``object`` for ``a[index]``, but we
> > know that (if the code gets to that point) it must be a string.  The
> > ``cast(t, x)`` call tells the type checker that we are confident that
> > the type of ``x`` is ``t``.
>
> Is the type checker supposed to unconditionally believe the cast, or
> only if the cast is more constrained than the infered type (like str and
> object, or bool and int)?
>
> E.g. if the type checker infers int, and the cast says list, I'm not
> entirely sure I would trust the programmer more than the type checker.
>
> My feeling here is that some type checkers will unconditionally trust
> the cast, and some will flag the mismatch, or offer a config option to
> swap between the two, and that will be a feature for type checkers to
> compete on.
>
>
> I'm also going to bike-shed the order of arguments. It seems to me that
> we want to say:
>
>     cast(x, T)  # pronounced "cast x to T"
>
> rather than Yoda-speak "cast T x to we shall" *wink*. That also matches
> the order of isinstance(obj, type) calls and makes it easier to
> remember.
>
>
> > At runtime a cast always returns the
> > expression unchanged -- it does not check the type, and it does not
> > convert or coerce the value.
>
> I'm a little concerned about cast() being a function. I know that it's a
> do-nothing function, but there's still the overhead of the name lookup
> and function call. It saddens me that giving a hint to the type checker
> has a runtime cost, small as it is.
>
> (I know that *technically* annotations have a runtime cost too, but
> they're once-only, at function definition time, not every time you call
> the function.)
>
> Your point below that cast() can be used inside expressions is a valid
> point, so there has to be a cast() function to support those cases, but
> for the example given here where the cast occurs in a return statement,
> wouldn't a type comment do?
>
>     return some_expression  # type: T
>
> hints that some_expression is to be treated as type T, regardless of
> what was infered.
>
>
> > Casts differ from type comments (see the previous section).  When
> > using a type comment, the type checker should still verify that the
> > inferred type is consistent with the stated type.  When using a cast,
> > the type checker trusts the programmer.  Also, casts can be used in
> > expressions, while type comments only apply to assignments.
>
>
> > Stub Files
> > ==========
> [...]
> > Stub files may use the ``.py`` extension or alternatively may use the
> > ``.pyi`` extension.  The latter makes it possible to maintain stub
> > files in the same directory as the corresponding real module.
>
> I don't like anything that could cause confusion between stub files and
> actual Python files. If we allow .py extension on stub files, I'm sure
> there will be confusing errors where people somehow manage to get the
> stub file imported instead of the actual module they want.
>
> Is there any advantage to allowing stub files use a .py extension? If
> not, then don't allow it.
>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150321/e1c6aa1d/attachment-0001.html>

From probiner at gmail.com  Sun Mar 22 07:38:07 2015
From: probiner at gmail.com (pedro santos)
Date: Sun, 22 Mar 2015 06:38:07 +0000
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing
	rules?
Message-ID: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>

Hi

I'm an Art and CG student learning Python and today's exercise was about
positions in a tiled room. The fact that I had to check if a position was
inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0
was considered out, it kept me thinking about 0-base indexing iterables and
slicing.

Read some articles and discussions, some pros and cons to each 0-base and
1-base, concerns about slicing, etc. But ultimately the question that got
stuck in me and didn't found an answer was:
Why can't both 0-base and 1-base indexing exist in the same language, and
why can't slicing be customized?



If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense
to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so
that accordingly rul[0]=0, rul[5]=5.
If I'm indexing the blue circles, natural number quantities, objects,
spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and
cir[5]=5.

Now, a lot of the discussion was to do with slicing coupled with the
indexing and I don't totally understand why.

a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up
to the the one before 3"), so on one side, you put the finger on what you
want and on the other, on what you don't want. But this method does have
the neat property of producing neighbor selections that border perfectly,
as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be
unexpected as it returns values off-by-one from its counterpart like;
L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's
consistent with the rule a ? x < b, grabbing the lower limit item, but it
can feel strange by not producing the same selection with inverse order.

a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"),
so you're putting the finger on the things you want. If you inverse the
order(-1) it's still very easy to grasp what are you picking because
whatever you select it's included like: L=[0,1,2,3,4,5] so that
L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when
trying to do neighbor selections, where one would have to do
[:a][a+1:b][b+1:c] to have the border perfectly. That terrible?

Even though one could see a ? x < b to be more adept to 0-base, and a ? x ?
b to be more adept to 1-base, the way I see it, index base and slicing
rules could be somehow independent. And one would index and slice the way
it would fit the rationale of the problem or the data, because even slicing
a 1-base indexed array with a ? x < b, would still produce an expected
outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2].
Same thing applying a ? x ? b to a 0-base indexed array, as in
rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2].

Given that python is an example of human friendly code language,
emphasizing readability, wouldn't having 0 and 1 base indexing and custom
slicing methods, improve the thought process when writing and reading the
code, by fitting them better to specific contexts or data?
Is there some language that provides both or each language picks only one?

Cheers

-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150322/3c668594/attachment.html>

From techtonik at gmail.com  Sat Mar 21 10:46:48 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 21 Mar 2015 12:46:48 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
Message-ID: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>

I think it should be in stdlib.

11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.

This is from Wikipedia.
-- 
anatoly t.

From luciano at ramalho.org  Sun Mar 22 08:24:12 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Sun, 22 Mar 2015 04:24:12 -0300
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
Message-ID: <CALxg4FXPWOXHvL+eo+YL4=zg7-f4Np2i-HYVAinXH0wqirRGYg@mail.gmail.com>

On Sun, Mar 22, 2015 at 3:38 AM, pedro santos <probiner at gmail.com> wrote:
> Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized?

Pedro,

The best answer ever is by one of the most important computer
scientists of the 20th century, Edsger W. Dijkstra -- who was also
Dutch, like Guido van Rossum:

https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF

The answer is also visually beautiful, as are all Dijkstra's
manuscripts. Some font designer should create a font based on
Dijkstra's handwriting.

Cheers,

Luciano


>
>
>
> If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5.
> If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5.
>
> Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why.
>
> a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order.
>
> a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible?
>
> Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2].
> Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2].
>
> Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data?
> Is there some language that provides both or each language picks only one?
>
> Cheers
>
>
> --
> ________________________________
>
> Pedro Alpiar?a dos Santos
> Animator  3DModeler  Illustrator
> >>  http://probiner.x10.mx/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/




-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From ben+python at benfinney.id.au  Sun Mar 22 08:54:28 2015
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 22 Mar 2015 18:54:28 +1100
Subject: [Python-ideas] =?utf-8?b?TyDigJNBUE9P4oCTIE9XVERJICh3YXM6IDAtYmFz?=
 =?utf-8?q?e_and_1-base_indexed_iterables=3F_Custom_slicing_rules=3F=29?=
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
 <CALxg4FXPWOXHvL+eo+YL4=zg7-f4Np2i-HYVAinXH0wqirRGYg@mail.gmail.com>
Message-ID: <85d24179hn.fsf_-_@benfinney.id.au>

Luciano Ramalho <luciano at ramalho.org>
writes:

> The best answer ever is by one of the most important computer
> scientists of the 20th century, Edsger W. Dijkstra -- who was also
> Dutch, like Guido van Rossum:
>
> https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF

Fascinatingly ? to me, at least ? Dijkstra's orthography for parenthetic
en dash is the opposite of that Tim Peters used in the thirteenth
aphorism of the Zen of Python.

Dijkstra, EWD831:

    Exclusion of the lower bound ?as in a) and b)? forces [?]

Peters, Zen of Python:

    There should be one? and preferably only one ?obvious way to do it.

How wonderful to see another dimension of combinations for that style
issue, in a Dijkstra paper talking about exploring the dimensions of
another style issue :-)

-- 
 \       ?When I get new information, I change my position. What, sir, |
  `\             do you do with new information?? ?John Maynard Keynes |
_o__)                                                                  |
Ben Finney


From rosuav at gmail.com  Sun Mar 22 11:11:38 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 22 Mar 2015 21:11:38 +1100
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
Message-ID: <CAPTjJmpxexJfY50U=kNydE5wLVzE5qkijjma3sCCoj5QKMWYHw@mail.gmail.com>

On Sat, Mar 21, 2015 at 8:46 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> I think it should be in stdlib.
>
> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>
> This is from Wikipedia.

Put some code together, stick it on PyPI... oh, and while you're
there, see if someone else has already written it. You never know, it
might well be there!

ChrisA

From liik.joonas at gmail.com  Sun Mar 22 15:15:39 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Sun, 22 Mar 2015 16:15:39 +0200
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CALxg4FXPWOXHvL+eo+YL4=zg7-f4Np2i-HYVAinXH0wqirRGYg@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
 <CALxg4FXPWOXHvL+eo+YL4=zg7-f4Np2i-HYVAinXH0wqirRGYg@mail.gmail.com>
Message-ID: <CAB1GNpQLGjOhiNSBP1fSz2yPRHJzqN6UmBUVzS_MVJ2z9j9k9Q@mail.gmail.com>

Hey

In one of my projects I'm stuck in an environment where there are multiple
co-existing coding environments, one of which has a convention of using 0
based indexing and the other using 1 based indexing. All i can say is that
is is a VERY (i cant possibly emphasize this enough) bad idea. it is an
unending source of confusion and off by 1 bugs.

It is like the line endings issue all over again ... CR or LF and then you
end up with CR, LF and CR LF, choosing 1 is definitely better than choosing
both. Python got this right, please do not inflict this upon the python
community!

I got real life experience with this and i'm telling you this is a disaster
waiting to happen, please don't..


On 22 March 2015 at 09:24, Luciano Ramalho <luciano at ramalho.org> wrote:

> On Sun, Mar 22, 2015 at 3:38 AM, pedro santos <probiner at gmail.com> wrote:
> > Why can't both 0-base and 1-base indexing exist in the same language,
> and why can't slicing be customized?
>
> Pedro,
>
> The best answer ever is by one of the most important computer
> scientists of the 20th century, Edsger W. Dijkstra -- who was also
> Dutch, like Guido van Rossum:
>
> https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF
>
> The answer is also visually beautiful, as are all Dijkstra's
> manuscripts. Some font designer should create a font based on
> Dijkstra's handwriting.
>
> Cheers,
>
> Luciano
>
>
> >
> >
> >
> > If I'm indexing the ruler marks, intervals, boundaries, dots, makes
> sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my
> ruler so that accordingly rul[0]=0, rul[5]=5.
> > If I'm indexing the blue circles, natural number quantities, objects,
> spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and
> cir[5]=5.
> >
> > Now, a lot of the discussion was to do with slicing coupled with the
> indexing and I don't totally understand why.
> >
> > a ? x < b is not so intuitive when dealing with objects ("I want balls 1
> up to the the one before 3"), so on one side, you put the finger on what
> you want and on the other, on what you don't want. But this method does
> have the neat property of producing neighbor selections that border
> perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results
> can be unexpected as it returns values off-by-one from its counterpart
> like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's
> consistent with the rule a ? x < b, grabbing the lower limit item, but it
> can feel strange by not producing the same selection with inverse order.
> >
> > a ? x ? b is a natural way to select objets ("I want the balls 1 up to
> 3"), so you're putting the finger on the things you want. If you inverse
> the order(-1) it's still very easy to grasp what are you picking because
> whatever you select it's included like: L=[0,1,2,3,4,5] so that
> L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when
> trying to do neighbor selections, where one would have to do
> [:a][a+1:b][b+1:c] to have the border perfectly. That terrible?
> >
> > Even though one could see a ? x < b to be more adept to 0-base, and a ?
> x ? b to be more adept to 1-base, the way I see it, index base and slicing
> rules could be somehow independent. And one would index and slice the way
> it would fit the rationale of the problem or the data, because even slicing
> a 1-base indexed array with a ? x < b, would still produce an expected
> outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2].
> > Same thing applying a ? x ? b to a 0-base indexed array, as in
> rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2].
> >
> > Given that python is an example of human friendly code language,
> emphasizing readability, wouldn't having 0 and 1 base indexing and custom
> slicing methods, improve the thought process when writing and reading the
> code, by fitting them better to specific contexts or data?
> > Is there some language that provides both or each language picks only
> one?
> >
> > Cheers
> >
> >
> > --
> > ________________________________
> >
> > Pedro Alpiar?a dos Santos
> > Animator  3DModeler  Illustrator
> > >>  http://probiner.x10.mx/
> >
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> |     http://shop.oreilly.com/product/0636920032519.do
> |  Professor em: http://python.pro.br
> |  Twitter: @ramalhoorg
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150322/715e714a/attachment.html>

From rosuav at gmail.com  Sun Mar 22 15:30:12 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 23 Mar 2015 01:30:12 +1100
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
Message-ID: <CAPTjJmre-mGrdEZY4zVPtvjZTqbLUr3dK3Pu3=n5bEpVkvBHmw@mail.gmail.com>

On Sun, Mar 22, 2015 at 5:38 PM, pedro santos <probiner at gmail.com> wrote:
>
> a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order.
>
> a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible?
>

To add to what other people have said in support of Python's indexing
style, I'd like to point out one situation where fully-closed ranges
are used: Scripture references. Have a look at this page:

https://www.biblegateway.com/passage/?search=Mark+3&version=NIV

This is one chapter from the Bible. Passages within that chapter can
be identified by starting and ending verse, inclusive at both ends.
So, for instance, the paragraph under the heading "Jesus Appoints the
Twelve" would be Mark 3:13-19. Note how the end point is actually one
less than the number that starts the next passage. Python identifies
the boundaries between elements, which corresponds perfectly to a
numbering system such as this; noting the passage as "Mark 3:13-20"
would mean you start reading at the indicator for verse 13, and stop
reading as soon as you reach the indicator for verse 20, which makes a
lot more sense.

Obviously it's way WAY too hard to change something as extensively
used as Bible references - the backward incompatibility concern is
just too strong - but this may give some indication of why it's more
sensible to work with half-open ranges. A half-open range also gives a
convenient notation for "to the end of the chapter": instead of "Mark
3:20-3:35", it would simply be "Mark 3:20-4:1". (Chapters and verses
are numbered from one, not zero, but in an ideal world, this would
also change, so it would be "3:19-4:0".)

ChrisA

From abarnert at yahoo.com  Sun Mar 22 15:52:07 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 22 Mar 2015 07:52:07 -0700
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
	slicing rules?
In-Reply-To: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
Message-ID: <E9970897-275D-4176-85F8-32B61F40F8DF@yahoo.com>

Your first example is 0-based even though it's closed (L[1:3]=[1,2,3]), which weakens your whole argument for 1-based indexing. and makes me think such a possibility might actually confuse you--which is exactly why we don't have it.

As you say, it may still make sense to have closed ranges and slices alongside half-open, even with 0-based indices. And many newer languages do that, often by using ".." vs. "...", which seems to be pretty readable. But there's no obvious way to add something similar to Python at this point. (Also, I think the fact that ranges and slices are different things with different syntax is a benefit of Python, not a deficit, and it's even harder to design a good solution without losing that benefit...)

Sent from my iPhone

> On Mar 21, 2015, at 11:38 PM, pedro santos <probiner at gmail.com> wrote:
> 
> Hi
> 
> I'm an Art and CG student learning Python and today's exercise was about positions in a tiled room. The fact that I had to check if a position was inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0 was considered out, it kept me thinking about 0-base indexing iterables and slicing.
> 
> Read some articles and discussions, some pros and cons to each 0-base and 1-base, concerns about slicing, etc. But ultimately the question that got stuck in me and didn't found an answer was:
> Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized?
> 
> 
> 
> 
> 
> 
> If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5.
> If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5.
> 
> Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why.
> 
> a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order.
> 
> a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible?
> 
> Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2].
> Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2].
> 
> Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data?
> Is there some language that provides both or each language picks only one?
> 
> Cheers
> 
> 
> -- 
> 
> Pedro Alpiar?a dos Santos 
> Animator  3DModeler  Illustrator 
> >>  http://probiner.x10.mx/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150322/7de1c6d0/attachment-0001.html>

From ron3200 at gmail.com  Sun Mar 22 19:50:16 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Sun, 22 Mar 2015 14:50:16 -0400
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
	slicing rules?
In-Reply-To: <CAPTjJmre-mGrdEZY4zVPtvjZTqbLUr3dK3Pu3=n5bEpVkvBHmw@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
 <CAPTjJmre-mGrdEZY4zVPtvjZTqbLUr3dK3Pu3=n5bEpVkvBHmw@mail.gmail.com>
Message-ID: <men2t9$fu1$1@ger.gmane.org>



On 03/22/2015 10:30 AM, Chris Angelico wrote:
> To add to what other people have said in support of Python's indexing
> style, I'd like to point out one situation where fully-closed ranges
> are used: Scripture references.

While indexing and slicing use natural numbers, the advantage of the half 
open interval becomes more obvious if you consider examples with real numbers.

When using a closed range interval with real numbers ...

     n1 <= i <= n2
     ? <= i <= n3

Picking the start value for the next adjacent sequence is not simply a 
matter of adding 1.

The half open sequence makes that case both trivial and elegant.

     n1 <= i < n2
     n2 <= i < n3

And it works just as well with natural numbers.  And works best if the end 
value is a multiple of the step value.


As to weather to use 0 or 1 indexing,  0 indexing is very useful for 
interfacing with the hardware.  0 is a valid memory address, and the 0th 
bit is the first bit of a byte.  Indexing at that level directly correspond 
to how the hardware is built.  For other higher level things, ones indexing 
can work equally well, (in my opinion), we just don't want to mix them.

     0 <= i < n        # n equals number of items

     1 <= i < n + 1    # Dijkstra prefers the first case.


But consider in programming, the indexes are frequently variables rather 
than literals.

     start <= i < start + count

This works in both cases.  In the above case of zero indexing it's really.

     0 <= i < 0 + n

It just happens that it can be simplified a bit in the case where there 
start value is zero.  And then the end index becomes the count, but only 
when the start index is zero.

When ones indexing is used, the zero index can be used to specify the end 
value when stepping towards zero from both directions.

When zero indexing is used. it allows for wrap around behaviour when the 
index crosses zero.  Sometimes that can be very nice.

A reason a language should choose one or the other, and not both, is that 
the choice defines how many other things interact with that choice.  Trying 
to do both can mean having two versions of many functions and statements.

Cheers,
    Ron






























From mertz at gnosis.cx  Sun Mar 22 20:52:40 2015
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 22 Mar 2015 12:52:40 -0700
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <men2t9$fu1$1@ger.gmane.org>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
 <CAPTjJmre-mGrdEZY4zVPtvjZTqbLUr3dK3Pu3=n5bEpVkvBHmw@mail.gmail.com>
 <men2t9$fu1$1@ger.gmane.org>
Message-ID: <CAEbHw4ZhQGq4Um2nLCbgLZ=TYxCFJOQ3P3xrdNQS_4=r2G+JzQ@mail.gmail.com>

The strongest proof that mixed index bases are a bad idea is that this is a
valid statement in VB6:

  Dim array(1 to 10) as Integer

For all X, if VB6 allows X, then X is a bad idea. :-)

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150322/e1fd40a1/attachment.html>

From tritium-list at sdamon.com  Sun Mar 22 22:15:39 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Sun, 22 Mar 2015 17:15:39 -0400
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
Message-ID: <550F30FB.5010000@sdamon.com>

On 3/21/2015 05:46, anatoly techtonik wrote:
> I think it should be in stdlib.
>
> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>
> This is from Wikipedia.
Why does this need to be in the standard library?

From solipsis at pitrou.net  Sun Mar 22 22:38:15 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Mar 2015 22:38:15 +0100
Subject: [Python-ideas] Geo coordinates conversion in stdlib
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com>
Message-ID: <20150322223815.06082280@fsol>

On Sun, 22 Mar 2015 17:15:39 -0400
Alexander Walters
<tritium-list at sdamon.com> wrote:
> On 3/21/2015 05:46, anatoly techtonik wrote:
> > I think it should be in stdlib.
> >
> > 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
> >
> > This is from Wikipedia.
> Why does this need to be in the standard library?

Because Anatoly needs it, of course :)

Regards

Antoine.



From abarnert at yahoo.com  Sun Mar 22 23:37:51 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 22 Mar 2015 15:37:51 -0700
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <20150322223815.06082280@fsol>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
Message-ID: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>

On Mar 22, 2015, at 2:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> On Sun, 22 Mar 2015 17:15:39 -0400
> Alexander Walters
> <tritium-list at sdamon.com> wrote:
>>> On 3/21/2015 05:46, anatoly techtonik wrote:
>>> I think it should be in stdlib.
>>> 
>>> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>>> 
>>> This is from Wikipedia.
>> Why does this need to be in the standard library?
> 
> Because Anatoly needs it, of course :)

Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems.

More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer. 

From chris.barker at noaa.gov  Mon Mar 23 00:13:57 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Sun, 22 Mar 2015 16:13:57 -0700
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
Message-ID: <CALGmxEJ0GBH7O9qPPkQ77ydqC33r14+0OnRY7b8xzS6sMq9K4Q@mail.gmail.com>

On Sun, Mar 22, 2015 at 3:37 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

>  If you can't figure out the complexity of multiplying and divmodding by
> 60 twice, you can borrow its code, or just wrap it and delegate. That seems
> trivial enough to do in any project that needs it.


I"m not suggesting this is an stdlib candiate, but it'snot quite that
simple. With FP precision issues you can end up with results like:

1? 60? 13?

which you don't want. I haven't checked to see if datetime deal siwththis
corectly but I'd guess it does :-)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150322/c88c492e/attachment-0001.html>

From davidfstr at gmail.com  Mon Mar 23 07:02:40 2015
From: davidfstr at gmail.com (David Foster)
Date: Sun, 22 Mar 2015 23:02:40 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <mailman.951.1426962635.10326.python-ideas@python.org>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
Message-ID: <550FAC80.8040405@gmail.com>

* It's a bit vague how local variables should be typed in the presence of
   multiple assignments. Consider:

   ```
   if x > 0:
       root = sqrt(x)  # type: int
   else:
       root = None
   ```

   It seems that the type of `root` should be inferred as `Union[int, None]`
   (AKA `Optional[int]`), but there's no discussion about this type of
   inference in the PEP.

   Cases like this are the only way that the proposed blank `Undefined`
   value makes sense to me, using an example like:

   ```
   root = Undefined    # type: Optional[int]
   if x > 0:
       root = sqrt(x)  # type: int
   else:
       root = None
   ```

* +1 for only allowing .pyi as the extension for stub files (and not 
also .py).
   Makes it very clear that they should only be interpreted as stubs.

* -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and
   directly corresponds conceptually to types like `int?` and `?int` from
   other languages.

* +1 for the argument order cast(x, T). This is consistent with (x: T) 
elsewhere.

--
David Foster
http://dafoster.net/

P.S. Apologies if this doesn't add to the message chain correctly. 
There's no clearly correct way to reply to a digest email from Mailman.

From techtonik at gmail.com  Mon Mar 23 07:08:23 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 23 Mar 2015 09:08:23 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
Message-ID: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>

On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> On Sun, 22 Mar 2015 17:15:39 -0400
>> Alexander Walters
>> <tritium-list at sdamon.com> wrote:
>>>> On 3/21/2015 05:46, anatoly techtonik wrote:
>>>> I think it should be in stdlib.
>>>>
>>>> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>>>>
>>>> This is from Wikipedia.
>>> Why does this need to be in the standard library?
>>
>> Because Anatoly needs it, of course :)
>
> Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems.
>
> More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer.

That's nice to know, but IIRC datetime is from the top 10 Python
modules that need a redesign. Things contained therein doesn't pass
human usability check, and are not used as a result.

My story for that day is that I needed to process geodata from
Wikipedia to see the distribution over the Earth surface, so I thought
that if I know the Python it is not a problem. It appeared that the
problem is there, and the problem is recurring (last time I had GPS
points). Python docs explain a lot about core OS concepts, but Python
is not only a systems language. I'd say that it is now being replaced
with Go for all system stuff, and more attempts are made to use is as
a data processing language. But that is not possible if Python doesn't
have a basic concepts of data representation from the Real Life (RL)
(tm) in its stdlib.

Handling geo coordinates in most used formats is as important as
handling .zip archives in stdlib.
-- 
anatoly t.

From rosuav at gmail.com  Mon Mar 23 08:21:32 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 23 Mar 2015 18:21:32 +1100
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
Message-ID: <CAPTjJmpyqiNgWgMCnVHfWTEyFQ2C8Bs0gTfBOs3R1uhzMv6gaQ@mail.gmail.com>

On Mon, Mar 23, 2015 at 5:08 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> Handling geo coordinates in most used formats is as important as
> handling .zip archives in stdlib.

That may or may not be the case, but if it is, the best way to argue
the point is:

1) Find or create a third-party package on PyPI
2) Demonstrate that this package has a stable API
3) Demonstrate or pledge support for the package
4) Appeal for the package's inclusion in the stdlib

You'll need to show that there's benefit to having it in the stdlib as
opposed to on PyPI - demonstrate the problem with "pip install
packagename" that means that you want it in the standard library. Just
*stating* that there is a need isn't sufficient - you need to prove
it.

ChrisA

From abarnert at yahoo.com  Mon Mar 23 09:29:16 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 23 Mar 2015 01:29:16 -0700
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
Message-ID: <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com>

On Mar 22, 2015, at 11:08 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> 
>>> On Sun, 22 Mar 2015 17:15:39 -0400
>>> Alexander Walters
>>> <tritium-list at sdamon.com> wrote:
>>>>> On 3/21/2015 05:46, anatoly techtonik wrote:
>>>>> I think it should be in stdlib.
>>>>> 
>>>>> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>>>>> 
>>>>> This is from Wikipedia.
>>>> Why does this need to be in the standard library?
>>> 
>>> Because Anatoly needs it, of course :)
>> 
>> Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems.
>> 
>> More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer.
> 
> That's nice to know, but IIRC datetime is from the top 10 Python
> modules that need a redesign. Things contained therein doesn't pass
> human usability check, and are not used as a result.

There are thousands of projects that actually do use it; thousands of people asking questions on StackOverflow, Python-tutor, etc. and getting actual answers; etc. So your theoretical argument that it's not usable doesn't really matter given that practically it is actually usable, and used.

If you just opened up an interactive interpreter and player with the datetime.timedelta class for a few seconds instead of trying to guess based on hearsay and abstract principles whether it might be useful, you could have figured this out for yourself.

> My story for that day is that I needed to process geodata from
> Wikipedia to see the distribution over the Earth surface, so I thought
> that if I know the Python it is not a problem. It appeared that the
> problem is there, and the problem is recurring (last time I had GPS
> points).

I suspect that people who do a lot of geodata processing in Python already have third-party libs that they use, and would have no interest in a very small limited subset being duplicated in the stdlib.

> Python docs explain a lot about core OS concepts, but Python
> is not only a systems language. I'd say that it is now being replaced
> with Go for all system stuff,

I don't know anyone outside Google who's using Go for any system (or sysadmin, which seems to be more likely what you're talking about) stuff. I'm sure there are exceptions, but the idea that Python is being universally replaced by Go in that area is so ridiculous that it's hard to believe you actually meant it seriously. It's like saying that Ruby is being replaced by Scala for all web server stuff just because Scala is the fastest-growing language in that area this week.

> and more attempts are made to use is as
> a data processing language. But that is not possible if Python doesn't
> have a basic concepts of data representation from the Real Life (RL)
> (tm) in its stdlib.

Most people I know who use Python as a serious data processing language don't care much what's in the stdlib because they're using numpy and/or pandas.

> Handling geo coordinates in most used formats is as important as
> handling .zip archives in stdlib.

Not even close. Zip archives are used in a ridiculously wide range of application areas; geoccordinates are used only in processing geocoordinate data (and there only in small enough scales that you don't need numpy or some specialized geodata library that I don't know about but that I'll bet exists).

From breamoreboy at yahoo.co.uk  Mon Mar 23 09:51:09 2015
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Mon, 23 Mar 2015 08:51:09 +0000
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
Message-ID: <meok61$tht$1@ger.gmane.org>

On 23/03/2015 06:08, anatoly techtonik wrote:
> On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> On Sun, 22 Mar 2015 17:15:39 -0400
>>> Alexander Walters
>>> <tritium-list at sdamon.com> wrote:
>>>>> On 3/21/2015 05:46, anatoly techtonik wrote:
>>>>> I think it should be in stdlib.
>>>>>
>>>>> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
>>>>>
>>>>> This is from Wikipedia.
>>>> Why does this need to be in the standard library?
>>>
>>> Because Anatoly needs it, of course :)
>>
>> Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems.
>>
>> More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer.
>
> That's nice to know, but IIRC datetime is from the top 10 Python
> modules that need a redesign.

I'm looking forward to seeing your redesign for all 10 modules on the 
bug tracker.  Can you get this done in time for 3.5 beta?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From tjreedy at udel.edu  Mon Mar 23 09:55:55 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Mar 2015 04:55:55 -0400
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <550FAC80.8040405@gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
Message-ID: <meokfb$2ri$1@ger.gmane.org>

On 3/23/2015 2:02 AM, David Foster wrote:
> * It's a bit vague how local variables should be typed in the presence of
>    multiple assignments. Consider:
>
>    ```
>    if x > 0:
>        root = sqrt(x)  # type: int
>    else:
>        root = None
>    ```

root = sqrt(x) if x > 0 or None  # type: Union(int, None)

> * +1 for only allowing .pyi as the extension for stub files (and not
> also .py).
>    Makes it very clear that they should only be interpreted as stubs.

Agreed

> * +1 for the argument order cast(x, T). This is consistent with (x: T)
> elsewhere.

cast x to T and is x instance of T, agreed.

-- 
Terry Jan Reedy


From mal at egenix.com  Mon Mar 23 10:16:55 2015
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 23 Mar 2015 10:16:55 +0100
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>	<550F30FB.5010000@sdamon.com>
 <20150322223815.06082280@fsol>	<21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
Message-ID: <550FDA07.5020704@egenix.com>

On 23.03.2015 07:08, anatoly techtonik wrote:
> My story for that day is that I needed to process geodata from
> Wikipedia to see the distribution over the Earth surface, so I thought
> that if I know the Python it is not a problem. It appeared that the
> problem is there, and the problem is recurring (last time I had GPS
> points). Python docs explain a lot about core OS concepts, but Python
> is not only a systems language. I'd say that it is now being replaced
> with Go for all system stuff, and more attempts are made to use is as
> a data processing language. But that is not possible if Python doesn't
> have a basic concepts of data representation from the Real Life (RL)
> (tm) in its stdlib.

Well, sorry Anatoly that we haven't already solved all your problems
for you ;-)

If you are really interested in a solution to your problem,
have a look at this large collection of tools for
doing GIS in Python:

https://pythongisresources.wordpress.com/categories/

More specifically, you're probably looking for one of these packages:

https://pythongisresources.wordpress.com/projections-2/

It's all just a pip install away.

BTW: If you expect the Python stdlib to make coffee, clean your house
and drive your kids to school, you're probably looking at the
wrong language - we only do batteries in the stdlib :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 23 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2015-03-12: Released mxODBC 3.3.2 ...             http://egenix.com/go71

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From random832 at fastmail.us  Mon Mar 23 12:36:11 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 23 Mar 2015 07:36:11 -0400
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CALGmxEJ0GBH7O9qPPkQ77ydqC33r14+0OnRY7b8xzS6sMq9K4Q@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CALGmxEJ0GBH7O9qPPkQ77ydqC33r14+0OnRY7b8xzS6sMq9K4Q@mail.gmail.com>
Message-ID: <1427110571.3985053.243972942.3E7E5E29@webmail.messagingengine.com>

On Sun, Mar 22, 2015, at 19:13, Chris Barker wrote:
> On Sun, Mar 22, 2015 at 3:37 PM, Andrew Barnert <
> abarnert at yahoo.com.dmarc.invalid> wrote:
> 
> >  If you can't figure out the complexity of multiplying and divmodding by
> > 60 twice, you can borrow its code, or just wrap it and delegate. That seems
> > trivial enough to do in any project that needs it.
> 
> 
> I"m not suggesting this is an stdlib candiate, but it'snot quite that
> simple. With FP precision issues you can end up with results like:
> 
> 1? 60? 13?

I'm not sure I believe this is true. Do you have a concrete example of
an actual floating point value that will do this?

From greg.ewing at canterbury.ac.nz  Mon Mar 23 09:15:26 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 23 Mar 2015 21:15:26 +1300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CALGmxEJ0GBH7O9qPPkQ77ydqC33r14+0OnRY7b8xzS6sMq9K4Q@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CALGmxEJ0GBH7O9qPPkQ77ydqC33r14+0OnRY7b8xzS6sMq9K4Q@mail.gmail.com>
Message-ID: <550FCB9E.3000709@canterbury.ac.nz>

Chris Barker wrote:
> With FP precision issues you can end up with results like:
> 
> 1? 60? 13? 

Do you have an example that gives this result?

-- 
Greg

From taleinat at gmail.com  Mon Mar 23 15:52:06 2015
From: taleinat at gmail.com (Tal Einat)
Date: Mon, 23 Mar 2015 16:52:06 +0200
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <20150321124732.0b47730e@fsol>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
Message-ID: <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>

On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
> Paddy3118 <paddy3118 at gmail.com> wrote:
>>
>> The  fix seems clear: deprecate os.path.commonprefix whilst creating a true
>> os.path.commonpath and str.commonprefix. The deprecated function should
>> hang around and the standard libs modified to switch to the new function(s)
>
> +1 from me.

+1

From p.f.moore at gmail.com  Mon Mar 23 16:24:22 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Mar 2015 15:24:22 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
Message-ID: <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>

On 23 March 2015 at 14:52, Tal Einat <taleinat at gmail.com> wrote:
> On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
>> Paddy3118 <paddy3118 at gmail.com> wrote:
>>>
>>> The  fix seems clear: deprecate os.path.commonprefix whilst creating a true
>>> os.path.commonpath and str.commonprefix. The deprecated function should
>>> hang around and the standard libs modified to switch to the new function(s)
>>
>> +1 from me.
>
> +1

+1. Maybe adding a commonprefix operation to pathlib would be a good
idea as well - using that would avoid the confusion between the
deprecated os.path.commonprefix and os.path.commonpath...

Paul

From encukou at gmail.com  Mon Mar 23 16:39:01 2015
From: encukou at gmail.com (Petr Viktorin)
Date: Mon, 23 Mar 2015 16:39:01 +0100
Subject: [Python-ideas] A macro for easier rich comparisons
In-Reply-To: <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com>
References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com>
 <CA+=+wqD6R5-vg9W250jbwUceCuizD8vJDJ=meYVr0Z3BmO4JGA@mail.gmail.com>
 <CA+=+wqCfv=hhuFzA6EMF4n7ru++fOO0aXfgVdPwKT=RCSPuVwg@mail.gmail.com>
 <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com>
Message-ID: <CA+=+wqB2RObV0=AUBasavYr3-y_-WCKkGDMMCRQMzBVmVb1huQ@mail.gmail.com>

On Fri, Mar 20, 2015 at 11:19 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Mar 20, 2015, at 7:24 AM, Petr Viktorin <encukou at gmail.com> wrote:
>>
>> A sequence of comparisons is less efficient than a switch statement.
>> Same with PyBool_FromLong vs. Py_RETURN_*
>> Also, it might be too complex for a macro.
>> These points could be solved by making it a function instead.
>
> Or by just making the macro a statement instead of a function; then the code can be exactly the same code you're abstracting out, and instead of using it like this:
>
>      return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op);
>
> ... you use it like this:
>
>     PY_RETURNRICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op);

Thanks Andrew.
This makes it harder to do cleanup after the comparison (e.g. jump to
a common block of Py_DECREFs, or do PyFPE_END_PROTECT in float code).
But those should be rare enough, or can be easily worked around, so a
"return" macro is still useful.

From tritium-list at sdamon.com  Mon Mar 23 17:44:43 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Mon, 23 Mar 2015 12:44:43 -0400
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
Message-ID: <551042FB.4000203@sdamon.com>

I could go on an on about my experience and what problems I solve, but...

Nope.  that is just flat wrong.  Zipfiles are part of the import system, 
and everyone uses them all the time in python.  Coordinate conversion is 
not a problem I have encountered, and a far as i can tell, not many 
people will.

On 3/23/2015 02:08, anatoly techtonik wrote:
> Handling geo coordinates in most used formats is as important as
> handling .zip archives in stdlib.


From alexander.belopolsky at gmail.com  Mon Mar 23 18:21:36 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 23 Mar 2015 13:21:36 -0400
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
Message-ID: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>

On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik <techtonik at gmail.com>
wrote:
>
> That's nice to know, but IIRC datetime is from the top 10 Python
> modules that need a redesign. Things contained therein doesn't pass
> human usability check, and are not used as a result.


Where have you been when PEP 3108 was discussed?  I have not seen any other
list of Python modules that needed a redesign, so I cannot tell what's on
your top ten list.

Speaking of the datetime module, in what sense does it not "pass human
usability check"?  It does have a few quirks, for example I would rather
see date accept a single argument in the constructor which may be a string,
another date or a tuple, but I am not even sure this desire is shared by
many other humans.  It would be nice if datetime classes were named in
CamelCase according to PEP 8 conventions, but again this is a very minor
quirk.

In my view, if anyone is to blame for the "human usability" of the datetime
module, it would be Pope Gregory XIII, Benjamin Franklin and scores of
unnamed astronomers who made modern timekeeping such a mess.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/d9641320/attachment-0001.html>

From tritium-list at sdamon.com  Mon Mar 23 18:32:49 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Mon, 23 Mar 2015 13:32:49 -0400
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
Message-ID: <55104E41.10609@sdamon.com>

It works fine for me.  It does not do EVERYTHING I need, but with 
dateutil and pytz, I am satisfied with the datetime infrastructure in 
python.  Sure, I would like things changed in it, but its not like I 
expect the every module in the standard library to work exactly how I want.


On 3/23/2015 13:21, Alexander Belopolsky wrote:
>
> Speaking of the datetime module, in what sense does it not "pass human 
> usability check"?  It does have a few quirks, for example I would 
> rather see date accept a single argument in the constructor which may 
> be a string, another date or a tuple, but I am not even sure this 
> desire is shared by many other humans.  It would be nice if datetime 
> classes were named in CamelCase according to PEP 8 conventions, but 
> again this is a very minor quirk.


From tjreedy at udel.edu  Mon Mar 23 19:59:55 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 23 Mar 2015 14:59:55 -0400
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
Message-ID: <mepnrr$ova$1@ger.gmane.org>

On 3/23/2015 1:21 PM, Alexander Belopolsky wrote:
>
> On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik
> <techtonik at gmail.com
> <mailto:techtonik at gmail.com>> wrote:
>  >
>  > That's nice to know, but IIRC datetime is from the top 10 Python
>  > modules that need a redesign. Things contained therein doesn't pass
>  > human usability check, and are not used as a result.

...
> in what sense does it not "pass human usability check"?
...

Please don't feed the troll.  He is known for opinionated digs like the 
above and historically, nothing good comes from trying to discuss them.

-- 
Terry Jan Reedy


From guido at python.org  Mon Mar 23 20:55:37 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Mar 2015 12:55:37 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <550FAC80.8040405@gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
Message-ID: <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>

On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com> wrote:

> * It's a bit vague how local variables should be typed in the presence of
>   multiple assignments. Consider:
>
>   ```
>   if x > 0:
>       root = sqrt(x)  # type: int
>   else:
>       root = None
>   ```
>
>   It seems that the type of `root` should be inferred as `Union[int, None]`
>   (AKA `Optional[int]`), but there's no discussion about this type of
>   inference in the PEP.
>
>   Cases like this are the only way that the proposed blank `Undefined`
>   value makes sense to me, using an example like:
>
>   ```
>   root = Undefined    # type: Optional[int]
>   if x > 0:
>       root = sqrt(x)  # type: int
>   else:
>       root = None
>   ```
>

Right. Another use case for Undefined is class variables (serving as
instance variable initialization/declaration).

If you feel strongly that this needs to be mentioned in the PEP, can you
send a PR to github.com/ambv/typehinting ?


> * +1 for only allowing .pyi as the extension for stub files (and not also
> .py).
>   Makes it very clear that they should only be interpreted as stubs.
>

Perhaps. Though there are also downsides.


> * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and
>   directly corresponds conceptually to types like `int?` and `?int` from
>   other languages.
>

Indeed, Optional[t] is much clearer to the human reader than Union[t, None].


> * +1 for the argument order cast(x, T). This is consistent with (x: T)
> elsewhere.
>

I disagree on this. It goes against the argument order of casts in other
languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
x(int).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/d7a5b334/attachment.html>

From rymg19 at gmail.com  Mon Mar 23 21:13:46 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 23 Mar 2015 15:13:46 -0500
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
Message-ID: <CAO41-mNpMAFm15Z-ee6XOTVcOEmR+icEEueJbKQjovku2Vwf8A@mail.gmail.com>

On Mon, Mar 23, 2015 at 2:55 PM, Guido van Rossum <guido at python.org> wrote:

> On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com>
> wrote:
>
>> * It's a bit vague how local variables should be typed in the presence of
>>   multiple assignments. Consider:
>>
>>   ```
>>   if x > 0:
>>       root = sqrt(x)  # type: int
>>   else:
>>       root = None
>>   ```
>>
>>   It seems that the type of `root` should be inferred as `Union[int,
>> None]`
>>   (AKA `Optional[int]`), but there's no discussion about this type of
>>   inference in the PEP.
>>
>>   Cases like this are the only way that the proposed blank `Undefined`
>>   value makes sense to me, using an example like:
>>
>>   ```
>>   root = Undefined    # type: Optional[int]
>>   if x > 0:
>>       root = sqrt(x)  # type: int
>>   else:
>>       root = None
>>   ```
>>
>
> Right. Another use case for Undefined is class variables (serving as
> instance variable initialization/declaration).
>
> If you feel strongly that this needs to be mentioned in the PEP, can you
> send a PR to github.com/ambv/typehinting ?
>
>
>> * +1 for only allowing .pyi as the extension for stub files (and not also
>> .py).
>>   Makes it very clear that they should only be interpreted as stubs.
>>
>
> Perhaps. Though there are also downsides.
>
>
>> * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and
>>   directly corresponds conceptually to types like `int?` and `?int` from
>>   other languages.
>>
>
> Indeed, Optional[t] is much clearer to the human reader than Union[t,
> None].
>
>
>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
>> elsewhere.
>>
>
> I disagree on this. It goes against the argument order of casts in other
> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
> x(int).
>

I brought this up somewhere before, but I still think this would be great
for casts:

cast[T](x)

Like C++'s named casts (static_cast<T>(x), dynamic_cast<T>(x), etc.). It
gives the notion that T is a type, not some random variable, and that cast
is NOT a normal function call.


>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/713f132d/attachment-0001.html>

From guido at python.org  Mon Mar 23 21:17:32 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Mar 2015 13:17:32 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAO41-mNpMAFm15Z-ee6XOTVcOEmR+icEEueJbKQjovku2Vwf8A@mail.gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
 <CAO41-mNpMAFm15Z-ee6XOTVcOEmR+icEEueJbKQjovku2Vwf8A@mail.gmail.com>
Message-ID: <CAP7+vJKWQTAJdX_+crxYZPuh2EuhJM-tWPrh8mYceAZ+1a=Xkg@mail.gmail.com>

Yeah, I've considered that, but it feels too punctuation-heavy (as does the
C++ cast<T>(x)). Also, we need cast() to be a fast no-op; cast[T](x)
necessarily means calling two functions rather than one.

On Mon, Mar 23, 2015 at 1:13 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> On Mon, Mar 23, 2015 at 2:55 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com>
>> wrote:
>>
>>> * It's a bit vague how local variables should be typed in the presence of
>>>   multiple assignments. Consider:
>>>
>>>   ```
>>>   if x > 0:
>>>       root = sqrt(x)  # type: int
>>>   else:
>>>       root = None
>>>   ```
>>>
>>>   It seems that the type of `root` should be inferred as `Union[int,
>>> None]`
>>>   (AKA `Optional[int]`), but there's no discussion about this type of
>>>   inference in the PEP.
>>>
>>>   Cases like this are the only way that the proposed blank `Undefined`
>>>   value makes sense to me, using an example like:
>>>
>>>   ```
>>>   root = Undefined    # type: Optional[int]
>>>   if x > 0:
>>>       root = sqrt(x)  # type: int
>>>   else:
>>>       root = None
>>>   ```
>>>
>>
>> Right. Another use case for Undefined is class variables (serving as
>> instance variable initialization/declaration).
>>
>> If you feel strongly that this needs to be mentioned in the PEP, can you
>> send a PR to github.com/ambv/typehinting ?
>>
>>
>>> * +1 for only allowing .pyi as the extension for stub files (and not
>>> also .py).
>>>   Makes it very clear that they should only be interpreted as stubs.
>>>
>>
>> Perhaps. Though there are also downsides.
>>
>>
>>> * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and
>>>   directly corresponds conceptually to types like `int?` and `?int` from
>>>   other languages.
>>>
>>
>> Indeed, Optional[t] is much clearer to the human reader than Union[t,
>> None].
>>
>>
>>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
>>> elsewhere.
>>>
>>
>> I disagree on this. It goes against the argument order of casts in other
>> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
>> x(int).
>>
>
> I brought this up somewhere before, but I still think this would be great
> for casts:
>
> cast[T](x)
>
> Like C++'s named casts (static_cast<T>(x), dynamic_cast<T>(x), etc.). It
> gives the notion that T is a type, not some random variable, and that cast
> is NOT a normal function call.
>
>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/f838e24b/attachment.html>

From p.f.moore at gmail.com  Mon Mar 23 22:05:04 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Mar 2015 21:05:04 +0000
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
Message-ID: <CACac1F-MpMEUyXoSWbVeYoxiTM4ezQqTD32H4+Zsk8H8NTMRNw@mail.gmail.com>

On 23 March 2015 at 19:55, Guido van Rossum <guido at python.org> wrote:
>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
>> elsewhere.
>
> I disagree on this. It goes against the argument order of casts in other
> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
> x(int).

However, in SQL, which is the only language I know of with an with an
explicitly named cast function (unless you count C++'s
static_cast<T>(foo) which isn't normal function syntax), the syntax is
CAST(foo AS type). So there's definitely a precedent (arguably
stronger, depending on your background) for cast(x, T).

Paul.

From njs at pobox.com  Mon Mar 23 22:17:20 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 23 Mar 2015 14:17:20 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
Message-ID: <CAPJVwBktkezjnkXTPK7KgtHz6bH3NyFGraa35EQv8OUvnaJ-DA@mail.gmail.com>

On Mar 23, 2015 1:03 PM, "Guido van Rossum" <guido at python.org> wrote:
>
> On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com>
wrote:
>>
>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
elsewhere.
>
>
> I disagree on this. It goes against the argument order of casts in other
languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
x(int).

I don't have any strong opinion here, but I don't find the consistency
argument convincing. In int(x), 'int' is the verb, and in English verbs
come before undergoers ("I am inting the x"). In cast(...), though, cast is
the verb, x remains the undergoer, and int is conceptualized as the
destination (or something like that), and destinations go in prepositional
clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm
throwing the ball to Sarah"). "I'm casting to int the x" is extremely weird.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/efb80a21/attachment.html>

From greg at krypto.org  Mon Mar 23 22:33:48 2015
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 23 Mar 2015 21:33:48 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
Message-ID: <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>

+1 pathlib would be the appropriate place for the correctly behaving
function to appear.

os.path.commonprefix()'s behavior would be better off as a str and bytes
method rather than a crazy function in the os.path library but I doubt
anyone *really* wants to add more methods to those.

On Mon, Mar 23, 2015 at 8:25 AM Paul Moore <p.f.moore at gmail.com> wrote:

> On 23 March 2015 at 14:52, Tal Einat <taleinat at gmail.com> wrote:
> > On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
> >> Paddy3118 <paddy3118 at gmail.com> wrote:
> >>>
> >>> The  fix seems clear: deprecate os.path.commonprefix whilst creating a
> true
> >>> os.path.commonpath and str.commonprefix. The deprecated function should
> >>> hang around and the standard libs modified to switch to the new
> function(s)
> >>
> >> +1 from me.
> >
> > +1
>
> +1. Maybe adding a commonprefix operation to pathlib would be a good
> idea as well - using that would avoid the confusion between the
> deprecated os.path.commonprefix and os.path.commonpath...
>
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/6ab67e40/attachment-0001.html>

From cs at zip.com.au  Mon Mar 23 22:41:45 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 24 Mar 2015 08:41:45 +1100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <meokfb$2ri$1@ger.gmane.org>
References: <meokfb$2ri$1@ger.gmane.org>
Message-ID: <20150323214145.GA94958@cskk.homeip.net>

On 23Mar2015 04:55, Terry Reedy <tjreedy at udel.edu> wrote:
>On 3/23/2015 2:02 AM, David Foster wrote:
>>* It's a bit vague how local variables should be typed in the presence of
>>   multiple assignments. Consider:
>>
>>   ```
>>   if x > 0:
>>       root = sqrt(x)  # type: int
>>   else:
>>       root = None
>>   ```
>
>root = sqrt(x) if x > 0 or None  # type: Union(int, None)

Should people need to reshape their code for this?
In the example above it is small, though less readable already.

But it scales very badly; what of a 4 way if etc? It moves from a bit less 
readable to a nightmare.

Cheers,
Cameron Simpson <cs at zip.com.au>

Do not cross the oncoming lanes of death that are California's highways.
        - SJ Mercury News

From abarnert at yahoo.com  Mon Mar 23 22:59:21 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 23 Mar 2015 14:59:21 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150323214145.GA94958@cskk.homeip.net>
References: <meokfb$2ri$1@ger.gmane.org>
 <20150323214145.GA94958@cskk.homeip.net>
Message-ID: <76D24940-328A-46A0-86D0-A073F8CB51D7@yahoo.com>

On Mar 23, 2015, at 2:41 PM, Cameron Simpson <cs at zip.com.au> wrote:
> 
>> On 23Mar2015 04:55, Terry Reedy <tjreedy at udel.edu> wrote:
>>> On 3/23/2015 2:02 AM, David Foster wrote:
>>> * It's a bit vague how local variables should be typed in the presence of
>>>  multiple assignments. Consider:
>>> 
>>>  ```
>>>  if x > 0:
>>>      root = sqrt(x)  # type: int
>>>  else:
>>>      root = None
>>>  ```
>> 
>> root = sqrt(x) if x > 0 or None  # type: Union(int, None)
> 
> Should people need to reshape their code for this?
> In the example above it is small, though less readable already.

If this is going to be the example we keep focusing on...

Why would you have a custom int sqrt function whose return type is neither declared nor inferrable? I'd suspect that most variables that are initialized with a function call don't need a type hint, and if there are special cases where it comes up, they may be special enough that a real-life-plausible example is worth having rather than an example that's almost certainly unnecessary (or, in this case, an explicit type error, or code that type checks but only because you accidentally fooled the checker, because the last time I actually wrote, used, or wanted an int sqrt function it was in 6502 assembly...)

> But it scales very badly; what of a 4 way if etc? It moves from a bit less readable to a nightmare.

Agreed.

In fact, even when what you have is an expression in the first place, Python makes it easy (and relatively cheap) to break it up into a sequence of statements (e.g., assignments to temporary variables), and that's more often a good idea than the other way around, so typing had better work nicely for the statement version of things (and Jukka and Guido seem to have worked hard to make that true in general).

> 
> Cheers,
> Cameron Simpson <cs at zip.com.au>
> 
> Do not cross the oncoming lanes of death that are California's highways.
>       - SJ Mercury News
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From solipsis at pitrou.net  Mon Mar 23 23:08:06 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 23:08:06 +0100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
 <CAPJVwBktkezjnkXTPK7KgtHz6bH3NyFGraa35EQv8OUvnaJ-DA@mail.gmail.com>
Message-ID: <20150323230806.7a0653b9@fsol>

On Mon, 23 Mar 2015 14:17:20 -0700
Nathaniel Smith <njs at pobox.com> wrote:

> On Mar 23, 2015 1:03 PM, "Guido van Rossum" <guido at python.org> wrote:
> >
> > On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com>
> wrote:
> >>
> >> * +1 for the argument order cast(x, T). This is consistent with (x: T)
> elsewhere.
> >
> >
> > I disagree on this. It goes against the argument order of casts in other
> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
> x(int).
> 
> I don't have any strong opinion here, but I don't find the consistency
> argument convincing. In int(x), 'int' is the verb, and in English verbs
> come before undergoers ("I am inting the x"). In cast(...), though, cast is
> the verb, x remains the undergoer, and int is conceptualized as the
> destination (or something like that), and destinations go in prepositional
> clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm
> throwing the ball to Sarah"). "I'm casting to int the x" is extremely weird.

I agree with cast(x, T). It also nicely mirrors isinstance(x, T).

Regards

Antoine.



From solipsis at pitrou.net  Mon Mar 23 23:09:37 2015
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Mar 2015 23:09:37 +0100
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
Message-ID: <20150323230937.0af5ee32@fsol>

On Mon, 23 Mar 2015 21:33:48 +0000
"Gregory P. Smith" <greg at krypto.org> wrote:
> +1 pathlib would be the appropriate place for the correctly behaving
> function to appear.

Patches welcome :-)

Regards

Antoine.


> 
> os.path.commonprefix()'s behavior would be better off as a str and bytes
> method rather than a crazy function in the os.path library but I doubt
> anyone *really* wants to add more methods to those.
> 
> On Mon, Mar 23, 2015 at 8:25 AM Paul Moore <p.f.moore at gmail.com> wrote:
> 
> > On 23 March 2015 at 14:52, Tal Einat <taleinat at gmail.com> wrote:
> > > On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net>
> > wrote:
> > >> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT)
> > >> Paddy3118 <paddy3118 at gmail.com> wrote:
> > >>>
> > >>> The  fix seems clear: deprecate os.path.commonprefix whilst creating a
> > true
> > >>> os.path.commonpath and str.commonprefix. The deprecated function should
> > >>> hang around and the standard libs modified to switch to the new
> > function(s)
> > >>
> > >> +1 from me.
> > >
> > > +1
> >
> > +1. Maybe adding a commonprefix operation to pathlib would be a good
> > idea as well - using that would avoid the confusion between the
> > deprecated os.path.commonprefix and os.path.commonpath...
> >
> > Paul
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> 



From chris.barker at noaa.gov  Mon Mar 23 23:19:29 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 23 Mar 2015 15:19:29 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150323214145.GA94958@cskk.homeip.net>
References: <meokfb$2ri$1@ger.gmane.org>
 <20150323214145.GA94958@cskk.homeip.net>
Message-ID: <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>

On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson <cs at zip.com.au> wrote:

> But it scales very badly; what of a 4 way if etc? It moves from a bit less
> readable to a nightmare.
>

in python one particular type or None is a very common idiom -- well worth
special casing. Somethign that could take or return 4 different types is a
far less usual case, we can be more verbose there.

Note that Cython and Pyrex default to "or NOne", and you have to do

func( int x not None)

if you don't want to allow the None. That's because None as a not-set-yet
value is so common. In this case, it's a mistake to make it the default, as
Cython does not generate checks later for None, so it can get ugly if you
don't specifically test for it in your code. None the less, it's very
useful.

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/32ab8e64/attachment.html>

From guido at python.org  Mon Mar 23 23:35:23 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Mar 2015 15:35:23 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
References: <meokfb$2ri$1@ger.gmane.org>
 <20150323214145.GA94958@cskk.homeip.net>
 <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
Message-ID: <CAP7+vJK0dONCovkjVbr7iFyuo24z1h-_urvKkvv=K8zG4LvagQ@mail.gmail.com>

On Mon, Mar 23, 2015 at 3:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:

>
> On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson <cs at zip.com.au> wrote:
>
>> But it scales very badly; what of a 4 way if etc? It moves from a bit
>> less readable to a nightmare.
>>
>
> in python one particular type or None is a very common idiom -- well worth
> special casing. Somethign that could take or return 4 different types is a
> far less usual case, we can be more verbose there.
>
> Note that Cython and Pyrex default to "or NOne", and you have to do
>
> func( int x not None)
>
> if you don't want to allow the None. That's because None as a not-set-yet
> value is so common. In this case, it's a mistake to make it the default, as
> Cython does not generate checks later for None, so it can get ugly if you
> don't specifically test for it in your code. None the less, it's very
> useful.
>

There's a lot of discussion in the typehinting tracker and also in the mypy
tracker about what to do with "or None" types. mypy currently doesn't
complain about this because it's deemed too noisy (Jukka regularly tries to
convert various real-world code bases to type hints, and he learns a lot
from these).

That said, about the specific example used here, the best way to deal with
this currently is to put an annotation comment on the first line, like this:

    if a < 0:
        r = None  # type: Optional[float]
    else:
        r = sqrt(a)

There's currently one place in the PEP where Optional[t] is automatically
inferred: an argument that's annotated and has None for a default:

def foo(x: int = None):
    ...

The actual type of x here is Optional[int].

Note that the PEP doesn't prescribe that the type of a variable becomes the
union of the type assigned to in in separate branches of a conditional.
This is intentional -- except for a few special cases (like discussed here)
giving a variable a different type in different branches (if it is used
after the branches join) is more likely a bug than desired.

Experimenting with this in mypy is difficult because it explicitly silences
complaints about None (see above); if you want to understand what it does
in general it's better to experiment with two distinct non-None types, e.g.
float and str. Using this approach I found that when you use an
if-statement, you should put a #type: comment on the assignment in the
first branch to silence it; when you use an if-expression, there's no way
to shut it up! (This is exactly the opposite of the assumption that was
made earlier in this thread.)

PS. About the cast() argument order, I've opened a tracker issue for this:
https://github.com/ambv/typehinting/issues/63

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/05a75e8c/attachment-0001.html>

From p.f.moore at gmail.com  Mon Mar 23 23:48:47 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Mar 2015 22:48:47 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <20150323230937.0af5ee32@fsol>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
Message-ID: <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>

On 23 March 2015 at 22:09, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 23 Mar 2015 21:33:48 +0000
> "Gregory P. Smith" <greg at krypto.org> wrote:
>> +1 pathlib would be the appropriate place for the correctly behaving
>> function to appear.
>
> Patches welcome :-)

I'll see what I can do. Basically it's just

def commonprefix(p1, p2):
    cp = []
    for (pp1, pp2) in zip(p1.parts, p2.parts):
        if pp1 != pp2:
            break
        cp.append(pp1)
    return pathlib.Path(*cp)

(extended to an arbitrary number of args) but getting the corner cases
right is a little more tricky. If there is no common prefix, I guess
returning None is correct - it's the nearest equivalent to
os.path.commonprefix returning ''. The type of the return value should
be a concrete type - probably type(p1)?

Maybe this should be a method on Path objects - p.commonprefix(p1, p2,
...) although it seems a bit odd that the first argument (self) is
treated differently in the signature.

Anyone got any preferences?
Paul

From p.f.moore at gmail.com  Mon Mar 23 23:56:17 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Mar 2015 22:56:17 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
Message-ID: <CACac1F_YGkEfJ6-KL4obYKXinK_ELJ+YWePY9=O2ggedNBmKsQ@mail.gmail.com>

On 23 March 2015 at 22:48, Paul Moore <p.f.moore at gmail.com> wrote:
> I'll see what I can do. Basically it's just

More specifically, as a pathlib.Path method:

def commonprefix(self, *rest):
    cp = []
    for p0, *ps in zip(self.parts, *[pp.parts for pp in rest]):
        if any(p0 != pi for pi in ps):
            break
        cp.append(p0)
    if not cp:
        return None
    return type(self)(*cp)

Paul

From guido at python.org  Tue Mar 24 00:13:57 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Mar 2015 16:13:57 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150321153731.GN31862@ando.pearwood.info>
References: <CAP7+vJL_XR7e+FUMYWYVhsy+ZsNg6xiOg8VVoew3_JBh86b2Rw@mail.gmail.com>
 <20150321153731.GN31862@ando.pearwood.info>
Message-ID: <CAP7+vJKE9n0NQhY1nVmgwVi+dJ-Kk2wCvEzy8P2SvNEZ-pd0kw@mail.gmail.com>

(This one took longer to respond to because there are so many nits. :-)

On Sat, Mar 21, 2015 at 8:37 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote:
>
> > Union types
> > -----------
> [...]
> > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;
>
> That only saves three characters. Is it worth it?
>

Yes (see other messages in the thread).


> > An optional type is also automatically assumed when the default value is
> > ``None``, for example::
> >
> >   def handle_employee(e: Employee = None): ...
>
> Should that apply to all default values or just None? E.g. if I have
>
>     def spam(s: str = 23): ...
>
> should that be inferred as Union[str, int] or be flagged as a type
> error? I think that we want a type error here, and it's only None that
> actually should be treated as special. Perhaps that should be made
> explicit in the PEP.
>

I only want this special treatment for None.  The PEP seems to be pretty
clear about this.


> [...]
> > For the purposes of type hinting, the type checker assumes ``__debug__``
> > is set to ``True``, in other words the ``-O`` command-line option is not
> > used while type checking.
>
> I'm afraid I don't understand what you are trying to say here. I would
> have expected that __debug__ and -O and the type checker would be
> independent of each other.
>

Well, __debug__ tracks -O (__debug__ is false iff -O is given). But a type
checker usually doesn't have the luxury to know whether the code is meant
to run with -O or without it. So it assumes -O is *not* given, i.e.
__debug__ is True.

(You could try to type-check twice, once with -O and once without, but this
idea doesn't really scale when you consider all the other flags that might
be treated this way. And it's not really worth the trouble.)


> [...]
> > To mark portions of the program that should not be covered by type
> > hinting, use the following:
> >
> > * a ``@no_type_check`` decorator on classes and functions
> >
> > * a ``# type: ignore`` comment on arbitrary lines
> >
> > .. FIXME: should we have a module-wide comment as well?
>
> I think so, if for no other reason than it will reduce the fear of some
> people that type checks will be mandatory.
>

That fear ought to be reduced to zero by words in the PEP; have you got any
suggestions? The mere presence of a module-wide comment to disable type
checks might actually *increase* the fear that (absent such a comment) type
checks might be mandatory.

I could imagine other reasons for wanting a file-scoped directive, but I'm
not sure -- there's a lot of discussion in
https://github.com/ambv/typehinting/issues/35, maybe you can make sense of
it.


> > Type Hints on Local and Global Variables
> > ========================================
> >
> > No first-class syntax support for explicitly marking variables as being
> > of a specific type is added by this PEP.  To help with type inference in
> > complex cases, a comment of the following format may be used::
> >
> >   x = []   # type: List[Employee]
> >
> > In the case where type information for a local variable is needed before
> > it is declared, an ``Undefined`` placeholder might be used::
> >
> >   from typing import Undefined
> >
> >   x = Undefined   # type: List[Employee]
> >   y = Undefined(int)
>
> How is that better than just bringing forward the variable declaration?
>
>     x = []   # type: List[Employee]
>     y = 0
>

The actual initialization might have to happen later, separately in
different branches of an if-statement; or this might be a class variable.
Jukka gave some more reasons for having Undefined in
https://github.com/ambv/typehinting/issues/20


> > Casts
> > =====
> >
> > Occasionally the type checker may need a different kind of hint: the
> > programmer may know that an expression is of a more constrained type
> > than the type checker infers.  For example::
> >
> >   from typing import List
> >
> >   def find_first_str(a: List[object]) -> str:
> >       index = next(i for i, x in enumerate(a) if isinstance(x, str))
> >       # We only get here if there's at least one string in a
> >       return cast(str, a[index])
> >
> > The type checker infers the type ``object`` for ``a[index]``, but we
> > know that (if the code gets to that point) it must be a string.  The
> > ``cast(t, x)`` call tells the type checker that we are confident that
> > the type of ``x`` is ``t``.
>
> Is the type checker supposed to unconditionally believe the cast, or
> only if the cast is more constrained than the infered type (like str and
> object, or bool and int)?
>
> E.g. if the type checker infers int, and the cast says list, I'm not
> entirely sure I would trust the programmer more than the type checker.
>
> My feeling here is that some type checkers will unconditionally trust
> the cast, and some will flag the mismatch, or offer a config option to
> swap between the two, and that will be a feature for type checkers to
> compete on.
>

It should unconditionally believe the cast. (Reference:
https://github.com/ambv/typehinting/issues/15#issuecomment-69136820)


> I'm also going to bike-shed the order of arguments. It seems to me that
> we want to say:
>
>     cast(x, T)  # pronounced "cast x to T"
>
> rather than Yoda-speak "cast T x to we shall" *wink*. That also matches
> the order of isinstance(obj, type) calls and makes it easier to
> remember.
>

 Seems you're not alone here. :-) I've opened
https://github.com/ambv/typehinting/issues/63


> > At runtime a cast always returns the
> > expression unchanged -- it does not check the type, and it does not
> > convert or coerce the value.
>
> I'm a little concerned about cast() being a function. I know that it's a
> do-nothing function, but there's still the overhead of the name lookup
> and function call. It saddens me that giving a hint to the type checker
> has a runtime cost, small as it is.
>
> (I know that *technically* annotations have a runtime cost too, but
> they're once-only, at function definition time, not every time you call
> the function.)
>
> Your point below that cast() can be used inside expressions is a valid
> point, so there has to be a cast() function to support those cases, but
> for the example given here where the cast occurs in a return statement,
> wouldn't a type comment do?
>
>     return some_expression  # type: T
>
> hints that some_expression is to be treated as type T, regardless of
> what was infered.
>

That has slightly different semantics -- while the type checker should
unconditionally believe cast(), #type: comments are required to be
consistent.

In the example from the PEP (find_first_str()) the cas() is required, since
the derived type is object.


> > Casts differ from type comments (see the previous section).  When
> > using a type comment, the type checker should still verify that the
> > inferred type is consistent with the stated type.  When using a cast,
> > the type checker trusts the programmer.  Also, casts can be used in
> > expressions, while type comments only apply to assignments.
>
>
> > Stub Files
> > ==========
> [...]
> > Stub files may use the ``.py`` extension or alternatively may use the
> > ``.pyi`` extension.  The latter makes it possible to maintain stub
> > files in the same directory as the corresponding real module.
>
> I don't like anything that could cause confusion between stub files and
> actual Python files. If we allow .py extension on stub files, I'm sure
> there will be confusing errors where people somehow manage to get the
> stub file imported instead of the actual module they want.
>
> Is there any advantage to allowing stub files use a .py extension? If
> not, then don't allow it.
>

I'm tracking this at https://github.com/ambv/typehinting/issues/64 now.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/327e52d0/attachment-0001.html>

From cs at zip.com.au  Tue Mar 24 00:50:53 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 24 Mar 2015 10:50:53 +1100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
References: <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
Message-ID: <20150323235053.GA36178@cskk.homeip.net>

On 23Mar2015 15:19, Chris Barker - NOAA Federal <chris.barker at noaa.gov> wrote:
>On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson <cs at zip.com.au> wrote:
>> But it scales very badly; what of a 4 way if etc? It moves from a bit less
>> readable to a nightmare.
>
>in python one particular type or None is a very common idiom -- well worth
>special casing. Somethign that could take or return 4 different types is a
>far less usual case, we can be more verbose there.

I did not mean 4 different types, I meant a four (or more) way decision which 
gave a (probably) single type or None. If "a if b else c" construction rapidly 
becomes ery hard to read/verify.

Cheers,
Cameron Simpson <cs at zip.com.au>

A crash reduces
your expensive computer
to a simple stone.
- Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html

From rosuav at gmail.com  Tue Mar 24 00:56:46 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Mar 2015 10:56:46 +1100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CACac1F-MpMEUyXoSWbVeYoxiTM4ezQqTD32H4+Zsk8H8NTMRNw@mail.gmail.com>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
 <CACac1F-MpMEUyXoSWbVeYoxiTM4ezQqTD32H4+Zsk8H8NTMRNw@mail.gmail.com>
Message-ID: <CAPTjJmqOP1mWbfdHtnZiFY=8Kn4R0CE9krz3aijzSrmq4_w7AA@mail.gmail.com>

On Tue, Mar 24, 2015 at 8:05 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 23 March 2015 at 19:55, Guido van Rossum <guido at python.org> wrote:
>>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
>>> elsewhere.
>>
>> I disagree on this. It goes against the argument order of casts in other
>> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
>> x(int).
>
> However, in SQL, which is the only language I know of with an with an
> explicitly named cast function (unless you count C++'s
> static_cast<T>(foo) which isn't normal function syntax), the syntax is
> CAST(foo AS type). So there's definitely a precedent (arguably
> stronger, depending on your background) for cast(x, T).

I would *hope* that some stronger argument is used than "SQL puts it
this way". SQL has so many bizarre oddities of both syntax and
semantics that it's seldom a good guideline.

In this case, the isinstance() parallel is a better argument than the
SQL cast, but it's a small point and I could easily be tipped back to
preferring cast(T, x).

ChrisA

From mertz at gnosis.cx  Tue Mar 24 01:03:47 2015
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 23 Mar 2015 17:03:47 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150323230806.7a0653b9@fsol>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
 <CAPJVwBktkezjnkXTPK7KgtHz6bH3NyFGraa35EQv8OUvnaJ-DA@mail.gmail.com>
 <20150323230806.7a0653b9@fsol>
Message-ID: <CAEbHw4agi-6KeUKCnsAD4KB6Ez-0Vx98ZmkWP83yrEuU24C57Q@mail.gmail.com>

For what it's worth, 'cast(x, T)' also reads far more naturally to me.

On Mon, Mar 23, 2015 at 3:08 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 23 Mar 2015 14:17:20 -0700
> Nathaniel Smith <njs at pobox.com> wrote:
>
> > On Mar 23, 2015 1:03 PM, "Guido van Rossum" <guido at python.org> wrote:
> > >
> > > On Sun, Mar 22, 2015 at 11:02 PM, David Foster <davidfstr at gmail.com>
> > wrote:
> > >>
> > >> * +1 for the argument order cast(x, T). This is consistent with (x: T)
> > elsewhere.
> > >
> > >
> > > I disagree on this. It goes against the argument order of casts in
> other
> > languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
> > x(int).
> >
> > I don't have any strong opinion here, but I don't find the consistency
> > argument convincing. In int(x), 'int' is the verb, and in English verbs
> > come before undergoers ("I am inting the x"). In cast(...), though, cast
> is
> > the verb, x remains the undergoer, and int is conceptualized as the
> > destination (or something like that), and destinations go in
> prepositional
> > clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm
> > throwing the ball to Sarah"). "I'm casting to int the x" is extremely
> weird.
>
> I agree with cast(x, T). It also nicely mirrors isinstance(x, T).
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150323/db6b2828/attachment.html>

From cs at zip.com.au  Tue Mar 24 01:11:02 2015
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 24 Mar 2015 11:11:02 +1100
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAPTjJmqOP1mWbfdHtnZiFY=8Kn4R0CE9krz3aijzSrmq4_w7AA@mail.gmail.com>
References: <CAPTjJmqOP1mWbfdHtnZiFY=8Kn4R0CE9krz3aijzSrmq4_w7AA@mail.gmail.com>
Message-ID: <20150324001102.GA77518@cskk.homeip.net>

On 24Mar2015 10:56, Chris Angelico <rosuav at gmail.com> wrote:
>On Tue, Mar 24, 2015 at 8:05 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 23 March 2015 at 19:55, Guido van Rossum <guido at python.org> wrote:
>>>> * +1 for the argument order cast(x, T). This is consistent with (x: T)
>>>> elsewhere.
>>>
>>> I disagree on this. It goes against the argument order of casts in other
>>> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not
>>> x(int).

Well, in C you write (T)x, but I would expect that to be a syntactic 
requirement. Had it been framed as a function call I would not find cast(x, T) 
surprising at all.

>> However, in SQL, which is the only language I know of with an with an
>> explicitly named cast function [...] the syntax is
>> CAST(foo AS type). [...]
>> stronger, depending on your background) for cast(x, T).
>
>I would *hope* that some stronger argument is used than "SQL puts it
>this way".

Shrug. It is a consistency argument. And language like SQL and (gasp) COBOL 
strive in a sense for English prose friendliness.

>In this case, the isinstance() parallel is a better argument than the
>SQL cast, but it's a small point and I could easily be tipped back to
>preferring cast(T, x).

I would not. Leaving aside object method calls (fp.write(blah)), usually the 
thing acted upon or through is the first function agrument:

  write(fd, data, size)
  find(string, substring)
  heapsort(mylist)
  frob(widget, frobbificationness)

I also like the instance() argument.

For the record, I am +1 for cast(x, T).

Cheers,
Cameron Simpson <cs at zip.com.au>

Artificial intelligence won't make a micrometer out of a monkeywrench.
        - Rick Gordon <rickg at crl.com>

From random832 at fastmail.us  Tue Mar 24 02:54:29 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Mon, 23 Mar 2015 21:54:29 -0400
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
Message-ID: <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>

On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote:
> The type of the return value should
> be a concrete type - probably type(p1)?

I'd argue it should be the common supertype, so a PurePosixPath if both
are posix and one is pure, a Path or PurePath if one is windows and the
other is posix.

From greg.ewing at canterbury.ac.nz  Tue Mar 24 03:58:46 2015
From: greg.ewing at canterbury.ac.nz (Greg)
Date: Tue, 24 Mar 2015 15:58:46 +1300
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <20150323230806.7a0653b9@fsol>
References: <mailman.951.1426962635.10326.python-ideas@python.org>
 <550FAC80.8040405@gmail.com>
 <CAP7+vJLpTKKDHebJ-wJ2TjENxb0SXAnXiyqLyCg81K42L7nyhw@mail.gmail.com>
 <CAPJVwBktkezjnkXTPK7KgtHz6bH3NyFGraa35EQv8OUvnaJ-DA@mail.gmail.com>
 <20150323230806.7a0653b9@fsol>
Message-ID: <5510D2E6.2030802@canterbury.ac.nz>

On 24/03/2015 11:08 a.m., Antoine Pitrou wrote:

> I agree with cast(x, T). It also nicely mirrors isinstance(x, T).

I think that cast(T, x) is more readable when x is a long expression.
It puts the type next to the thing using or being assigned the result,
making it easier to visually verify that the types match up.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Mar 24 03:59:41 2015
From: greg.ewing at canterbury.ac.nz (Greg)
Date: Tue, 24 Mar 2015 15:59:41 +1300
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <CAP7+vJK0dONCovkjVbr7iFyuo24z1h-_urvKkvv=K8zG4LvagQ@mail.gmail.com>
References: <meokfb$2ri$1@ger.gmane.org>
 <20150323214145.GA94958@cskk.homeip.net>
 <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
 <CAP7+vJK0dONCovkjVbr7iFyuo24z1h-_urvKkvv=K8zG4LvagQ@mail.gmail.com>
Message-ID: <5510D31D.3080307@canterbury.ac.nz>

On 24/03/2015 11:35 a.m., Guido van Rossum wrote:
> when you use an
> if-expression, there's no way to shut it up!

Maybe an if-expression should assume the union type, then?

-- 
Greg


From p.f.moore at gmail.com  Tue Mar 24 11:05:17 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 10:05:17 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
Message-ID: <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>

On 24 March 2015 at 01:54,  <random832 at fastmail.us> wrote:
> On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote:
>> The type of the return value should
>> be a concrete type - probably type(p1)?
>
> I'd argue it should be the common supertype, so a PurePosixPath if both
> are posix and one is pure, a Path or PurePath if one is windows and the
> other is posix.

That's not really possible in the face of the possibility that an
argument could be a user-defined class, possibly not even a
pathlib.Path subclass (given duck typing). That's a clear benefit of a
Path method, actually - the type of the return value is easy to
specify - it's the type of self.

Actually, in many ways, this is really a list (sequence) method -
common_prefix - applied to the "parts" property of a Path. It's a
shame there isn't a sequence utils module in the stdlib...

One thing my implementation doesn't (yet) handle is case sensitivity.
The common prefix of WindowsPath('c:\\FOO\\bar') and
WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not
for PosixPath. (And again, when they are mixed, which is silly but
possible, what behaviour should apply? "Work like self" is the obvious
answer if we have a method).

Paul

From probiner at gmail.com  Tue Mar 24 11:31:12 2015
From: probiner at gmail.com (pedro santos)
Date: Tue, 24 Mar 2015 10:31:12 +0000
Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom
 slicing rules?
In-Reply-To: <CAEbHw4ZhQGq4Um2nLCbgLZ=TYxCFJOQ3P3xrdNQS_4=r2G+JzQ@mail.gmail.com>
References: <CAMrmD0r8fiHNzu+_a_bWYaAw+3opnWW1z+4fPTnjdrtQq_1whQ@mail.gmail.com>
 <CAPTjJmre-mGrdEZY4zVPtvjZTqbLUr3dK3Pu3=n5bEpVkvBHmw@mail.gmail.com>
 <men2t9$fu1$1@ger.gmane.org>
 <CAEbHw4ZhQGq4Um2nLCbgLZ=TYxCFJOQ3P3xrdNQS_4=r2G+JzQ@mail.gmail.com>
Message-ID: <CAMrmD0ro=NY07hhsObU4yGR1t4Tr_FNBKbAg-JJazAJOOtA4AA@mail.gmail.com>

*Newbie slowly steps down from the table*

I guess then 1-indexing in most cases the it's something relegated to
fleshing, so that the index offset is done on the user level and not at the
code level. Yes because no one will pick a gambling dice with face 0 :P
I guess it's a matter of brain education... I did fell I needed to run a
bunch of tests just to re-assure myself about the 1-indexed problem being
solved with a 0-base representation. But I totally understand the arguments
presented here as code consistency also makes it more malleable and
somewhat more black-boxy. As someone that deals a lot with vectors with
linear color workflows I know that what could be a clean work environment
can be become a minefield of conversion routines that add an overhead to
the brain when trying to operate Frankenstein operations into data flow
setups.

Thanks for the extra discussion, especially when indexing was addressed
separately from slicing routines.

Cheers

On Sun, Mar 22, 2015 at 7:52 PM, David Mertz <mertz at gnosis.cx> wrote:

> The strongest proof that mixed index bases are a bad idea is that this is
> a valid statement in VB6:
>
>   Dim array(1 to 10) as Integer
>
> For all X, if VB6 allows X, then X is a bad idea. :-)
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150324/ed9e656f/attachment.html>

From p.f.moore at gmail.com  Tue Mar 24 12:56:25 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 11:56:25 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
Message-ID: <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>

On 23 March 2015 at 21:33, Gregory P. Smith <greg at krypto.org> wrote:
> +1 pathlib would be the appropriate place for the correctly behaving
> function to appear.

OK, so here's a question. What actual use cases exist for a
common_prefix function? The reason I ask is that I'm looking at some
of the edge cases, and the obvious behaviour isn't particularly clear
to me.

For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
wanting the common *directory* containing both files. But taken
literally, that's only possible if you check the filesystem, so it
would no longer be a PurePath operation.

And what about common_prefix('foo/bar', '../here/foo')? Or
common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing
.. because the meaning could change in the face of symlinks. I believe
the same applies here. Maybe you need to call resolve() before doing
the common prefix operation (but that gives an absolute path).

With the above limitations, would a common_prefix function actually
help typical use cases? In my experience, doing list operations on
pathobj.parts is often simple enough that I don't need specialised
functions like common_prefix...

Getting the edge cases right is fiddly enough for common_prefix that a
specialised function is a reasonable idea, but only if the "obvious"
behaviour is clear. If there's a lot of conflicting possibilities,
maybe a recipe in the docs would be a better option.

Paul

From p.f.moore at gmail.com  Tue Mar 24 13:08:25 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 12:08:25 +0000
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather
	than strings?
Message-ID: <CACac1F_5v6o+uVsymq_q6Xy_-ND=_NqZWXLDsiDVcGZQ3g6VkQ@mail.gmail.com>

One occasional problem I have with pathlib.Path.parts is that it
returns a tuple of strings. For WindowsPath objects, paths are case
insensitive but strings aren't. So we get the situation that

>>> p1 = pathlib.PureWindowsPath('foo')
>>> p2 = pathlib.PureWindowsPath('FOO')
>>> p1 == p2
True
>>> p1.parts == p2.parts
False

Making p1.parts return a tuple of objects of type p1.__class__ would
address this:

>>> [p1.__class__(p) for p in p1.parts] == [p2.__class__(p) for p in p2.parts]
True

Is this something that's worth changing (pathlib is still provisional,
so the backward compatibility break would technically be allowed, even
if it's not ideal)? Or would an extra property, say p.path_parts that
returned a tuple of path objects be worth considering?

As it is, use of path.parts is a potential source of portability bugs,
where Unix users don't consider the case sensitivity issue. (Heck, as
a Windows user, I almost missed the problem in my implementation of
common_prefix, so it's not just Unix users...)

Paul

From markus at unterwaditzer.net  Tue Mar 24 13:43:57 2015
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Tue, 24 Mar 2015 13:43:57 +0100
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects
	rather than strings?
In-Reply-To: <CACac1F9ckKHZ6Dbc+ZwUEeSgzO2Y7NhmrWSpycy4N2QWcB9ZMA@mail.gmail.com>
References: <CACac1F_5v6o+uVsymq_q6Xy_-ND=_NqZWXLDsiDVcGZQ3g6VkQ@mail.gmail.com>
 <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
 <CACac1F9ckKHZ6Dbc+ZwUEeSgzO2Y7NhmrWSpycy4N2QWcB9ZMA@mail.gmail.com>
Message-ID: <85174692-228C-4725-8972-3B81350FF136@unterwaditzer.net>

No, I always forget. I've forwarded the message.

I understand that the current behavior creates bugs too, however, I'd say the drawbacks I described are much bigger.

The next option would be to create a separate type for path segments which has the appropriate comparison behavior: PathSegment, or more generically, PathString.

On 24 March 2015 13:37:34 CET, Paul Moore <p.f.moore at gmail.com> wrote:
>On 24 March 2015 at 12:25, Markus Unterwaditzer
><markus at unterwaditzer.net> wrote:
>> I'd say treating each part as full-blown Path doesn't make sense from
>a semantical POV.
>>
>> Simply mapping Path over the results is also a potential source of
>bugs, since each part is now a relative path to the CWD, instead of the
>previous segment.
>
>Fair point. Although I tend to view them as just "relative" - to what
>depends on context (in this case, "to the preceding bits in the list"
>is implied). But I agree, people may not be expecting that and so
>could make mistakes.
>
>I still think using strings is equally a bug magnet, though. There's
>actually no clean way of getting the correct case sensitivity
>behaviour *apart* from making the parts into path objects.
>
>I don't think your suggestion of each part being a prefix of the next
>would actually help in any of the places I've ever used .parts.
>
>Paul
>
>PS BTW, did you mean to send to just me rather than the list? I'd like
>to know what other people think of your point.


From markus at unterwaditzer.net  Tue Mar 24 13:39:55 2015
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Tue, 24 Mar 2015 13:39:55 +0100
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects
	rather	than strings?
In-Reply-To: <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
References: <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
Message-ID: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net>

Forgot to reply to list!


-------- Original Message --------
From: Markus Unterwaditzer <markus at unterwaditzer.net>
Sent: 24 March 2015 13:25:27 CET
To: Paul Moore <p.f.moore at gmail.com>
Subject: Re: [Python-ideas] pathlib.Path.parts - tuple of path objects rather	than strings?

I'd say treating each part as full-blown Path doesn't make sense from a semantical POV.

Simply mapping Path over the results is also a potential source of bugs, since each part is now a relative path to the CWD, instead of the previous segment.

I'd suggest that an additional property is added where:

Path("/usr/bin/true").newproperty == (Path("/"),
 Path("/usr"),
 Path("/usr/bin"),
 Path("/usr/bin/true"))

I.e. each part is a prefix of the next one.

But I'm not sure where you'd use that.

-- Markus

On 24 March 2015 13:08:25 CET, Paul Moore <p.f.moore at gmail.com> wrote:
>One occasional problem I have with pathlib.Path.parts is that it
>returns a tuple of strings. For WindowsPath objects, paths are case
>insensitive but strings aren't. So we get the situation that
>
>>>> p1 = pathlib.PureWindowsPath('foo')
>>>> p2 = pathlib.PureWindowsPath('FOO')
>>>> p1 == p2
>True
>>>> p1.parts == p2.parts
>False
>
>Making p1.parts return a tuple of objects of type p1.__class__ would
>address this:
>
>>>> [p1.__class__(p) for p in p1.parts] == [p2.__class__(p) for p in
>p2.parts]
>True
>
>Is this something that's worth changing (pathlib is still provisional,
>so the backward compatibility break would technically be allowed, even
>if it's not ideal)? Or would an extra property, say p.path_parts that
>returned a tuple of path objects be worth considering?
>
>As it is, use of path.parts is a potential source of portability bugs,
>where Unix users don't consider the case sensitivity issue. (Heck, as
>a Windows user, I almost missed the problem in my implementation of
>common_prefix, so it's not just Unix users...)
>
>Paul
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/


From random832 at fastmail.us  Tue Mar 24 14:36:23 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Tue, 24 Mar 2015 09:36:23 -0400
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
Message-ID: <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com>

On Tue, Mar 24, 2015, at 06:05, Paul Moore wrote:
> One thing my implementation doesn't (yet) handle is case sensitivity.
> The common prefix of WindowsPath('c:\\FOO\\bar') and
> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not
> for PosixPath. (And again, when they are mixed, which is silly but
> possible, what behaviour should apply? "Work like self" is the obvious
> answer if we have a method).

So, speaking of windows path oddities... what should be done for paths
where one has a drive and the other does not? No common prefix?
Interpret as the current drive? Do the former for pure paths and the
latter for concrete paths? Pass in an option?

Keeping in mind that people may write "portable" code but put in no
effort to specifically support windows.

> And again, when they are mixed, which is silly but possible, what behaviour should apply

For concrete paths, using the actual names of the files if they exist
would be another option.

From p.f.moore at gmail.com  Tue Mar 24 14:37:13 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 13:37:13 +0000
Subject: [Python-ideas] Fwd: pathlib.Path.parts - tuple of path objects
 rather than strings?
In-Reply-To: <CACac1F9ckKHZ6Dbc+ZwUEeSgzO2Y7NhmrWSpycy4N2QWcB9ZMA@mail.gmail.com>
References: <CACac1F_5v6o+uVsymq_q6Xy_-ND=_NqZWXLDsiDVcGZQ3g6VkQ@mail.gmail.com>
 <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
 <CACac1F9ckKHZ6Dbc+ZwUEeSgzO2Y7NhmrWSpycy4N2QWcB9ZMA@mail.gmail.com>
Message-ID: <CACac1F8JEtvOPePo_A3euzTZ6_aLU4j783uzS2GXCPQqtkJmag@mail.gmail.com>

For completeness, here's my reply that got sent off-list...

On 24 March 2015 at 12:25, Markus Unterwaditzer
<markus at unterwaditzer.net> wrote:
> I'd say treating each part as full-blown Path doesn't make sense from a semantical POV.
>
> Simply mapping Path over the results is also a potential source of bugs, since each part is now a relative path to the CWD, instead of the previous segment.

Fair point. Although I tend to view them as just "relative" - to what
depends on context (in this case, "to the preceding bits in the list"
is implied). But I agree, people may not be expecting that and so
could make mistakes.

I still think using strings is equally a bug magnet, though. There's
actually no clean way of getting the correct case sensitivity
behaviour *apart* from making the parts into path objects.

I don't think your suggestion of each part being a prefix of the next
would actually help in any of the places I've ever used .parts.

Paul

PS BTW, did you mean to send to just me rather than the list? I'd like
to know what other people think of your point.

From random832 at fastmail.us  Tue Mar 24 14:41:51 2015
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Tue, 24 Mar 2015 09:41:51 -0400
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
Message-ID: <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com>

On Tue, Mar 24, 2015, at 07:56, Paul Moore wrote:
> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
> prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
> wanting the common *directory* containing both files. But taken
> literally, that's only possible if you check the filesystem, so it
> would no longer be a PurePath operation.

Or you could _always_ reject the last component. That is, even if
"file.c" is a directory, return the directory it is in rather than
itself.

Does parts differentiate between "a/b/c" and "a/b/c/"? There are
certainly contexts where real filesystems differentiate between them
(symlinks, for example).

From p.f.moore at gmail.com  Tue Mar 24 14:43:44 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 13:43:44 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
 <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com>
Message-ID: <CACac1F8aywnH35KPjkW0w2S0eOOWA0qD8Bbu_+bckrvuoaVfiQ@mail.gmail.com>

On 24 March 2015 at 13:36,  <random832 at fastmail.us> wrote:
> On Tue, Mar 24, 2015, at 06:05, Paul Moore wrote:
>> One thing my implementation doesn't (yet) handle is case sensitivity.
>> The common prefix of WindowsPath('c:\\FOO\\bar') and
>> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not
>> for PosixPath. (And again, when they are mixed, which is silly but
>> possible, what behaviour should apply? "Work like self" is the obvious
>> answer if we have a method).
>
> So, speaking of windows path oddities... what should be done for paths
> where one has a drive and the other does not? No common prefix?
> Interpret as the current drive? Do the former for pure paths and the
> latter for concrete paths? Pass in an option?

It's worth pointing out that all of these edge cases only occur for
relative paths. It's quite possible that the only actual use cases
would be completely fine if the operation was only defined on absolute
paths.

My instinct says you do this *purely* based on the common prefix of
path.parts. The behaviour is then easy to define, and if it's not
precisely what someone wants they can convert the paths to absolute
and do the operations with that. I'd have to see real use cases to
justify anything else.

> Keeping in mind that people may write "portable" code but put in no
> effort to specifically support windows.

Precisely :-)

>> And again, when they are mixed, which is silly but possible, what behaviour should apply
>
> For concrete paths, using the actual names of the files if they exist
> would be another option.

You can't have mixed concrete paths, you can only ever instantiate one
of PosixPath or WindowsPath on a given system, AIUI.
Paul

From taleinat at gmail.com  Tue Mar 24 14:44:39 2015
From: taleinat at gmail.com (Tal Einat)
Date: Tue, 24 Mar 2015 15:44:39 +0200
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects
 rather than strings?
In-Reply-To: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net>
References: <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
 <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net>
Message-ID: <CALWZvp7QN49rJQbKvpcn67eaRPM8zCxZ9u=X9pNioVqS7UGJ3g@mail.gmail.com>

On Tue, Mar 24, 2015 at 2:39 PM, Markus Unterwaditzer
<markus at unterwaditzer.net> wrote:
>
> I'd suggest that an additional property is added where:
>
> Path("/usr/bin/true").newproperty == (Path("/"),
>  Path("/usr"),
>  Path("/usr/bin"),
>  Path("/usr/bin/true"))
>
> I.e. each part is a prefix of the next one.

Path.parents already exists:

Python 3.4.2 (default, Feb 23 2015, 21:16:28)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> list(pathlib.Path("/usr/bin/true").parents)
[PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')]

- Tal Einat

From p.f.moore at gmail.com  Tue Mar 24 14:45:37 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 13:45:37 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
 <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com>
Message-ID: <CACac1F8-0skaJcDkBTt9uvcNYpMF3AskBXvnwU=2bdefweWDXA@mail.gmail.com>

On 24 March 2015 at 13:41,  <random832 at fastmail.us> wrote:
> On Tue, Mar 24, 2015, at 07:56, Paul Moore wrote:
>> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
>> prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
>> wanting the common *directory* containing both files. But taken
>> literally, that's only possible if you check the filesystem, so it
>> would no longer be a PurePath operation.
>
> Or you could _always_ reject the last component. That is, even if
> "file.c" is a directory, return the directory it is in rather than
> itself.

I'm pretty sure there would be cases where that's not right. For
example, commonprefix('a/b', 'a/b/file.c') would be 'a'. That seems
wrong.

> Does parts differentiate between "a/b/c" and "a/b/c/"? There are
> certainly contexts where real filesystems differentiate between them
> (symlinks, for example).

No, it doesn't:

>>> pathlib.Path('a/b/').parts
('a', 'b')
>>> pathlib.Path('a/b').parts
('a', 'b')

Paul

From abarnert at yahoo.com  Tue Mar 24 14:44:16 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 06:44:16 -0700
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
Message-ID: <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com>

On Mar 24, 2015, at 3:05 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 24 March 2015 at 01:54,  <random832 at fastmail.us> wrote:
>>> On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote:
>>> The type of the return value should
>>> be a concrete type - probably type(p1)?
>> 
>> I'd argue it should be the common supertype, so a PurePosixPath if both
>> are posix and one is pure, a Path or PurePath if one is windows and the
>> other is posix.
> 
> That's not really possible in the face of the possibility that an
> argument could be a user-defined class, possibly not even a
> pathlib.Path subclass (given duck typing). That's a clear benefit of a
> Path method, actually - the type of the return value is easy to
> specify - it's the type of self.
> 
> Actually, in many ways, this is really a list (sequence) method -
> common_prefix - applied to the "parts" property of a Path. It's a
> shame there isn't a sequence utils module in the stdlib...

That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too:

    def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]:
        for a, b in zip(x, y):
            if a != b: return
            yield a

(Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.)

If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that.

> One thing my implementation doesn't (yet) handle is case sensitivity.
> The common prefix of WindowsPath('c:\\FOO\\bar') and
> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo').

Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self.

> But not
> for PosixPath. (And again, when they are mixed, which is silly but
> possible, what behaviour should apply? "Work like self" is the obvious
> answer if we have a method).

Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows.

From p.f.moore at gmail.com  Tue Mar 24 14:51:30 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 13:51:30 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
 <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com>
Message-ID: <CACac1F-XhyRUZpXcr+UtFMsAB1LZMV=H+14Vc9=rp3jZgX+maQ@mail.gmail.com>

On 24 March 2015 at 13:44, Andrew Barnert <abarnert at yahoo.com> wrote:
>> Actually, in many ways, this is really a list (sequence) method -
>> common_prefix - applied to the "parts" property of a Path. It's a
>> shame there isn't a sequence utils module in the stdlib...
>
> That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too:
>
>     def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]:
>         for a, b in zip(x, y):
>             if a != b: return
>             yield a
>
> (Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.)
>
> If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that.

I was thinking that there might be a reason it wouldn't work for
arbitrary iterators, so you'd need at least a Sequence. But you're
right, an itertool is sufficient. Although given the itertools focus
on building blocks, it may end up being simply a recipe.

>> One thing my implementation doesn't (yet) handle is case sensitivity.
>> The common prefix of WindowsPath('c:\\FOO\\bar') and
>> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo').
>
> Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self.

Technically, WindowsPath('C:\\FOO') and WindowsPath('C:\\Foo') are the
same, so I stand by what I said :-) But yeah, the natural
implementation would give you the relevant part of self.

>> But not
>> for PosixPath. (And again, when they are mixed, which is silly but
>> possible, what behaviour should apply? "Work like self" is the obvious
>> answer if we have a method).
>
> Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows.

(p.__class__(pp) for pp in p.parts)

(See the thread about parts containing strings - technically using
Path objects is dodgy, but as long as you don't leak the working
values out of your function it's perfectly adequate).

Paul

From abarnert at yahoo.com  Tue Mar 24 14:55:18 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 06:55:18 -0700
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
Message-ID: <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>

On Mar 24, 2015, at 4:56 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 23 March 2015 at 21:33, Gregory P. Smith <greg at krypto.org> wrote:
>> +1 pathlib would be the appropriate place for the correctly behaving
>> function to appear.
> 
> OK, so here's a question. What actual use cases exist for a
> common_prefix function? The reason I ask is that I'm looking at some
> of the edge cases, and the obvious behaviour isn't particularly clear
> to me.
> 
> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
> prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
> wanting the common *directory* containing both files. But taken
> literally, that's only possible if you check the filesystem, so it
> would no longer be a PurePath operation.

The traditional way to handle this is that the basename (the part after the last '/') is assumed to be a file (if you don't want that, include the trailing slash). POSIX even defines the technical term "path prefix" to mean everything up to the last slash, so something called a "common path prefix" sounds like it should be the common prefix of the path prefixes, right? Except that not command and function in POSIX works this way, requiring you to memorize or look up the man page to see what someone chose as "obvious" back in the 1970s....

At any rate, we probably don't need to figure this out from first principles; I'm pretty sure some subset of {Java, Boost, Cocoa, .NET, JUCE, one overwhelming popular CPAN library, etc.} have already come up with an answer, and if most of them agree, we probably want to follow suit (even if it seems silly).

> And what about common_prefix('foo/bar', '../here/foo')? Or
> common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing
> .. because the meaning could change in the face of symlinks. I believe
> the same applies here. Maybe you need to call resolve() before doing
> the common prefix operation (but that gives an absolute path).
> 
> With the above limitations, would a common_prefix function actually
> help typical use cases? In my experience, doing list operations on
> pathobj.parts is often simple enough that I don't need specialised
> functions like common_prefix...
> 
> Getting the edge cases right is fiddly enough for common_prefix that a
> specialised function is a reasonable idea, but only if the "obvious"
> behaviour is clear. If there's a lot of conflicting possibilities,
> maybe a recipe in the docs would be a better option.
> 
> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Tue Mar 24 15:07:30 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 07:07:30 -0700
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F-XhyRUZpXcr+UtFMsAB1LZMV=H+14Vc9=rp3jZgX+maQ@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
 <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com>
 <CACac1F-XhyRUZpXcr+UtFMsAB1LZMV=H+14Vc9=rp3jZgX+maQ@mail.gmail.com>
Message-ID: <B6786BE5-5033-4991-B4F1-9B66E5958CB7@yahoo.com>

On Mar 24, 2015, at 6:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> On 24 March 2015 at 13:44, Andrew Barnert <abarnert at yahoo.com> wrote:
>>> Actually, in many ways, this is really a list (sequence) method -
>>> common_prefix - applied to the "parts" property of a Path. It's a
>>> shame there isn't a sequence utils module in the stdlib...
>> 
>> That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too:
>> 
>>    def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]:
>>        for a, b in zip(x, y):
>>            if a != b: return
>>            yield a
>> 
>> (Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.)
>> 
>> If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that.
> 
> I was thinking that there might be a reason it wouldn't work for
> arbitrary iterators, so you'd need at least a Sequence. But you're
> right, an itertool is sufficient. Although given the itertools focus
> on building blocks, it may end up being simply a recipe.
> 
>>> One thing my implementation doesn't (yet) handle is case sensitivity.
>>> The common prefix of WindowsPath('c:\\FOO\\bar') and
>>> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo').
>> 
>> Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self.
> 
> Technically, WindowsPath('C:\\FOO') and WindowsPath('C:\\Foo') are the
> same, so I stand by what I said :-) But yeah, the natural
> implementation would give you the relevant part of self.
> 
>>> But not
>>> for PosixPath. (And again, when they are mixed, which is silly but
>>> possible, what behaviour should apply? "Work like self" is the obvious
>>> answer if we have a method).
>> 
>> Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows.
> 
> (p.__class__(pp) for pp in p.parts)

Sure, but then your whole expression looks something like:

    p1.__class__(*more_itertools.common_prefix(
        (p1.__class__(pp) for pp in p1.parts),
        (p2.__class__(pp) for pp in p2.parts)))

Which doesn't read quite as nicely as "just call an itertools function on the parts and construct a Path from them" sounds like it should.

Which implies that you'd probably want at least a recipe in the pathlib docs that referenced the recipe in the itertools docs or something.

And that many people who aren't on Windows just wouldn't bother and would write something non-portable until they got a complaint from a Windows user and found it worth investigating...

(While we're at it: most POSIX OS's can handle both case-sensitive and case-insensitive filesystems, and at least some OS X functions take that into account, although that may not be true at the BSD level, only at the POSIX level. For that matter, doesn't the HFS+ filesystem also consider two paths equal if they have the same NFKD, even if they have different code points? But I guess if I'm remembering right, this would be no more or less broken than any other use of PosixPath on Mac, so it's not worth worrying about here, right?)

> (See the thread about parts containing strings - technically using
> Path objects is dodgy, but as long as you don't leak the working
> values out of your function it's perfectly adequate).

Yeah, I agree that it's safe here even though it isn't safe in general.

From p.f.moore at gmail.com  Tue Mar 24 15:16:58 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 14:16:58 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <B6786BE5-5033-4991-B4F1-9B66E5958CB7@yahoo.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <20150323230937.0af5ee32@fsol>
 <CACac1F_B46tq+2FBNrJAEpA8yf=vDy7meOyvHE86+6kQfH-VAQ@mail.gmail.com>
 <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com>
 <CACac1F8R6Jy7b3mf-Urg80AeAL1C5hJzSfU=9zT0JRSB0GQaeQ@mail.gmail.com>
 <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com>
 <CACac1F-XhyRUZpXcr+UtFMsAB1LZMV=H+14Vc9=rp3jZgX+maQ@mail.gmail.com>
 <B6786BE5-5033-4991-B4F1-9B66E5958CB7@yahoo.com>
Message-ID: <CACac1F-w3t+Me9HjyOq5rfAd3WtSev0mEd25nBL+ZtT8w6rGEQ@mail.gmail.com>

On 24 March 2015 at 14:07, Andrew Barnert <abarnert at yahoo.com> wrote:
>>> Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows.
>>
>> (p.__class__(pp) for pp in p.parts)
>
> Sure, but then your whole expression looks something like:
>
>     p1.__class__(*more_itertools.common_prefix(
>         (p1.__class__(pp) for pp in p1.parts),
>         (p2.__class__(pp) for pp in p2.parts)))
>
> Which doesn't read quite as nicely as "just call an itertools function on the parts and construct a Path from them" sounds like it should.

Agreed, absolutely :-)

> Which implies that you'd probably want at least a recipe in the pathlib docs that referenced the recipe in the itertools docs or something.
>
> And that many people who aren't on Windows just wouldn't bother and would write something non-portable until they got a complaint from a Windows user and found it worth investigating...

I still think it's worth considering as a path object method, for
convenience. I just think the *semantics* should be that of the
equivalent list operation, because that's easily understandable. But
to an extent, that's the trap that os.path.commonprefix fell into
(although to a much worse level...) Hence my concern that we find some
real use cases for the operation, so that we can ensure that the list
semantics match what people actually want the operation to do.

> (While we're at it: most POSIX OS's can handle both case-sensitive and case-insensitive filesystems, and at least some OS X functions take that into account, although that may not be true at the BSD level, only at the POSIX level. For that matter, doesn't the HFS+ filesystem also consider two paths equal if they have the same NFKD, even if they have different code points? But I guess if I'm remembering right, this would be no more or less broken than any other use of PosixPath on Mac, so it's not worth worrying about here, right?)

This (and the decision to treat a/b/ and a/b as the same) are
decisions that have already been made, for better or worse, by
pathlib, and I have no intention of getting sucked into them here.
IMO, a common_prefix method on pathlib objects should follow pathlib
semantics, and can easily do that by being built on top of existing
pathlib operations and uncontroversial list operations. That's the
only approach that I think makes sense - and the only remaining
question is whether it meets a real-world requirement, or whether it
will end up being an oddity that no-one ever uses because it doesn't
*quite* do what they want.

Paul

From abarnert at yahoo.com  Tue Mar 24 15:44:40 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 07:44:40 -0700
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
 <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>
Message-ID: <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com>

On Mar 24, 2015, at 6:55 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
> 
>> On Mar 24, 2015, at 4:56 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> 
>>> On 23 March 2015 at 21:33, Gregory P. Smith <greg at krypto.org> wrote:
>>> +1 pathlib would be the appropriate place for the correctly behaving
>>> function to appear.
>> 
>> OK, so here's a question. What actual use cases exist for a
>> common_prefix function? The reason I ask is that I'm looking at some
>> of the edge cases, and the obvious behaviour isn't particularly clear
>> to me.
>> 
>> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
>> prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
>> wanting the common *directory* containing both files. But taken
>> literally, that's only possible if you check the filesystem, so it
>> would no longer be a PurePath operation.
> 
> The traditional way to handle this is that the basename (the part after the last '/') is assumed to be a file (if you don't want that, include the trailing slash). POSIX even defines the technical term "path prefix" to mean everything up to the last slash, so something called a "common path prefix" sounds like it should be the common prefix of the path prefixes, right? Except that not command and function in POSIX works this way, requiring you to memorize or look up the man page to see what someone chose as "obvious" back in the 1970s....

Sorry, forgot to fill in the cite. See 3.2.69 at http://pubs.opengroup.org/stage7tc1/basedefs/V1_chap03.html for the 2008 definition of "path prefix".

> At any rate, we probably don't need to figure this out from first principles; I'm pretty sure some subset of {Java, Boost, Cocoa, .NET, JUCE, one overwhelming popular CPAN library, etc.} have already come up with an answer, and if most of them agree, we probably want to follow suit (even if it seems silly).

From a quick search, it looks like many other languages don't define a common prefix path method/function, but many do define a generic iterable common-prefix function (or a first-mismatch function and random-access iterables so you can easily build one trivially, as in C++).

This implies that the obvious solution in most languages will include the basename, not skip it. And it looks like http://rosettacode.org/wiki/Find_common_directory_path agrees with that.

From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So:

    def common_prefix(*iterables):
        for first, *rest in zip(*iterables):
            if any(first != part for part in rest):
                return
            yield first

And of course to wrap it up:

    def common_path_prefix(*paths):
        return paths[0].__class__(
            *(common_prefix(*((path.__class__(p) for p in path.parts) for path in paths)))

Except I'll bet I got the parens wrong somewhere; it's probably clearer as:

    def common_path_prefix(*paths):
        parts = (map(path.__class__, path.parts) for path in paths)
        prefix = common_prefix(*parts)
        return paths[0].__class__(*prefix)

>> And what about common_prefix('foo/bar', '../here/foo')? Or
>> common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing
>> .. because the meaning could change in the face of symlinks. I believe
>> the same applies here. Maybe you need to call resolve() before doing
>> the common prefix operation (but that gives an absolute path).
>> 
>> With the above limitations, would a common_prefix function actually
>> help typical use cases? In my experience, doing list operations on
>> pathobj.parts is often simple enough that I don't need specialised
>> functions like common_prefix...
>> 
>> Getting the edge cases right is fiddly enough for common_prefix that a
>> specialised function is a reasonable idea, but only if the "obvious"
>> behaviour is clear. If there's a lot of conflicting possibilities,
>> maybe a recipe in the docs would be a better option.
>> 
>> Paul
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Tue Mar 24 15:56:06 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 07:56:06 -0700
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects
	rather than strings?
In-Reply-To: <CALWZvp7QN49rJQbKvpcn67eaRPM8zCxZ9u=X9pNioVqS7UGJ3g@mail.gmail.com>
References: <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
 <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net>
 <CALWZvp7QN49rJQbKvpcn67eaRPM8zCxZ9u=X9pNioVqS7UGJ3g@mail.gmail.com>
Message-ID: <D91ACDC0-D020-4A3F-9F0E-A9AEAA798FA1@yahoo.com>

On Mar 24, 2015, at 6:44 AM, Tal Einat <talent at gmail.com> wrote:
> 
> On Tue, Mar 24, 2015 at 2:39 PM, Markus Unterwaditzer
> <markus at unterwaditzer.net> wrote:
>> 
>> I'd suggest that an additional property is added where:
>> 
>> Path("/usr/bin/true").newproperty == (Path("/"),
>> Path("/usr"),
>> Path("/usr/bin"),
>> Path("/usr/bin/true"))
>> 
>> I.e. each part is a prefix of the next one.
> 
> Path.parents already exists:
> 
> Python 3.4.2 (default, Feb 23 2015, 21:16:28)
> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import pathlib
>>>> list(pathlib.Path("/usr/bin/true").parents)
> [PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')]

Ooh, forgot about that!

That implies a conceptually simpler implementation: it's just first(last(takewhile(all_equal(reversed(map(parents)))))), but I'm too tired to finish translating that from Haskelly terms to itertoolsy terms in my head. :) I'm not sure if that would be simpler (or more efficient) in practice, however.

Also, it would give the opposite answer to is the 'true' bit a potential part of the result if you common-prefix /usr/bin/true with itself (or only pass one argument), which goes nicely with my intuition but not so nicely with the rosettacode page. But maybe that's just a matter of naming--with "common_parent" or "common_ancestor" instead of "common_prefix" it seems pretty obvious and unambiguous?

  

From p.f.moore at gmail.com  Tue Mar 24 16:15:45 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 15:15:45 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
 <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>
 <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com>
Message-ID: <CACac1F_gu7_TWRoNUSLinS42wpU-Fo0NGt6a0Es1XbkmVkAYJg@mail.gmail.com>

On 24 March 2015 at 14:44, Andrew Barnert <abarnert at yahoo.com> wrote:
> From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So:
>
>     def common_prefix(*iterables):
>         for first, *rest in zip(*iterables):
>             if any(first != part for part in rest):
>                 return
>             yield first

Thanks for the research. This is probably something that could be
included as a recipe in the itertools documentation. (I doubt it would
be viewed as a common enough requirement to be added to the module
itself...) Do you have any objection to me submitting a doc patch with
this code? (It's similar to what I came up with, but a lot cleaner).

Paul

From abarnert at yahoo.com  Tue Mar 24 16:32:51 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Mar 2015 08:32:51 -0700
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F_gu7_TWRoNUSLinS42wpU-Fo0NGt6a0Es1XbkmVkAYJg@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
 <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>
 <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com>
 <CACac1F_gu7_TWRoNUSLinS42wpU-Fo0NGt6a0Es1XbkmVkAYJg@mail.gmail.com>
Message-ID: <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com>

On Mar 24, 2015, at 8:15 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 24 March 2015 at 14:44, Andrew Barnert <abarnert at yahoo.com> wrote:
>> From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So:
>> 
>>    def common_prefix(*iterables):
>>        for first, *rest in zip(*iterables):
>>            if any(first != part for part in rest):
>>                return
>>            yield first
> 
> Thanks for the research. This is probably something that could be
> included as a recipe in the itertools documentation. (I doubt it would
> be viewed as a common enough requirement to be added to the module
> itself...) Do you have any objection to me submitting a doc patch with
> this code? (It's similar to what I came up with, but a lot cleaner).

I agree that it belongs as a recipe in itertools (and of course it can go into the third-party more-itertools library on PyPI for people who want to just import and use it, as with the other useful recipes).

But you probably want to test it first, and make sure I thought through the stupid edge cases like no iterables or empty iterables, since that's just off the top of my head, and while tired and recovering from a cold to boot. :)

Also, while I think this generic function is useful on its own merits, it doesn't solve the problem that started this thread--as pointed out by (I forget who), just using it on the string parts gives the wrong answer for Windows paths. Should there also be a recipe in the pathlib docs referencing the itertools recipe? (Is it even reasonable for a recipe in one module's docs to rely on a recipe in another's?) Or maybe, as Tal (I think) suggested, a better solution is to avoid the problem and use parents instead of parts, meaning you don't even need a common_prefix recipe, but a last_initial_match (although that's just last(common_prefix(*args)), so maybe that doesn't really matter.)

From p.f.moore at gmail.com  Tue Mar 24 16:45:01 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 15:45:01 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
 <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com>
 <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com>
 <CACac1F_gu7_TWRoNUSLinS42wpU-Fo0NGt6a0Es1XbkmVkAYJg@mail.gmail.com>
 <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com>
Message-ID: <CACac1F8VxKum0HZZz0OEiaEVw6SNZp7ehFewcrcS8OSQSLxiFw@mail.gmail.com>

On 24 March 2015 at 15:32, Andrew Barnert <abarnert at yahoo.com> wrote:
> But you probably want to test it first, and make sure I thought through the stupid edge cases like no iterables or empty iterables, since that's just off the top of my head, and while tired and recovering from a cold to boot. :)

lol Understood. It's still cleaner than my one :-)

> Also, while I think this generic function is useful on its own merits, it doesn't solve the problem that started this thread--as pointed out by (I forget who), just using it on the string parts gives the wrong answer for Windows paths. Should there also be a recipe in the pathlib docs referencing the itertools recipe? (Is it even reasonable for a recipe in one module's docs to rely on a recipe in another's?) Or maybe, as Tal (I think) suggested, a better solution is to avoid the problem and use parents instead of parts, meaning you don't even need a common_prefix recipe, but a last_initial_match (although that's just last(common_prefix(*args)), so maybe that doesn't really matter.)

I still think there's a case for a pathlib.PurePath.common_prefix()
method, if we get consensus on the behaviour, so if we did that, I'd
probably grab the itertools recipe and use it as a private function in
pathlib. It feels mildly clumsy to have a private implementation of an
itertools recipe in a different stdlib module, but I don't think it's
a big enough deal to justify pushing for the function to be actually
added to itertools (which would involve rewriting in C, for a start).

Paul

From guido at python.org  Tue Mar 24 17:42:19 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Mar 2015 09:42:19 -0700
Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft
In-Reply-To: <5510D31D.3080307@canterbury.ac.nz>
References: <meokfb$2ri$1@ger.gmane.org>
 <20150323214145.GA94958@cskk.homeip.net>
 <CALGmxELBR3JrgPE7r=HMdRu=sEqNQ8YwXVRhY=fz-i6C-o=b-A@mail.gmail.com>
 <CAP7+vJK0dONCovkjVbr7iFyuo24z1h-_urvKkvv=K8zG4LvagQ@mail.gmail.com>
 <5510D31D.3080307@canterbury.ac.nz>
Message-ID: <CAP7+vJJEMpssA=WU68HcOfkTS2CqF3zDdqhReU9rCSdH4HHJxA@mail.gmail.com>

On Mon, Mar 23, 2015 at 7:59 PM, Greg <greg.ewing at canterbury.ac.nz> wrote:

> On 24/03/2015 11:35 a.m., Guido van Rossum wrote:
>
>> when you use an
>> if-expression, there's no way to shut it up!
>>
>
> Maybe an if-expression should assume the union type, then?
>

This was flagged as a mypy bug, but the resolution is still up in the air:
https://github.com/JukkaL/mypy/issues/622

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150324/4ad532a1/attachment-0001.html>

From taleinat at gmail.com  Tue Mar 24 19:00:37 2015
From: taleinat at gmail.com (Tal Einat)
Date: Tue, 24 Mar 2015 20:00:37 +0200
Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects
 rather than strings?
In-Reply-To: <D91ACDC0-D020-4A3F-9F0E-A9AEAA798FA1@yahoo.com>
References: <C21CB882-F5BB-40F2-932C-CE1128056EE9@unterwaditzer.net>
 <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net>
 <CALWZvp7QN49rJQbKvpcn67eaRPM8zCxZ9u=X9pNioVqS7UGJ3g@mail.gmail.com>
 <D91ACDC0-D020-4A3F-9F0E-A9AEAA798FA1@yahoo.com>
Message-ID: <CALWZvp7c8Nkg1k4+Cw5qv3+mq3kTucMmHXxXFvj9QudER-YO1Q@mail.gmail.com>

On Tue, Mar 24, 2015 at 4:56 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Mar 24, 2015, at 6:44 AM, Tal Einat <talent at gmail.com> wrote:
>> Path.parents already exists:
>>
>> Python 3.4.2 (default, Feb 23 2015, 21:16:28)
>> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import pathlib
>>>>> list(pathlib.Path("/usr/bin/true").parents)
>> [PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')]
>
> Ooh, forgot about that!
>
> That implies a conceptually simpler implementation: it's just first(last(takewhile(all_equal(reversed(map(parents)))))), but I'm too tired to finish translating that from Haskelly terms to itertoolsy terms in my head. :) I'm not sure if that would be simpler (or more efficient) in practice, however.
>
> Also, it would give the opposite answer to is the 'true' bit a potential part of the result if you common-prefix /usr/bin/true with itself (or only pass one argument), which goes nicely with my intuition but not so nicely with the rosettacode page. But maybe that's just a matter of naming--with "common_parent" or "common_ancestor" instead of "common_prefix" it seems pretty obvious and unambiguous?

Well, the function I wrote was "common_parent" since that was what I
needed, and what it does is indeed clearer than "common_prefix". Off
the top of my head, I can't think of use cases where one would want
"common_prefix" instead of "common_parent" where Path.relative_to()
wouldn't be better. So maybe just provide "common_parent"?

- Tal Einat

From storchaka at gmail.com  Tue Mar 24 19:01:18 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Mar 2015 20:01:18 +0200
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <20150321124732.0b47730e@fsol>
 <CALWZvp6G2tMLFEdmY0u6Ey9yzrLOUYFm1PDWPqJtpGNtv8FgWQ@mail.gmail.com>
 <CACac1F9GQPzB19RBP_At4pjOR28gyQ_JEbmm6A9Q72V6V2Bp+g@mail.gmail.com>
 <CAGE7PNLnZ_gfk=Zutp=jVO46PGN6TqF-5UuwmCNo2eBjo0xVvA@mail.gmail.com>
 <CACac1F-Fkx2ithAi=eh=Lkmr6iw_-_JfmkOAE0OUTQN-cnXDUA@mail.gmail.com>
Message-ID: <mes8pe$8gn$1@ger.gmane.org>

On 24.03.15 13:56, Paul Moore wrote:
> On 23 March 2015 at 21:33, Gregory P. Smith <greg at krypto.org> wrote:
>> +1 pathlib would be the appropriate place for the correctly behaving
>> function to appear.
>
> OK, so here's a question. What actual use cases exist for a
> common_prefix function? The reason I ask is that I'm looking at some
> of the edge cases, and the obvious behaviour isn't particularly clear
> to me.
>
> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common
> prefix is obviously 'a/b/file.c' - but I can imagine people *actually*
> wanting the common *directory* containing both files. But taken
> literally, that's only possible if you check the filesystem, so it
> would no longer be a PurePath operation.
>
> And what about common_prefix('foo/bar', '../here/foo')? Or
> common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing
> ... because the meaning could change in the face of symlinks. I believe
> the same applies here. Maybe you need to call resolve() before doing
> the common prefix operation (but that gives an absolute path).
>
> With the above limitations, would a common_prefix function actually
> help typical use cases? In my experience, doing list operations on
> pathobj.parts is often simple enough that I don't need specialised
> functions like common_prefix...
>
> Getting the edge cases right is fiddly enough for common_prefix that a
> specialised function is a reasonable idea, but only if the "obvious"
> behaviour is clear. If there's a lot of conflicting possibilities,
> maybe a recipe in the docs would be a better option.

Your are asking the same questions that were asked about 
os,path.commonpath(). You can look at discussions on the tracker, in 
Python-Dev and in Python-Ideas.

http://bugs.python.org/issue10395
http://mail.python.org/pipermail/python-dev/2000-July/005897.html
http://mail.python.org/pipermail/python-dev/2000-August/008385.html
http://comments.gmane.org/gmane.comp.python.ideas/17719


From storchaka at gmail.com  Tue Mar 24 19:06:09 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Mar 2015 20:06:09 +0200
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
Message-ID: <mes92h$csh$1@ger.gmane.org>

On 21.03.15 06:41, Paddy3118 wrote:
> I had to add a comment to an RC entry
> <http://rosettacode..org/wiki/Longest_common_prefix#Python>that was
> using os.path..commonprefix to compute the longest common prefix which
> it does regardless of path separators, (which is what it should do).
>
> It has been discussed before <http://bugs.python.org/issue10395>- it
> goes back to the nineties it seems, but we are still left with a
> misplaced function, (it should be str.commonprefix), with an awful hack
> of saying "yes we know its ballsed up" in the documentation
> <https://docs.python.org/3.5/library/os.path.html?highlight=commonprefix#os.path.commonprefix>.
>
> I guess we missed a great opportunity to fix this when we moved to
> Python 3 too!?
>
> The  fix seems clear: deprecate os.path.commonprefix whilst creating a
> true os.path.commonpath and str.commonprefix. The deprecated function
> should hang around and the standard libs modified to switch to the new
> function(s)
>
> I've heard that some religious people put obvious faults in their work
> as only their god should get so close to perfection - maybe that's why
> we still have this wart :-)

OK, if there are no objections and comments, I'll commit the recent 
patch for issue10395 (with only the doc change as Raymond suggested).



From p.f.moore at gmail.com  Tue Mar 24 19:14:51 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 24 Mar 2015 18:14:51 +0000
Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut.
In-Reply-To: <mes92h$csh$1@ger.gmane.org>
References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com>
 <mes92h$csh$1@ger.gmane.org>
Message-ID: <CACac1F9rvw0riUJUYm2H_Lwspm4CxyT5cwZb+5Ghfd3ZP3rkDw@mail.gmail.com>

On 24 March 2015 at 18:06, Serhiy Storchaka <storchaka at gmail.com> wrote:
> OK, if there are no objections and comments, I'll commit the recent patch
> for issue10395 (with only the doc change as Raymond suggested).

+1 from me.
Paul

From guettliml at thomas-guettler.de  Wed Mar 25 12:00:02 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Wed, 25 Mar 2015 12:00:02 +0100
Subject: [Python-ideas] OrderedDict.insert()
Message-ID: <55129532.8030502@thomas-guettler.de>

I am missing OrderedDict.insert()

Other people, too:

   http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python

Why was it not implemented up to now?

Why not implement it for the future?

Regards,
   Thomas G?ttler


From eliben at gmail.com  Wed Mar 25 13:42:40 2015
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 25 Mar 2015 05:42:40 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
Message-ID: <CAF-Rda-Ox-GM-JTvAt8EyjpDt2LpQRXBM4VfWoCu+zCapFeu4Q@mail.gmail.com>

On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky <eliben at gmail.com> wrote:

> Interesting coincidence :-)
>
> We (at Google) are just on the verge of releasing an open-source tool for
> auto-formatting of Python code. It should be posted to Github within the
> next few days or weeks, under a permissive license (Apache 2.0 or something
> similar).
>
> The tool works like gofmt and clang-format (the C++/JS version provided as
> part of the Clang toolchain) - it only acts on whitespace and performs
> re-formatting of code (PEP-8 oriented for Python but it's configurable). It
> does not rename variables/classes/functions to conform to any particular
> style. The algorithm is based on clang-format, and lib2to3 is used for the
> parsing side - so it's very solid. We use it on Python 2 code internally,
> but Python 3 support is also planned and should be easy.
>
> We intend to keep actively developing the tool in the open and will be
> happy to accept test cases, fixes and patches in general.
>
> So stay tuned!
>
>
https://github.com/google/yapf

Eli





















> P.S. I'm also against such a tool being a part of the Python distribution,
> at least in the near term. A tool/library has to first prove itself outside
> of core for a while, and only when/if it proves to be widely used and
> stable, an incorporation into core should be considered. The stdlib is big
> as it is, and any additional code is a burden on all core developers.
> Incorporation into the stdlib also imposes a very slow pace of
> development/release on said library/tool, and hence should only be
> considered when it's been extremely stable for a while.
>
> Eli
>
>
>
> On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas <
> nicholas.chammas at gmail.com> wrote:
>
>> First class means it's like a language feature. Actually, in my case I am
>> using "first-class" and "included with the language" synonymously. Not sure
>> if that's appropriate terminology, but that's what I mean.
>>
>> To explain by rough analogy, Twisted is to asyncio what autopep8 would be
>> to a first-class auto-styler. (I say "rough" because I understand that
>> Twisted is not supposed to be replaced entirely by asyncio, but I hope my
>> point is clear.)
>>
>> Nick
>>
>> On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>> On 03/17/2015 06:54 PM, Nicholas Chammas wrote:
>>>
>>> > I am not proposing putting |autopep8| in the standard library. I am
>>> proposing that an auto-styler be included with
>>> > Python as a first-class utility. The emphasis here is on /included/
>>> and /first-class/.
>>>
>>> Perhaps you could explain what you mean by a /first-class/ utility?
>>>
>>> --
>>> ~Ethan~
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150325/bb103b6f/attachment.html>

From markus at unterwaditzer.net  Wed Mar 25 14:08:39 2015
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Wed, 25 Mar 2015 14:08:39 +0100
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <55129532.8030502@thomas-guettler.de>
References: <55129532.8030502@thomas-guettler.de>
Message-ID: <8772702B-D0F7-4CBA-9B69-C0C8292D42F3@unterwaditzer.net>

As far as I can see, no concrete usecase is shown in the SO question. What is yours?

-- Markus

On 25 March 2015 12:00:02 CET, "Thomas G?ttler" <guettliml at thomas-guettler.de> wrote:
>I am missing OrderedDict.insert()
>
>Other people, too:
>
>http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python
>
>Why was it not implemented up to now?
>
>Why not implement it for the future?
>
>Regards,
>   Thomas G?ttler
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/


From tjreedy at udel.edu  Wed Mar 25 17:38:09 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 25 Mar 2015 12:38:09 -0400
Subject: [Python-ideas] gofmt for Python: standardized styling as a
	language feature
In-Reply-To: <CAF-Rda-Ox-GM-JTvAt8EyjpDt2LpQRXBM4VfWoCu+zCapFeu4Q@mail.gmail.com>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <CAF-Rda-Ox-GM-JTvAt8EyjpDt2LpQRXBM4VfWoCu+zCapFeu4Q@mail.gmail.com>
Message-ID: <meuoa3$evi$1@ger.gmane.org>

On 3/25/2015 8:42 AM, Eli Bendersky wrote:
>
>
> On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky
> <eliben at gmail.com
> <mailto:eliben at gmail.com>> wrote:
>
>     Interesting coincidence :-)
>
>     We (at Google) are just on the verge of releasing an open-source
>     tool for auto-formatting of Python code. It should be posted to
>     Github within the next few days or weeks, under a permissive license
>     (Apache 2.0 or something similar).
>
>     The tool works like gofmt and clang-format (the C++/JS version
>     provided as part of the Clang toolchain) - it only acts on
>     whitespace and performs re-formatting of code (PEP-8 oriented for
>     Python but it's configurable). It does not rename
>     variables/classes/functions to conform to any particular style. The
>     algorithm is based on clang-format, and lib2to3 is used for the
>     parsing side - so it's very solid. We use it on Python 2 code
>     internally, but Python 3 support is also planned and should be easy.
>
>     We intend to keep actively developing the tool in the open and will
>     be happy to accept test cases, fixes and patches in general.
>
>     So stay tuned!
>
>
> https://github.com/google/yapf

yet-another-python-formatter

The so-called pep8 style in style.py has

# The number of spaces required before a trailing comment.
SPACES_BEFORE_COMMENT=1,

PEP8 actually says "Inline comments should be separated by at least two 
spaces from the statement.", which is what you have in the google style.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Wed Mar 25 17:46:04 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 25 Mar 2015 12:46:04 -0400
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <55129532.8030502@thomas-guettler.de>
References: <55129532.8030502@thomas-guettler.de>
Message-ID: <meuoou$nid$1@ger.gmane.org>

On 3/25/2015 7:00 AM, Thomas G?ttler wrote:
> I am missing OrderedDict.insert()
>
> Other people, too:
>
>
> http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python

I just added this answer, which I suspect covers move use cases.
===
If you know you will want a 'c' key, but do not know the value, insert 
'c' with a dummy value when you create the dict.

d1 = OrderedDict([('c', None), ('a', '1'), ('b', '2')])

and change the value later.

d1['c'] = 3




-- 
Terry Jan Reedy



From abarnert at yahoo.com  Wed Mar 25 17:51:17 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 25 Mar 2015 09:51:17 -0700
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <55129532.8030502@thomas-guettler.de>
References: <55129532.8030502@thomas-guettler.de>
Message-ID: <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com>

On Mar 25, 2015, at 04:00, Thomas G?ttler <guettliml at thomas-guettler.de> wrote:
> 
> I am missing OrderedDict.insert()

With what interface? Does it take an index, like list.insert, even though other OrderedDict methods (including __getitem__ and friends) don't? Or does it take another key? If so, does it insert before or after that key? And what if that key doesn't exist? Meanwhile, if the new key was already in the dict, should it move to the new position, update values in place, or raise an exception? Do you expect O(1) performance?

> Other people, too:
> 
>  http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python

That other person is asking for some way to put a new element on the left instead of the right, which doesn't have any of those issues. (And which is already possible with move_to_end.) And an anonymous low-rep user asking a question, getting a handful of upvotes, and accepting an answer that says "You can't, just build a new dict" with no comment doesn't exactly show a burning desire within a wide community.

In fact, I think this does a better job showing how OrderedDict is often an attractive nuisance than how it's incomplete--the guy can't explain why he wants this, and meanwhile he apparently thinks you have to build a new dict and call update to add a value (or at least that doing so should have a different result than just using __setitem__ with that new key and value). My guess is that either he didn't really want an OrderedDict in the first place, or Zagorulkin Dmitri is right that the real problem is that his program needs this feature and it should be rewritten so it doesn't. But without actually knowing the use case that's just a guess.

> Why was it not implemented up to now?
> 
> Why not implement it for the future?
> 
> Regards,
>  Thomas G?ttler
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From ron3200 at gmail.com  Wed Mar 25 20:59:49 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Wed, 25 Mar 2015 15:59:49 -0400
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com>
References: <55129532.8030502@thomas-guettler.de>
 <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com>
Message-ID: <mev43m$ia$1@ger.gmane.org>



On 03/25/2015 12:51 PM, Andrew Barnert wrote:
> On Mar 25, 2015, at 04:00, Thomas G?ttler<guettliml at thomas-guettler.de>
> wrote:

>>> I am missing OrderedDict.insert()

> With what interface? Does it take an index, like list.insert, even
> though other OrderedDict methods (including __getitem__ and friends)
> don't? Or does it take another key? If so, does it insert before or
> after that key? And what if that key doesn't exist? Meanwhile, if the
> new key was already in the dict, should it move to the new position,
> update values in place, or raise an exception? Do you expect O(1)
> performance?

>>> Other people, too:

>>> http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python

> That other person is asking for some way to put a new element on the
> left instead of the right, which doesn't have any of those issues. (And
> which is already possible with move_to_end.) And an anonymous low-rep
> user asking a question, getting a handful of upvotes, and accepting an
> answer that says "You can't, just build a new dict" with no comment
> doesn't exactly show a burning desire within a wide community.

> In fact, I think this does a better job showing how OrderedDict is often
> an attractive nuisance than how it's incomplete--the guy can't explain
> why he wants this, and meanwhile he apparently thinks you have to build
> a new dict and call update to add a value (or at least that doing so
> should have a different result than just using __setitem__ with that new
> key and value). My guess is that either he didn't really want an
> OrderedDict in the first place, or Zagorulkin Dmitri is right that the
> real problem is that his program needs this feature and it should be
> rewritten so it doesn't. But without actually knowing the use case
> that's just a guess.

As to OrderedDict being a an attractive nuisance, I'm not sure. I do think 
there are data structures that benefit from being index-able by both 
position and by name.  Scheme uses association lists for this, and 
documents that they are a bit slower than mappings.  But it is easy to 
convert one to the other and back again.

In python you can convert to a list of items and back.  But it can be 
expensive to keep two versions of the same data in memory or to constantly 
convert back and forth.

An ordered dict insures the conversion to an items list will result in a 
dependable order.  That may be the primary benefit.

One way to do an insert is to mirror how the __init__ method works with 
items, but with an index for the first value.

     Dict.insert_items(index, items)

Another possibility is to have both get_items and set_items methods than 
accept an index or slice object as the first argument.

The __getitem__ and __setitem__ methods should not take an index in a 
dictionary. It is a dictionary first, and ordered data second.  If it was 
an association list, then this would be reversed.

Cheers,
   Ron


From jeanpierreda at gmail.com  Thu Mar 26 01:34:34 2015
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Wed, 25 Mar 2015 17:34:34 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <meuoa3$evi$1@ger.gmane.org>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <CAF-Rda-Ox-GM-JTvAt8EyjpDt2LpQRXBM4VfWoCu+zCapFeu4Q@mail.gmail.com>
 <meuoa3$evi$1@ger.gmane.org>
Message-ID: <CABicbJJJXftAATwoTbMH_Hpp0d0oosWGopfKs5FsgG5Wtc_2-w@mail.gmail.com>

I'm not sure everyone that works on yapf actually reads this ML, so
it'd be a good idea to use github's bug tracker when reporting
problems in the future. I filed a ticket for you:
https://github.com/google/yapf/issues/14

-- Devin

On Wed, Mar 25, 2015 at 9:38 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 3/25/2015 8:42 AM, Eli Bendersky wrote:
>>
>>
>>
>> On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky
>> <eliben at gmail.com
>> <mailto:eliben at gmail.com>> wrote:
>>
>>     Interesting coincidence :-)
>>
>>     We (at Google) are just on the verge of releasing an open-source
>>     tool for auto-formatting of Python code. It should be posted to
>>     Github within the next few days or weeks, under a permissive license
>>     (Apache 2.0 or something similar).
>>
>>     The tool works like gofmt and clang-format (the C++/JS version
>>     provided as part of the Clang toolchain) - it only acts on
>>     whitespace and performs re-formatting of code (PEP-8 oriented for
>>     Python but it's configurable). It does not rename
>>     variables/classes/functions to conform to any particular style. The
>>     algorithm is based on clang-format, and lib2to3 is used for the
>>     parsing side - so it's very solid. We use it on Python 2 code
>>     internally, but Python 3 support is also planned and should be easy.
>>
>>     We intend to keep actively developing the tool in the open and will
>>     be happy to accept test cases, fixes and patches in general.
>>
>>     So stay tuned!
>>
>>
>> https://github.com/google/yapf
>
>
> yet-another-python-formatter
>
> The so-called pep8 style in style.py has
>
> # The number of spaces required before a trailing comment.
> SPACES_BEFORE_COMMENT=1,
>
> PEP8 actually says "Inline comments should be separated by at least two
> spaces from the statement.", which is what you have in the google style.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From eliben at gmail.com  Thu Mar 26 02:00:09 2015
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 25 Mar 2015 18:00:09 -0700
Subject: [Python-ideas] gofmt for Python: standardized styling as a
 language feature
In-Reply-To: <meuoa3$evi$1@ger.gmane.org>
References: <CAOhmDzfimDhrhXpmt-M0F-9fYZdjgMNE_T29Yo11iFjn0PHpsA@mail.gmail.com>
 <meafff$tjv$1@ger.gmane.org>
 <CAOhmDzfxp+2ocstZC=Wnch=F-e0JOiwkaeKwssh+W+bBNV5_Yg@mail.gmail.com>
 <5508DBB0.2010209@stoneleaf.us>
 <CAOhmDzfjbW0WYzrLYGbj2oxU8W6QA6GAOH63jC33wdG7E+7Qqw@mail.gmail.com>
 <CAF-Rda_Fn0pdLBGswUQraQK1mrms5n3Aw74nEkKpEoxOmFKmYQ@mail.gmail.com>
 <CAF-Rda-Ox-GM-JTvAt8EyjpDt2LpQRXBM4VfWoCu+zCapFeu4Q@mail.gmail.com>
 <meuoa3$evi$1@ger.gmane.org>
Message-ID: <CAF-Rda_e_+iVF1c8-cnUhy-5h6zAD_3O0gAa=yWN_C64s3Di8A@mail.gmail.com>

On Wed, Mar 25, 2015 at 9:38 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/25/2015 8:42 AM, Eli Bendersky wrote:
>
>>
>>
>> On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky
>> <eliben at gmail.com
>> <mailto:eliben at gmail.com>> wrote:
>>
>>     Interesting coincidence :-)
>>
>>     We (at Google) are just on the verge of releasing an open-source
>>     tool for auto-formatting of Python code. It should be posted to
>>     Github within the next few days or weeks, under a permissive license
>>     (Apache 2.0 or something similar).
>>
>>     The tool works like gofmt and clang-format (the C++/JS version
>>     provided as part of the Clang toolchain) - it only acts on
>>     whitespace and performs re-formatting of code (PEP-8 oriented for
>>     Python but it's configurable). It does not rename
>>     variables/classes/functions to conform to any particular style. The
>>     algorithm is based on clang-format, and lib2to3 is used for the
>>     parsing side - so it's very solid. We use it on Python 2 code
>>     internally, but Python 3 support is also planned and should be easy.
>>
>>     We intend to keep actively developing the tool in the open and will
>>     be happy to accept test cases, fixes and patches in general.
>>
>>     So stay tuned!
>>
>>
>> https://github.com/google/yapf
>>
>
> yet-another-python-formatter
>
> The so-called pep8 style in style.py has
>
> # The number of spaces required before a trailing comment.
> SPACES_BEFORE_COMMENT=1,
>
> PEP8 actually says "Inline comments should be separated by at least two
> spaces from the statement.", which is what you have in the google style.
>
>
Thanks Terry. Devin opened an issue on Github and we'll fix this.

Eli






> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150325/30b8a7f1/attachment.html>

From probiner at gmail.com  Thu Mar 26 02:45:29 2015
From: probiner at gmail.com (pedro santos)
Date: Thu, 26 Mar 2015 01:45:29 +0000
Subject: [Python-ideas] list.pop() accepting a list of integers as argument
Message-ID: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>

Is this best way to remove multiple items from a list?
Because if so I think passing a list of integers through pop would be
cleaner.

Code here: http://repl.it/f6E

Cheers




-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150326/496ba187/attachment-0001.html>

From probiner at gmail.com  Thu Mar 26 03:39:38 2015
From: probiner at gmail.com (pedro santos)
Date: Thu, 26 Mar 2015 02:39:38 +0000
Subject: [Python-ideas] list.pop() accepting a list of integers as
	argument
In-Reply-To: <CAFx9zocFPxGJXZ+rbwXyzN1unCj88=VLgYGooYOX3-1WpU+PSA@mail.gmail.com>
References: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>
 <CAFx9zocFPxGJXZ+rbwXyzN1unCj88=VLgYGooYOX3-1WpU+PSA@mail.gmail.com>
Message-ID: <CAMrmD0oVYNHRq2hf4e+90h6hpNqHR48Y_sLoobPgGv7VztRanw@mail.gmail.com>

Hi David
Thanks for the idea. Just some considerations:

- Del is a list with indices and not the items themselves, so if you
replace the code with that line it won't output the same result.
- *del *deletes all items with an equal value so if I have L=[2,2,2,1,2]
and have Del=[2], I end up with L=[1], which useful, but not my goal here
and the reason I went to pop() instead, so I can address indices. Same
remark for *in*.
- I wouldn't mind if* in* and *del* approach would work if you instead of
checking the value would be checking if a list had such indices and then
deleting all at the same time if so.

Again the same code with comments. http://repl.it/f6E/3

Thanks



On Thu, Mar 26, 2015 at 2:04 AM, David Blaschke <dwblas at gmail.com> wrote:

> ist comprehension is straight forward IMHO, and possibly faster than
> pop depending on whether the items are removed one at a time or just
> marked for deletion later
> new_list=[item for item in L if item not in Del]
>
> On 3/25/15, pedro santos <probiner at gmail.com> wrote:
> > Is this best way to remove multiple items from a list?
> > Because if so I think passing a list of integers through pop would be
> > cleaner.
> >
> > Code here: http://repl.it/f6E
> >
> > Cheers
> >
> >
> >
> >
> > --
> >
> >
> >
> >
> > *------------------------------[image:
> > http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
> > Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
> > http://probiner.x10.mx/ <http://probiner.x10.mx/>*
> >
>
>
> --
> Only when you actually get to the state where there is neither
> delusion nor enlightenment are you finally comfortable...Foyan
>



-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150326/007ce133/attachment.html>

From ron3200 at gmail.com  Thu Mar 26 04:36:04 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Wed, 25 Mar 2015 23:36:04 -0400
Subject: [Python-ideas] list.pop() accepting a list of integers as
	argument
In-Reply-To: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>
References: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>
Message-ID: <mevur5$igf$1@ger.gmane.org>



On 03/25/2015 09:45 PM, pedro santos wrote:
> Is this best way to remove multiple items from a list?
> Because if so I think passing a list of integers through pop would be cleaner.

This is probably the fastest way to remove N items from the end.

 >>> L = list("123456789")
 >>> L
['1', '2', '3', '4', '5', '6', '7', '8', '9']
 >>> L[-3:] = []
 >>> L
['1', '2', '3', '4', '5', '6']

Cheers,
    Ron


From probiner at gmail.com  Thu Mar 26 07:58:57 2015
From: probiner at gmail.com (pedro santos)
Date: Thu, 26 Mar 2015 06:58:57 +0000
Subject: [Python-ideas] list.pop() accepting a list of integers as
	argument
In-Reply-To: <mevur5$igf$1@ger.gmane.org>
References: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>
 <mevur5$igf$1@ger.gmane.org>
Message-ID: <CAMrmD0oQ06ectNcAe3Puv9ckOBPiqiFXyWP93DYsGkE0EuXcKg@mail.gmail.com>

Hi Ron
Wasn't looking for the n items from the end but good to know.

Cheers


On Thu, Mar 26, 2015 at 3:36 AM, Ron Adam <ron3200 at gmail.com> wrote:

>
>
> On 03/25/2015 09:45 PM, pedro santos wrote:
>
>> Is this best way to remove multiple items from a list?
>> Because if so I think passing a list of integers through pop would be
>> cleaner.
>>
>
> This is probably the fastest way to remove N items from the end.
>
> >>> L = list("123456789")
> >>> L
> ['1', '2', '3', '4', '5', '6', '7', '8', '9']
> >>> L[-3:] = []
> >>> L
> ['1', '2', '3', '4', '5', '6']
>
> Cheers,
>    Ron
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 




*------------------------------[image:
http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro
Alpiar?a dos Santos Animator  3DModeler  Illustrator >>
http://probiner.x10.mx/ <http://probiner.x10.mx/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150326/0756bf47/attachment.html>

From storchaka at gmail.com  Thu Mar 26 08:41:53 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 26 Mar 2015 09:41:53 +0200
Subject: [Python-ideas] list.pop() accepting a list of integers as
	argument
In-Reply-To: <mevur5$igf$1@ger.gmane.org>
References: <CAMrmD0pVAk=3sT2_pKuQZYVYJGPeLd2Om4m6TQ8JXbSNv014Sw@mail.gmail.com>
 <mevur5$igf$1@ger.gmane.org>
Message-ID: <mf0d81$1g6$1@ger.gmane.org>

On 26.03.15 05:36, Ron Adam wrote:
> This is probably the fastest way to remove N items from the end.
>
>  >>> L = list("123456789")
>  >>> L
> ['1', '2', '3', '4', '5', '6', '7', '8', '9']
>  >>> L[-3:] = []
>  >>> L
> ['1', '2', '3', '4', '5', '6']

del L[-3:]



From szport at gmail.com  Thu Mar 26 06:17:48 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Wed, 25 Mar 2015 22:17:48 -0700 (PDT)
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
Message-ID: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>



???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho 
???????:
>
> Sometimes we need a simple class to hold some mutable attributes, 
> provide a nice repr, support == for testing, and support iterable 
> unpacking, so you can write: 
>
> >>> p = Point(3, 4) 
> >>> x, y = p 
>
> That's very much like the classes built by namedtuple, but mutable. 
>
> I propose we add to the collections module another class factory. I am 
> calling it plainclass, but perhaps we can think of a better name. Here 
> is how it would be used: 
>
> >>> import collections 
> >>> Point = collections.plainclass('Point', 'x y') 
>
> The signature of the plainclass function would be exactly the same as 
> namedtuple, supporting the same alternative ways of naming the 
> attributes. 
>
>
There is an attempt to make such alternative: recordarray 
<https://pypi.python.org/pypi/recordarray>.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150325/98edc8fe/attachment-0001.html>

From guettliml at thomas-guettler.de  Thu Mar 26 09:59:06 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Thu, 26 Mar 2015 09:59:06 +0100
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <mev43m$ia$1@ger.gmane.org>
References: <55129532.8030502@thomas-guettler.de>
 <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com> <mev43m$ia$1@ger.gmane.org>
Message-ID: <5513CA5A.1020001@thomas-guettler.de>

Am 25.03.2015 um 20:59 schrieb Ron Adam:
>
>
> On 03/25/2015 12:51 PM, Andrew Barnert wrote:
>> On Mar 25, 2015, at 04:00, Thomas G?ttler<guettliml at thomas-guettler.de>
>> wrote:
>
>>>> I am missing OrderedDict.insert()
>
>> With what interface? Does it take an index, like list.insert, even
>> though other OrderedDict methods (including __getitem__ and friends)
>> don't? Or does it take another key? If so, does it insert before or
>> after that key? And what if that key doesn't exist? Meanwhile, if the
>> new key was already in the dict, should it move to the new position,
>> update values in place, or raise an exception? Do you expect O(1)
>> performance?

My use case is that django drops support for SortedDict, and you should use
collection.OrderedDict in the future:

   https://code.djangoproject.com/wiki/SortedDict

We have one line where we use my_ordered_dict.insert() and this line is already updated.
The issue is not very important for me.

To your question:

The interface of django was insert(index, key, value) but you are right. An interface
which is relative to an existing key would be better.

insert_before(other_key, new_key, new_value)

insert_behind(other_key, new_key, new_value)

Both methods should raise KeyError if other_key is not in the dict and
should raise KeyError if new_key is already in the dict.

Performance: I don't care. I store big data in a database. I don't carry it
around in the memory if the python interpreter.

Regards,
   Thomas G?ttler

From abarnert at yahoo.com  Thu Mar 26 11:08:56 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 26 Mar 2015 03:08:56 -0700
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <5513CA5A.1020001@thomas-guettler.de>
References: <55129532.8030502@thomas-guettler.de>
 <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com> <mev43m$ia$1@ger.gmane.org>
 <5513CA5A.1020001@thomas-guettler.de>
Message-ID: <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com>

On Mar 26, 2015, at 01:59, Thomas G?ttler <guettliml at thomas-guettler.de> wrote:
> 
>> Am 25.03.2015 um 20:59 schrieb Ron Adam:
>> 
>> 
>>> On 03/25/2015 12:51 PM, Andrew Barnert wrote:
>>> On Mar 25, 2015, at 04:00, Thomas G?ttler<guettliml at thomas-guettler.de>
>>> wrote:
>> 
>>>>> I am missing OrderedDict.insert()
>> 
>>> With what interface? Does it take an index, like list.insert, even
>>> though other OrderedDict methods (including __getitem__ and friends)
>>> don't? Or does it take another key? If so, does it insert before or
>>> after that key? And what if that key doesn't exist? Meanwhile, if the
>>> new key was already in the dict, should it move to the new position,
>>> update values in place, or raise an exception? Do you expect O(1)
>>> performance?
> 
> My use case is that django drops support for SortedDict, and you should use
> collection.OrderedDict in the future:
> 
>  https://code.djangoproject.com/wiki/SortedDict

From a quick scan (see https://github.com/django/django/commit/7379d9aceab01e007966b5fe1f47ba7590deb887), it looks like Django deprecated insert in 1.5--two versions before deprecating SortedDict itself, while the type was still being heavily used all over Django itself. That implies that they didn't think it belonged there either (although I didn't find the issue or discussion where they made that decision, so don't take that as gospel--for all I know maybe they just discovered a bad bug in their implementation and decided deprecating it would break less code than changing it...).

Meanwhile, that's not really a use case. Without any idea of why you used SortedDict.insert, I still have no clue why someone would use OrderedDict.insert. I don't think it's worth adding stuff to Python's stdlib just to make it easier for people porting pre-Django-1.5 code to Django 1.9 to work around an intentional change in Django.

> We have one line where we use my_ordered_dict.insert() and this line is already updated.
> The issue is not very important for me.
> 
> To your question:
> 
> The interface of django was insert(index, key, value) but you are right. An interface
> which is relative to an existing key would be better.

I'm not sure about that. I suspect inserting at the front would be one of the more common uses for this (again, just guessing, because I don't know of _any_ real-life uses... but that was what the SO question you linked to wanted...), which would be pretty clumsy with before-key instead of before-index.

But that was part of the point. I don't think the meaning or interface of OrderedDict.insert is obvious, and you didn't say what you wanted it to do. In fact what you really wanted and what Django used to do are different, so what are the odds that whichever one we chose, some users would have guessed the opposite?

> insert_before(other_key, new_key, new_value)
> 
> insert_behind(other_key, new_key, new_value)
> 
> Both methods should raise KeyError if other_key is not in the dict and
> should raise KeyError if new_key is already in the dict.
> 
> Performance: I don't care. I store big data in a database. I don't carry it
> around in the memory if the python interpreter.
> 
> Regards,
>  Thomas G?ttler
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From ericsnowcurrently at gmail.com  Thu Mar 26 23:06:06 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 26 Mar 2015 16:06:06 -0600
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <55129532.8030502@thomas-guettler.de>
References: <55129532.8030502@thomas-guettler.de>
Message-ID: <CALFfu7DVeYnZ58Q1C4QXs=i7htmtrPuEPuRFuGNP5HbhUv64qw@mail.gmail.com>

On Wed, Mar 25, 2015 at 5:00 AM, Thomas G?ttler
<guettliml at thomas-guettler.de> wrote:
> I am missing OrderedDict.insert()
>
> Other people, too:
>
>
> http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python
>
> Why was it not implemented up to now?
>
> Why not implement it for the future?

Ordered dict is a dict that preserves insertion order (and exposes via
iteration-related methods like __iter__ and the views).  That's it.
It is not a dict that also implements Sequence.  Keep in mind that
OrderedDict is a fairly simply adaptation of dict.  Unfortunately
there is a lingering temptation to think of it as a sequence.

-eric

From liik.joonas at gmail.com  Fri Mar 27 05:08:46 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Fri, 27 Mar 2015 06:08:46 +0200
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
Message-ID: <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>

namedlist perhaps? :)

if you want:"namedtuple, but mutable. " then namedlist seems like the
obvious alternative..

On 26 March 2015 at 07:17, Zaur Shibzukhov <szport at gmail.com> wrote:

>
>
> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho
> ???????:
>>
>> Sometimes we need a simple class to hold some mutable attributes,
>> provide a nice repr, support == for testing, and support iterable
>> unpacking, so you can write:
>>
>> >>> p = Point(3, 4)
>> >>> x, y = p
>>
>> That's very much like the classes built by namedtuple, but mutable.
>>
>> I propose we add to the collections module another class factory. I am
>> calling it plainclass, but perhaps we can think of a better name. Here
>> is how it would be used:
>>
>> >>> import collections
>> >>> Point = collections.plainclass('Point', 'x y')
>>
>> The signature of the plainclass function would be exactly the same as
>> namedtuple, supporting the same alternative ways of naming the
>> attributes.
>>
>>
> There is an attempt to make such alternative: recordarray
> <https://pypi.python.org/pypi/recordarray>.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/d3321f9c/attachment.html>

From szport at gmail.com  Fri Mar 27 05:26:52 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Fri, 27 Mar 2015 07:26:52 +0300
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
Message-ID: <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>

---
*Zaur Shibzukhov*


2015-03-27 7:08 GMT+03:00 Joonas Liik <liik.joonas at gmail.com>:

> namedlist perhaps? :)
>
> if you want:"namedtuple, but mutable. " then namedlist seems like the
> obvious alternative..
>

The name 'namedlist' is already used in
https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in
https://pypi.python.org/pypi/mutabletuple.

The name recordarray is means that it 1) array of objects and 2) record
with access to fields by attributes.


>
> On 26 March 2015 at 07:17, Zaur Shibzukhov <szport at gmail.com> wrote:
>
>>
>>
>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho
>> ???????:
>>>
>>> Sometimes we need a simple class to hold some mutable attributes,
>>> provide a nice repr, support == for testing, and support iterable
>>> unpacking, so you can write:
>>>
>>> >>> p = Point(3, 4)
>>> >>> x, y = p
>>>
>>> That's very much like the classes built by namedtuple, but mutable.
>>>
>>> I propose we add to the collections module another class factory. I am
>>> calling it plainclass, but perhaps we can think of a better name. Here
>>> is how it would be used:
>>>
>>> >>> import collections
>>> >>> Point = collections.plainclass('Point', 'x y')
>>>
>>> The signature of the plainclass function would be exactly the same as
>>> namedtuple, supporting the same alternative ways of naming the
>>> attributes.
>>>
>>>
>> There is an attempt to make such alternative: recordarray
>> <https://pypi.python.org/pypi/recordarray>.
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/422b5e08/attachment.html>

From abarnert at yahoo.com  Fri Mar 27 05:32:45 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 26 Mar 2015 21:32:45 -0700
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
Message-ID: <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com>

On Mar 26, 2015, at 21:08, Joonas Liik <liik.joonas at gmail.com> wrote:
> 
> namedlist perhaps? :)
> 
> if you want:"namedtuple, but mutable. " then namedlist seems like the obvious alternative..

But, as discussed earlier in the thread, a list isn't right, or at least isn't obviously right, because lists can change size, and what would it mean for a namedlist to, say, delete element 3?

And that's really the problem: Python's entire infrastructure is designed around things which are reshapable like lists, or immutable like tuples, and this doesn't fit either one.

>> On 26 March 2015 at 07:17, Zaur Shibzukhov <szport at gmail.com> wrote:
>> 
>> 
>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????:
>>> 
>>> Sometimes we need a simple class to hold some mutable attributes, 
>>> provide a nice repr, support == for testing, and support iterable 
>>> unpacking, so you can write: 
>>> 
>>> >>> p = Point(3, 4) 
>>> >>> x, y = p 
>>> 
>>> That's very much like the classes built by namedtuple, but mutable. 
>>> 
>>> I propose we add to the collections module another class factory. I am 
>>> calling it plainclass, but perhaps we can think of a better name. Here 
>>> is how it would be used: 
>>> 
>>> >>> import collections 
>>> >>> Point = collections.plainclass('Point', 'x y') 
>>> 
>>> The signature of the plainclass function would be exactly the same as 
>>> namedtuple, supporting the same alternative ways of naming the 
>>> attributes.
>> 
>> There is an attempt to make such alternative: recordarray

The name "array" seems really unfortunate. It doesn't give you any clue that this thing is halfway between a tuple and a list. Also, we've already got a bytearray, array.array, and the NumPy array types, all of which hold homogeneous simple-value types and can be accessed as buffers. And bytearray and array.array are resizable. And np.ndarray--like a C array, in effect--returns views when sliced rather than copies. Just about everything this name implies is misleading.

And as for recordarray, that doesn't exactly scream "like a namedtuple, but with array instead of tuple".

Also, in the weeks this thing has been discussed, no one has yet come up with a use case where. Look at the motivating example, Point--do you think location[2] meaning the same thing as location.z is a good thing? That's neither easy nor common in C and its OO descendants, or SmallTalk and its, or most other languages, and I can't remember ever being bothered by that.

The best anyone has come up with is that it might be more space-efficient than a SimpleNamespace or a standard __slots__ class, for all those times when you really need a billion points and can afford to needlessly waste 140 bytes per point instead of 12, but can't afford to waste 188.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150326/98b068c9/attachment-0001.html>

From abarnert at yahoo.com  Fri Mar 27 05:40:46 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 26 Mar 2015 21:40:46 -0700
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
Message-ID: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>

On Mar 26, 2015, at 21:26, Zaur Shibzukhov <szport at gmail.com> wrote:
> 2015-03-27 7:08 GMT+03:00 Joonas Liik <liik.joonas at gmail.com>:
>> namedlist perhaps? :)
>> 
>> if you want:"namedtuple, but mutable. " then namedlist seems like the obvious alternative..
> 
> The name 'namedlist' is already used in https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in https://pypi.python.org/pypi/mutabletuple.

If you're trying to provide the same concept, why use a completely unrelated name? That's like saying 'I want an ordered set, but there's already an "orderedset" on PyPI so I went with "sortedsequence"'.

> The name recordarray is means that it 1) array of objects and 2) record with access to fields by attributes.

But how is being "an array of objects" any different from what a tuple, list, array.array, bytearray, bytes, str, etc. already are? What's specifically array-like about this type as opposed to all of those? And what's specifically record-like about your type compared to namedtuple, Struct, or SimpleNamespace?

>  
>> 
>>> On 26 March 2015 at 07:17, Zaur Shibzukhov <szport at gmail.com> wrote:
>>> 
>>> 
>>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????:
>>>> 
>>>> Sometimes we need a simple class to hold some mutable attributes, 
>>>> provide a nice repr, support == for testing, and support iterable 
>>>> unpacking, so you can write: 
>>>> 
>>>> >>> p = Point(3, 4) 
>>>> >>> x, y = p 
>>>> 
>>>> That's very much like the classes built by namedtuple, but mutable. 
>>>> 
>>>> I propose we add to the collections module another class factory. I am 
>>>> calling it plainclass, but perhaps we can think of a better name. Here 
>>>> is how it would be used: 
>>>> 
>>>> >>> import collections 
>>>> >>> Point = collections.plainclass('Point', 'x y') 
>>>> 
>>>> The signature of the plainclass function would be exactly the same as 
>>>> namedtuple, supporting the same alternative ways of naming the 
>>>> attributes.
>>> 
>>> There is an attempt to make such alternative: recordarray.
>>>  
>>> 
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150326/70cf7dc7/attachment.html>

From guettliml at thomas-guettler.de  Fri Mar 27 09:52:02 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Fri, 27 Mar 2015 09:52:02 +0100
Subject: [Python-ideas] OrderedDict.insert()
In-Reply-To: <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com>
References: <55129532.8030502@thomas-guettler.de>
 <F95397ED-68C8-461C-9B93-D1089E19E87C@yahoo.com> <mev43m$ia$1@ger.gmane.org>
 <5513CA5A.1020001@thomas-guettler.de>
 <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com>
Message-ID: <55151A32.3050003@thomas-guettler.de>

I withdraw the idea.

Thank you all for your answers.


Am 26.03.2015 um 11:08 schrieb Andrew Barnert:
> On Mar 26, 2015, at 01:59, Thomas G?ttler <guettliml at thomas-guettler.de> wrote:
>>
>>> Am 25.03.2015 um 20:59 schrieb Ron Adam:
>>>
>>>
>>>> On 03/25/2015 12:51 PM, Andrew Barnert wrote:
>>>> On Mar 25, 2015, at 04:00, Thomas G?ttler<guettliml at thomas-guettler.de>
>>>> wrote:
>>>
>>>>>> I am missing OrderedDict.insert()
>>>
>>>> With what interface? Does it take an index, like list.insert, even
>>>> though other OrderedDict methods (including __getitem__ and friends)
>>>> don't? Or does it take another key? If so, does it insert before or
>>>> after that key? And what if that key doesn't exist? Meanwhile, if the
>>>> new key was already in the dict, should it move to the new position,
>>>> update values in place, or raise an exception? Do you expect O(1)
>>>> performance?
>>
>> My use case is that django drops support for SortedDict, and you should use
>> collection.OrderedDict in the future:
>>
>>   https://code.djangoproject.com/wiki/SortedDict
>
>  From a quick scan (see https://github.com/django/django/commit/7379d9aceab01e007966b5fe1f47ba7590deb887), it looks like Django deprecated insert in 1.5--two versions before deprecating SortedDict itself, while the type was still being heavily used all over Django itself. That implies that they didn't think it belonged there either (although I didn't find the issue or discussion where they made that decision, so don't take that as gospel--for all I know maybe they just discovered a bad bug in their implementation and decided deprecating it would break less code than changing it...).
>
> Meanwhile, that's not really a use case. Without any idea of why you used SortedDict.insert, I still have no clue why someone would use OrderedDict.insert. I don't think it's worth adding stuff to Python's stdlib just to make it easier for people porting pre-Django-1.5 code to Django 1.9 to work around an intentional change in Django.
>
>> We have one line where we use my_ordered_dict.insert() and this line is already updated.
>> The issue is not very important for me.
>>
>> To your question:
>>
>> The interface of django was insert(index, key, value) but you are right. An interface
>> which is relative to an existing key would be better.
>
> I'm not sure about that. I suspect inserting at the front would be one of the more common uses for this (again, just guessing, because I don't know of _any_ real-life uses... but that was what the SO question you linked to wanted...), which would be pretty clumsy with before-key instead of before-index.
>
> But that was part of the point. I don't think the meaning or interface of OrderedDict.insert is obvious, and you didn't say what you wanted it to do. In fact what you really wanted and what Django used to do are different, so what are the odds that whichever one we chose, some users would have guessed the opposite?
>
>> insert_before(other_key, new_key, new_value)
>>
>> insert_behind(other_key, new_key, new_value)
>>
>> Both methods should raise KeyError if other_key is not in the dict and
>> should raise KeyError if new_key is already in the dict.
>>
>> Performance: I don't care. I store big data in a database. I don't carry it
>> around in the memory if the python interpreter.
>>
>> Regards,
>>   Thomas G?ttler
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/

From toddrjen at gmail.com  Fri Mar 27 10:03:26 2015
From: toddrjen at gmail.com (Todd)
Date: Fri, 27 Mar 2015 10:03:26 +0100
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com>
Message-ID: <CAFpSVpLQNYdvBB8sqLUjdPN32_pA5gwa41gcbcj0OcoY7x9-Pg@mail.gmail.com>

On Mar 27, 2015 5:33 AM, "Andrew Barnert" <abarnert at yahoo.com.dmarc.invalid>
wrote:
>
> On Mar 26, 2015, at 21:08, Joonas Liik <liik.joonas at gmail.com> wrote:
>> On 26 March 2015 at 07:17, Zaur Shibzukhov <szport at gmail.com> wrote:
>>>
>>>
>>>
>>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho
???????:
>>>>
>>>> Sometimes we need a simple class to hold some mutable attributes,
>>>> provide a nice repr, support == for testing, and support iterable
>>>> unpacking, so you can write:
>>>>
>>>> >>> p = Point(3, 4)
>>>> >>> x, y = p
>>>>
>>>> That's very much like the classes built by namedtuple, but mutable.
>>>>
>>>> I propose we add to the collections module another class factory. I am
>>>> calling it plainclass, but perhaps we can think of a better name. Here
>>>> is how it would be used:
>>>>
>>>> >>> import collections
>>>> >>> Point = collections.plainclass('Point', 'x y')
>>>>
>>>> The signature of the plainclass function would be exactly the same as
>>>> namedtuple, supporting the same alternative ways of naming the
>>>> attributes.
>>>>
>>>
>>> There is an attempt to make such alternative: recordarray
>
>
> The name "array" seems really unfortunate. It doesn't give you any clue
that this thing is halfway between a tuple and a list. Also, we've already
got a bytearray, array.array, and the NumPy array types, all of which hold
homogeneous simple-value types and can be accessed as buffers. And
bytearray and array.array are resizable. And np.ndarray--like a C array, in
effect--returns views when sliced rather than copies. Just about everything
this name implies is misleading.
>
> And as for recordarray, that doesn't exactly scream "like a namedtuple,
but with array instead of tuple".

Numpy already has the concept of a "record array", which they call
"recarray", which is a numpy array that is partially accessible in a
similar manner to named tuples. However, otherwise they are the same as
numpy arrays, which means they have a fixed size (usually) but mutable
contents.

Whether you consider this a point in favor or a point against, however,
probably depends on your point of view.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/e8092c3e/attachment-0001.html>

From szport at gmail.com  Fri Mar 27 14:17:22 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Fri, 27 Mar 2015 16:17:22 +0300
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
Message-ID: <CAPOE21SsMFJ1BXct74gOKFY3044QLoiihKCS4=VPYS=DYZQkcw@mail.gmail.com>

---
*Zaur Shibzukhov*


2015-03-27 7:40 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:

> On Mar 26, 2015, at 21:26, Zaur Shibzukhov <szport at gmail.com> wrote:
>
> 2015-03-27 7:08 GMT+03:00 Joonas Liik <liik.joonas at gmail.com>:
>
>> namedlist perhaps? :)
>>
>> if you want:"namedtuple, but mutable. " then namedlist seems like the
>> obvious alternative..
>>
>
> The name 'namedlist' is already used in
> https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in
> https://pypi.python.org/pypi/mutabletuple.
>
>
> If you're trying to provide the same concept, why use a completely
> unrelated name? That's like saying 'I want an ordered set, but there's
> already an "orderedset" on PyPI so I went with "sortedsequence"'.
>
> The name recordarray is means that it 1) array of objects and 2) record
> with access to fields by attributes.
>
>
> But how is being "an array of objects" any different from what a tuple,
> list, array.array, bytearray, bytes, str, etc. already are? What's
> specifically array-like about this type as opposed to all of those? And
> what's specifically record-like about your type compared to namedtuple,
> Struct, or SimpleNamespace?
>
>
I am inclined to the fact that it's better to rename `objectarray` to
'mutabletuple' in order to be explicit about what is that.
'recordarray' is a factory function that make exactly the same as
'namedtuple' factory does (except that it create subclass of 'mutabletuple'
and make '_replace' to update the 'self', not make a copy). So may be it's
better to call it as 'record' or 'recordtype', or even 'recordfactory'?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/5731e21c/attachment.html>

From jsbueno at python.org.br  Fri Mar 27 14:22:54 2015
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 27 Mar 2015 10:22:54 -0300
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
Message-ID: <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>

On 27 March 2015 at 01:40, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> But how is being "an array of objects" any different from what a tuple,
> list, array.array, bytearray, bytes, str, etc. already are? What's
> specifically array-like about this type as opposed to all of those? And
> what's specifically record-like about your type compared to namedtuple,
> Struct, or SimpleNamespace?

Acutally, on my understanding, the request on this thread is for
something that is quite concrete,
existing in other languages, and that can be done in Python in a few
lines, but is not in
the stdlib:

The Python equivalent of a C Struct.

Just that.
An easy to create class, with named fields,  with possible
type-enforcement for those fields.

Or maybe it _does_ exist in Python, and it is a matter of having a
nice example in the docs:
for example a "blank" class with "__slots__"  would do it.
Or a blank class with slots that could serialize and deserialize
itself to a sequence
in a seamless way.

class Base:
   __slots__ = ()
   def __init__(self, seq=None):
       if not seq: return
       for attr, val in zip(self.slots, seq):
            setattr(self, attr, val)
   def __iter__(self):
       for attr in self.__slots__:
           yield getattr(self, attr)


def NamedList(name, fields):
   ... # split string with space separated fields, and other niceities here
   return type(name, (Base,), dict(__slots__=fields))

And 10-15 more lines if one wants type-checking, default values,
__repr__ into that. I think getting a proper recipe for this, and
publicizing it on the documentation ecosystem is enough  - maybe a
Pypi module adding some more goodies - and if that would get any
traction - the usual consideration for inclusion could apply.

From rosuav at gmail.com  Fri Mar 27 15:04:07 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Mar 2015 01:04:07 +1100
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xKX1imsRaLxU6Za3zme=mWq_KO0pCGspGaQ6jKbkX_EOA@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <CAPTjJmpyqiNgWgMCnVHfWTEyFQ2C8Bs0gTfBOs3R1uhzMv6gaQ@mail.gmail.com>
 <CAPkN8xKX1imsRaLxU6Za3zme=mWq_KO0pCGspGaQ6jKbkX_EOA@mail.gmail.com>
Message-ID: <CAPTjJmrD5t1nGR59yJ1UBqoOG314jei+Q8OCCXus1oSeAMfiwg@mail.gmail.com>

On Sat, Mar 28, 2015 at 12:52 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> Why the wave module is there? Or ioctl module? Are they so widely used?
>
> This process is flawed. If anybody would sponsor the research on popularity
> of stdlib, I believe that math will be excluded from top 50 as well. Will
> stdlib
> without math will still be ok?
>
> The measure to estimate the usefullness of standard library is not in
> download score, but in coverage across different areas. There thousands of
> people downloading stuff from youtube, so should we include youtube-dl
> instead of math?

The criteria for the *removal* from the stdlib would be insanely
tight, compared to the criteria for *addition*. So some modules might
now simply be grandfathered in (I suspect wave is one of them), due to
the shift in ease of grabbing stuff off PyPI. But the math module is
pretty important and fundamental, so it's not going anywhere.

As to youtube-dl, though: Definitely not. No matter how popular it is,
it still fails the criterion of release schedule synchronization - it
needs to be agile enough to update as often as it needs, according to
host site changes, and would be shackled rather badly by inclusion in
the stdlib. (You probably don't want to get it from your OS repo
either.)

But none of this changes the recommendations I made earlier. You need
to *demonstrate*, not simply state, a need; and one of the best ways
to demonstrate support, API stability, usage, etc, is to stick a
package up on PyPI. Not everything needs to be in the stdlib, but if
you're going to argue that something should be, what better way than
by pointing out what it's already doing?

ChrisA

From alexander.belopolsky at gmail.com  Fri Mar 27 16:21:19 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 27 Mar 2015 11:21:19 -0400
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
 <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>
Message-ID: <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>

On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik <techtonik at gmail.com>
wrote:

> Where have you been when PEP 3108 was discussed?  I have not seen any
>> other list of Python modules that needed a redesign, so I cannot tell
>> what's on your top ten list.
>>
>
> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html
>

Interesting.  I did not see this back in the day.  The top entry (urllib
and friends) makes sense and there were heavily redesigned in Python 3.  I
am surprised that distutils redesign got less support than logging and
datetime.  I suspect that this may have to do with logging and datetime
APIs not being PEP8 compliant.  Popular votes tend to exaggerate the
importance of trivial things such as the spelling of class and method names
and brush off more subtle, but important design flaws.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/22215ea2/attachment.html>

From alexander.belopolsky at gmail.com  Fri Mar 27 17:55:36 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 27 Mar 2015 12:55:36 -0400
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLTiaz7V_-TFXJVnJOp9SUbHAbq2kxk50+8MkbK4PD2BQ@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
 <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>
 <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>
 <CAPkN8xLTiaz7V_-TFXJVnJOp9SUbHAbq2kxk50+8MkbK4PD2BQ@mail.gmail.com>
Message-ID: <CAP7h-xYp9tX+8xS5W9w3s5xcWjFcmmaj_q4_771ttCgrxh1zQg@mail.gmail.com>

On Fri, Mar 27, 2015 at 12:13 PM, anatoly techtonik <techtonik at gmail.com>
wrote:
>
> Here is something that can be used as an example that it is not about
PEP8 https://code.google.com/p/rainforce/wiki/WartsOfPython#measure_time
And it takes a lot of energy to collect something like that for the
reference.


Well, in that document, I see the total of four "warts" related to the
datetime module.  If four warts bring a module to the top of the "worst
designed stdlib modules" list, it can only mean that stdlib is almost
perfect!

For each "wart" the author has a "What can be done?" section, but no
suggested solution involves a major redesign of the datetime module.

The author complains about non-obviousness of strftime and indeed, for
users without C background, neither name nor semantics is familiar.  But
the proposed solution

>>> time.format('{{hours}}:{{minutes}}:{{seconds}}', 1090)
'00:18:10'


Does not look like a big win over the already available solution:

>>> '{t.hour}:{t.minute}:{t.second}'.format(t=datetime.now())
'12:37:38'

Overall, while I agree that there are a few warts in the datetime module, I
have not seen any serious enough to warrant a major redesign or even any
non backward compatible changes.  Maintaining backward compatibility in the
datetime module is indeed non-trivial, but this is the way I think it
should be developed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/daceda25/attachment-0001.html>

From szport at gmail.com  Fri Mar 27 18:57:28 2015
From: szport at gmail.com (Zaur Shibzukhov)
Date: Fri, 27 Mar 2015 20:57:28 +0300
Subject: [Python-ideas] A mutable alternative to namedtuple: recordarray ->
	recordclass:
Message-ID: <CAPOE21Tp6wAyZG=NGH=cJ5OuDQj7U89ozDLJYDj5FEXDdfLiYw@mail.gmail.com>

---
*Zaur Shibzukhov*


2015-03-27 16:17 GMT+03:00 Zaur Shibzukhov <szport at gmail.com>:

>
>
> ---
> *Zaur Shibzukhov*
>
>
> 2015-03-27 7:40 GMT+03:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> On Mar 26, 2015, at 21:26, Zaur Shibzukhov <szport at gmail.com> wrote:
>>
>> 2015-03-27 7:08 GMT+03:00 Joonas Liik <liik.joonas at gmail.com>:
>>
>>> namedlist perhaps? :)
>>>
>>> if you want:"namedtuple, but mutable. " then namedlist seems like the
>>> obvious alternative..
>>>
>>
>> The name 'namedlist' is already used in
>> https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in
>> https://pypi.python.org/pypi/mutabletuple.
>>
>>
>> If you're trying to provide the same concept, why use a completely
>> unrelated name? That's like saying 'I want an ordered set, but there's
>> already an "orderedset" on PyPI so I went with "sortedsequence"'.
>>
>> The name recordarray is means that it 1) array of objects and 2) record
>> with access to fields by attributes.
>>
>>
>> But how is being "an array of objects" any different from what a tuple,
>> list, array.array, bytearray, bytes, str, etc. already are? What's
>> specifically array-like about this type as opposed to all of those? And
>> what's specifically record-like about your type compared to namedtuple,
>> Struct, or SimpleNamespace?
>>
>>
> I am inclined to the fact that it's better to rename `objectarray` to
> 'mutabletuple' in order to be explicit about what is that.
> 'recordarray' is a factory function that make exactly the same as
> 'namedtuple' factory does (except that it create subclass of 'mutabletuple'
> and make '_replace' to update the 'self', not make a copy). So may be it's
> better to call it as 'record' or 'recordtype', or even 'recordfactory'?
>
>
After consideration of all arguments recordarray project was renamed tand
moved to recordclass <https://pypi.python.org/pypi/recordclass>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/96f1f8c4/attachment.html>

From jw14896.2014 at my.bristol.ac.uk  Fri Mar 27 22:36:34 2015
From: jw14896.2014 at my.bristol.ac.uk (Jamie Willis)
Date: Fri, 27 Mar 2015 21:36:34 +0000
Subject: [Python-ideas] Proposal for new augmented assignment operator for
	method calls
Message-ID: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>

(As posted on python-list)

I would like to propose a new piece of syntax for the python language; .=

In short, the operator is form of syntactic sugar, for instance consider
the following code:

hello = "hello world              "
> hello = hello.strip()


This could be written as:

hello = "hello world              "
> hello .= strip()


In this slightly contrived example, the programmer saved (a small amount
of) time when writing the code. With code with longer variable names, or
lots of similar statements all in a row, this helps to keep code more
concise, and could cut down on potential bugs due to typos (thanks to Skip
Montanaro for pointing this out).

The operator would be constricted to one method or field on the right-hand
side, which must belong to the object on the left hand side.

To provide another example, consider this basic LinkedList class:

class LinkedList:
>
>     def __init__(self, data, head=None):
>         self.data = data
>         self.next = None
>         self.head = head if head != None else self
>


    def append(self, data):

        new_node = LinkedList(data, self.head)
>         self.next = new_node
>         return new_node
>


linked_list = LinkedList(7)
> linked_list = linked_list.append(8)
> linked_list = linked_list.append(9)
>


loop_node = linked_list.head

while loop_node != None:
>     print(loop_node.data)
>     loop_node = loop_node.next


Taking a look at "*linked_list = linked_list.append(8)*", we could write
this as:

linked_list .= append(8)


under the new syntax and then "*loop_node = loop_node.next*" could simply
become:

loop_node .= next


Would this idea get much support if submitted as a PEP? What are the
potential concerns I could consider with the syntax (Skip has already
pointed out that the "dot" is pretty invisible in the operator, so perhaps
a different character could be used)?

Thanks,

Jamie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/5606b319/attachment.html>

From alexander.belopolsky at gmail.com  Fri Mar 27 23:01:17 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 27 Mar 2015 18:01:17 -0400
Subject: [Python-ideas] Proposal for new augmented assignment operator
 for method calls
In-Reply-To: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
References: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
Message-ID: <CAP7h-xaoFXvwBc_w_+UMytdoTAxm25+7aO4WLELnVhVkQ81SQw@mail.gmail.com>

On Fri, Mar 27, 2015 at 5:36 PM, Jamie Willis <jw14896.2014 at my.bristol.ac.uk>
wrote:
>
> I would like to propose a new piece of syntax for the python language; .=
>
> In short, the operator is form of syntactic sugar, for instance consider
the following code:
>
>> hello = "hello world              "
>> hello = hello.strip()
>
>
> This could be written as:
>
>> hello = "hello world              "
>> hello .= strip()


"Syntax must not look like grit on Tim's screen!"

-1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/cc37ae59/attachment.html>

From abarnert at yahoo.com  Sat Mar 28 00:13:46 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Mar 2015 16:13:46 -0700
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
Message-ID: <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>

On Mar 27, 2015, at 06:22, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
> 
> On 27 March 2015 at 01:40, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> But how is being "an array of objects" any different from what a tuple,
>> list, array.array, bytearray, bytes, str, etc. already are? What's
>> specifically array-like about this type as opposed to all of those? And
>> what's specifically record-like about your type compared to namedtuple,
>> Struct, or SimpleNamespace?
> 
> Acutally, on my understanding, the request on this thread is for
> something that is quite concrete,
> existing in other languages, and that can be done in Python in a few
> lines, but is not in
> the stdlib:
> 
> The Python equivalent of a C Struct.

But a C struct is not a subtype of, or substitutable for, a C array. It's not indexable. And the same is true with the equivalents in other languages. In fact, the dichotomy between struct--heterogeneous collection of fixed named fields--and array--homogeneous collection of indexed fields--goes back way before C. So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python.

And a class already is the Python of a C struct, it's just that it can do _more_ than a C struct. A language like C++ that wants to share code and values with C has to bend over backward to make it possible to write a C++ struct (or class) that doesn't use any of its extra features and is therefore exactly equivalent to a C struct, but Python has no need for that. (Except when you actually do want to share values with C code--but for that case, we've got ctypes.Struct, which is exactly what you want in that situation.)

> Just that.
> An easy to create class, with named fields,

Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes).

>  with possible
> type-enforcement for those fields.

Of course namedtuple doesn't have type-enforcement for the fields.

I'm not sure whether you're talking about MyPy static type checking, or runtime checking, but either way, it's easier to add onto a regular class than to a namedtuple-like class factory.

> Or maybe it _does_ exist in Python, and it is a matter of having a
> nice example in the docs:
> for example a "blank" class with "__slots__"  would do it.

A blank class without __slots__ can also do it. There are times when __slots__ are useful, but usually you're fine with just a plain __dict__. Encouraging people to use it when they have no need for it just because it's more like idiomatic C would be a bad idea. (It's like encouraging people to use @property to get something more like idiomatic .NET or ObjC, when actually they should just be writing idiomatic Python and using attributes directly.)

> Or a blank class with slots that could serialize and deserialize
> itself to a sequence
> in a seamless way.

Why do you want to serialize and deserialize to a sequence? A C struct can't do that, and neither can equivalent types in other languages.

> class Base:
>   __slots__ = ()
>   def __init__(self, seq=None):
>       if not seq: return
>       for attr, val in zip(self.slots, seq):
>            setattr(self, attr, val)
>   def __iter__(self):
>       for attr in self.__slots__:
>           yield getattr(self, attr)
> 
> 
> def NamedList(name, fields):
>   ... # split string with space separated fields, and other niceities here
>   return type(name, (Base,), dict(__slots__=fields))
> 
> And 10-15 more lines if one wants type-checking, default values,
> __repr__ into that.

Default values and __repr__ are _also_ not part of a C struct. So, again, if what you're looking for is the equivalent of a C struct, you can replace all of the above with:

    def Base: pass

If you want other features that C structs don't have, then yes, you may have to write them, but the same is true in C (and, in fact, it's clumsier and more difficult in C).

> Ithink getting a proper recipe for this, and
> publicizing it on the documentation ecosystem is enough  - maybe a
> Pypi module adding some more goodies - and if that would get any
> traction - the usual consideration for inclusion could apply.

From mrocklin at gmail.com  Sat Mar 28 00:19:54 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Fri, 27 Mar 2015 16:19:54 -0700
Subject: [Python-ideas] History on proposals for Macros?
Message-ID: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>

I want Python to have macros.  This is obviously a hard sell.  I'm willing
to do some legwork to demonstrate value.

What would a good proposal include?  Are there good examples of failed
proposals on this topic?

Is the core team open to this topic?

Thank you for your time,
 - Mathew Rocklin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/53a4bc61/attachment.html>

From carl at oddbird.net  Sat Mar 28 00:23:41 2015
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 27 Mar 2015 17:23:41 -0600
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
Message-ID: <5515E67D.6010102@oddbird.net>

On 03/27/2015 05:19 PM, Matthew Rocklin wrote:
> I want Python to have macros.  This is obviously a hard sell.  I'm
> willing to do some legwork to demonstrate value.
> 
> What would a good proposal include?  Are there good examples of failed
> proposals on this topic?
> 
> Is the core team open to this topic?

You probably want to have a look at MacroPy [1].

I don't think it was ever seriously proposed for core, though.

Carl


 [1] https://github.com/lihaoyi/macropy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/11298f20/attachment.sig>

From luciano at ramalho.org  Sat Mar 28 00:24:22 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Fri, 27 Mar 2015 20:24:22 -0300
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
Message-ID: <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>

On Fri, Mar 27, 2015 at 8:19 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:
> I want Python to have macros.  This is obviously a hard sell.  I'm willing
> to do some legwork to demonstrate value.

You're probably aware of this "prior art", but anyway, it's worth a
link. Looks solid to me:

https://github.com/lihaoyi/macropy

Cheers,

Luciano

-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
|     http://shop.oreilly.com/product/0636920032519.do
|  Professor em: http://python.pro.br
|  Twitter: @ramalhoorg

From alexander.belopolsky at gmail.com  Sat Mar 28 00:28:23 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 27 Mar 2015 19:28:23 -0400
Subject: [Python-ideas] Proposal for new augmented assignment operator
 for method calls
In-Reply-To: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
References: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
Message-ID: <CAP7h-xawz6M-Xj5RCNMZ0bhahA=8e7desN9p+axUvQek9uU9pg@mail.gmail.com>

On Fri, Mar 27, 2015 at 5:36 PM, Jamie Willis <jw14896.2014 at my.bristol.ac.uk
> wrote:

> Would this idea get much support if submitted as a PEP?


I should have read your post to the end before sending a witty reply.   I
think you are facing an uphill battle regardless of the syntax.  In many
ways even the augmented assignment operators that are already in the
language are "un-pythonic" and many dark corners have been discovered since
they were introduced.


> What are the potential concerns I could consider with the syntax (Skip has
> already pointed out that the "dot" is pretty invisible in the operator, so
> perhaps a different character could be used)?


@= would be a better choice, but @ has been taken for the matrix
multiplication operator.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/c4c68e9c/attachment.html>

From luciano at ramalho.org  Sat Mar 28 00:29:28 2015
From: luciano at ramalho.org (Luciano Ramalho)
Date: Fri, 27 Mar 2015 20:29:28 -0300
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
Message-ID: <CALxg4FUW_pCKXMVEpDON_zVCjE4qPsw63O4xWbDuwdihZDXCnw@mail.gmail.com>

On Fri, Mar 27, 2015 at 8:13 PM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python.

Yes, there is: iterable unpacking.

> Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes).

Boilerplate with lots of repetition, with little added value. For
example, in a basic __init__ each attribute name usually occurs three
times: as an argument name in the method declaration, and then twice
when it's assigned to self. Ruby does much better, for example.

Best,

Luciano

From abarnert at yahoo.com  Sat Mar 28 01:21:05 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Mar 2015 17:21:05 -0700
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
	Geo coordinates conversion in stdlib
In-Reply-To: <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
 <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>
 <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>
Message-ID: <6F7DC05A-2563-4DD1-B20A-F8F1C9D37630@yahoo.com>

On Mar 27, 2015, at 08:21, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
> 
> 
> On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>> Where have you been when PEP 3108 was discussed?  I have not seen any other list of Python modules that needed a redesign, so I cannot tell what's on your top ten list.
>> 
>> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html
> 
> Interesting.  I did not see this back in the day.  The top entry (urllib and friends) makes sense and there were heavily redesigned in Python 3.  I am surprised that distutils redesign got less support than logging and datetime. 

I'm not sure how much weight to put on an informal poll of 176 people self-selected as readers of a particular blog in the first place, but...

Back in those days, I think most people had no idea what they wanted from a redesign of distutils, and until at least one of the competing projects to extend/fix/replace it was ready for prime time (at least design-wise) the idea of changing the stdlib to follow one of them was a bit scary.

> I suspect that this may have to do with logging and datetime APIs not being PEP8 compliant.  Popular votes tend to exaggerate the importance of trivial things such as the spelling of class and method names and brush off more subtle, but important design flaws. 

Interesting point. And I think you're right--but I think you can take it farther.

Even beyond the PEP 8 renaming, what kinds of things did people really want from those modules? People also wanted the 90%-compatible formatting/parsing functions to be 100% Java-compatible for logging and 100% my-plafform's-C-lib-compatible for datetime. And they wanted to be able to plug pytz into datetime without making it so easy to write what looks like correct timezone-aware code but actually isn't. And they wanted easier conversion between datetime's types and those in time and elsewhere.

Those changes aren't as trivial as PEP 8 renaming, but they're still simple to express, concrete, and unambiguous (while still potentially requiring a backward-incompatible change). Who wouldn't vote for that?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/fd18b091/attachment-0001.html>

From abarnert at yahoo.com  Sat Mar 28 02:09:03 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Mar 2015 18:09:03 -0700
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CALxg4FUW_pCKXMVEpDON_zVCjE4qPsw63O4xWbDuwdihZDXCnw@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <CALxg4FUW_pCKXMVEpDON_zVCjE4qPsw63O4xWbDuwdihZDXCnw@mail.gmail.com>
Message-ID: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com>

On Mar 27, 2015, at 16:29, Luciano Ramalho <luciano at ramalho.org> wrote:
> 
> On Fri, Mar 27, 2015 at 8:13 PM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python.
> 
> Yes, there is: iterable unpacking.

Why?

You can't do the equivalent in C or any of its descendants (or most other languages with a struct/record type, or most pedagogical or theoretical struct/record concepts). Nor can you do anything even vaguely similar. So why would anyone expect that "the equivalent of a C struct" in Python should be able to do something that a C struct, and its equivalents in other languages, can't?

Also, the desire to _not_ have to use iterable unpacking is why we have namedtuple (and structseq in the C API) in the first place: to make tuples that can be used as records, not the other way around. A namedtuple stat result allows your users to access the fields by name instead of by index, which not only makes their code more readable, it also means stat can return different sets of extra fields on different platforms and in new versions without breaking their code. Even in C, this is important: because you access the PyObject fields by name, I can hand you a PyList* cast to a PyObject* and you can use it; if C allowed you to access it by iterable unpacking and you did so, I'd instead have to copy the PyObject fields of the PyList into a new PyObject that didn't have any extra fields.

>> Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes).
> 
> Boilerplate with lots of repetition, with little added value. For
> example, in a basic __init__ each attribute name usually occurs three
> times: as an argument name in the method declaration, and then twice
> when it's assigned to self. Ruby does much better, for example.

Let's compare some C code and the equivalent Python:

    struct Person {
        const char *name;
        int age;
    }
    struct Person person_make(const char *name, int age) {
        struct Person p;
        p.name = strdup(name);
        p.age = age;
        return p;
    }

    class Person
        def __init__(self, name: str, age: int):
            self.name = name
            self.age = age

You really think that this is not like a C struct because it has too much boilerplate compared to the C equivalent?

Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these. And to add in other functionality that C structs (and, except for the first, Python namedtuples) don't have that your project needs, like a nice repr, default values, runtime type checking, a JSON serialization schema, an ORM mapping, an HTML form representation, etc.

If you really want to add in being a sequence, you can add that too--but again, what's the use case for that? It's certainly not being more like a C struct.


From abarnert at yahoo.com  Sat Mar 28 02:31:12 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Mar 2015 18:31:12 -0700
Subject: [Python-ideas] Proposal for new augmented assignment operator
	for method calls
In-Reply-To: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
References: <CAPauRLOGAoc_LbSLAuzHG_XRJn6mraVjyR5Hxt=KC++Mzs6DFA@mail.gmail.com>
Message-ID: <D0F99CFB-DAA4-4EFD-80AD-DAF53E394C20@yahoo.com>

On Mar 27, 2015, at 14:36, Jamie Willis <jw14896.2014 at my.bristol.ac.uk> wrote:
> 
> (As posted on python-list)
> 
> I would like to propose a new piece of syntax for the python language; .=
> 
> In short, the operator is form of syntactic sugar, for instance consider the following code:
> 
>> hello = "hello world              "
>> hello = hello.strip()
> 
> This could be written as:
> 
>> hello = "hello world              "
>> hello .= strip()
> 
> In this slightly contrived example, the programmer saved (a small amount of) time when writing the code. With code with longer variable names, or lots of similar statements all in a row, this helps to keep code more concise, and could cut down on potential bugs due to typos (thanks to Skip Montanaro for pointing this out). 
> 
> The operator would be constricted to one method or field on the right-hand side, which must belong to the object on the left hand side.
> 
> To provide another example, consider this basic LinkedList class:
> 
>> class LinkedList:
>>     
>>     def __init__(self, data, head=None):
>>         self.data = data
>>         self.next = None
>>         self.head = head if head != None else self
>>  
>>     def append(self, data):
>>         new_node = LinkedList(data, self.head)
>>         self.next = new_node
>>         return new_node
>>  
>> linked_list = LinkedList(7)
>> linked_list = linked_list.append(8)
>> linked_list = linked_list.append(9)
>>  
>> loop_node = linked_list.head
>> while loop_node != None:
>>     print(loop_node.data)
>>     loop_node = loop_node.next

This is a pretty weird variation on the linked list data structure that's hard to imagine anyone actually using. And it seems like it's only because you've made it so weird that you need an append function that (unpythonically) mutates self and returns some other but related object. And it's only because you've made that append function that you need this new syntax at all.

It's certainly possible that a more reasonable class would also give you a use case for this feature, but it's hard to guess without seeing one.

Also, a much more Pythonic way to write your loop would be to write a list iterating function (whether an __iter__ method, or an external function) and just loop over that, which means the ugly "node = node.next" would either never appear, or appear only once inside that iterator rather than all over user code. 

Your improved version is only marginally better:

    loop_node = linked_list.head
    while loop_node != None:
        print(loop_node.data)
        loop_node .= next

While doing it right gives you:

    for loop_node in linked_list:
        print(loop_node.data)

(Also, as a side note, you shouldn't compare to None with !=; use is not.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/fae20605/attachment.html>

From jsbueno at python.org.br  Sat Mar 28 02:45:38 2015
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 27 Mar 2015 22:45:38 -0300
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <CALxg4FUW_pCKXMVEpDON_zVCjE4qPsw63O4xWbDuwdihZDXCnw@mail.gmail.com>
 <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com>
Message-ID: <CAH0mxTRxPmw-tXFM5c8n6Sz_28r==9_X3ifRDeq=gm_d3taCcA@mail.gmail.com>

On 27 March 2015 at 22:09, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these. And to add in other functionality that C structs (and, except for the first, Python namedtuples) don't have that your project needs, like a nice repr, default values, runtime type checking, a JSON serialization schema, an ORM mapping, an HTML form representation, etc.

So -that is the point - it is trivial to do away with the boiler
plate, as I've shown on the other message,  - but there is no way to
do it in the stdlib, so it is a wheel that is reinvented everytime.
NamedTuples are a way to do _almost_ that: make it tirvial to create a
class with only this fixed set of attributes, with as little
boilerplate as one can think of - but one can't change the attributes
on an instance of it.

That is why the start of the thread is about a "mutable named tuple" -
not because it is a tuple - but because it creates a basic class with
fixed attributes that works nicely, with minimal boiler plate.
Rethinking at my example, I think it does fit exactly in "too small to
be an external dependence in itself, and with too many subtle
behaviors to get done right again and again in several projects".

  js
 -><-

From rosuav at gmail.com  Sat Mar 28 03:01:30 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Mar 2015 13:01:30 +1100
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <CALxg4FUW_pCKXMVEpDON_zVCjE4qPsw63O4xWbDuwdihZDXCnw@mail.gmail.com>
 <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com>
Message-ID: <CAPTjJmoPdY8SpCQ25xB-GnKQ2bnZy+vSa_EGqGYi=4xXYph5DQ@mail.gmail.com>

On Sat, Mar 28, 2015 at 12:09 PM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
>     class Person
>         def __init__(self, name: str, age: int):
>             self.name = name
>             self.age = age
>
> Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these.

Here's a crazy thought: you could use functools.wraps() to abuse **kwargs.

def make_attributes(func):
    @functools.wraps(func)
    def inner(self, **args):
        self.__dict__.update(args)
        inner(self, **args)
    return inner

class Person:
    @make_attributes
    def __init__(self, *, name: str, age: int):
        pass

Thanks to wraps(), you still have your parameter names for
introspection and help() and so on. Thanks to **args, you can do bulk
operations on all the args.

It's a bit naughty (and it does preclude positional args, though a
little bit more work in the decorator could support that too), but it
would work.....

ChrisA

From abarnert at yahoo.com  Sat Mar 28 03:22:58 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Mar 2015 19:22:58 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <5515E67D.6010102@oddbird.net>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <5515E67D.6010102@oddbird.net>
Message-ID: <662C946E-992A-46EB-8B45-CACCB69D5B57@yahoo.com>

On Mar 27, 2015, at 16:23, Carl Meyer <carl at oddbird.net> wrote:
> 
>> On 03/27/2015 05:19 PM, Matthew Rocklin wrote:
>> I want Python to have macros.  This is obviously a hard sell.  I'm
>> willing to do some legwork to demonstrate value.
>> 
>> What would a good proposal include?  Are there good examples of failed
>> proposals on this topic?
>> 
>> Is the core team open to this topic?
> 
> You probably want to have a look at MacroPy [1].
> 
> I don't think it was ever seriously proposed for core, though.

I think it might be worth looking at what smaller core changes could allow MacroPy to be better or simpler than it is. Some of the changes to the import mechanism and the ast module that we've already had between 3.0 and 3.4 have done that, and also have the benefit of making a Python implementation easier to understand and to hack on. I suspect the same would be true for other possible changes. And then, once those changes are in and MacroPy is as simple, pleasant, and wart-free as it can be, it would probably be an easier sell (although still maybe not easy...) to get something like it incorporated into the core compiler and/or importer instead of as an external import hook.

Of course you may look at MacroPy and think, "This is way too big and complicated, I just want to be able to do what Language X had without learning all this other stuff", but then it'll at least help focus the ideas on what you do and don't want, right?

Anyway, I think people who proposed macros in the past have had two major problems.

First, they didn't think through the design. Lisp-style macros don't work in a language with syntax more complex than s-expressions. And making them partially text-based doesn't help either, given the way whitespace works in Python. And once you get to the point where you're doing AST transforms with the ast module, that doesn't really feel like macros anymore. If you instead come at it from the perspective of Dylan or one of the ML-with-macros variants, where what you're looking for is a way to write AST transformers in a declarative language that fits into the host language, you might get a lot father. 

Second, they didn't think through the marketing. Ask any famous Lispers why macros are important, and the answer is that they let you reprogram the language first, into a language that's easier to write your actual program in. That's a blatantly anti-Pythonic thing to do. You often can't read good Lisp code until you first learn the new, project-specific language they've built on Lisp, and nobody wants that for Python. Even smaller versions of that, like being able to create new flow control syntax, etc. are an anti-selling-point for Python. So, you need to think of a new way to sell macros that explicitly disavows the idea of creating new syntax, but still shows how macros can usefully do things that look and feel Pythonic but can't be done in Python.


From steve at pearwood.info  Sat Mar 28 14:37:09 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Mar 2015 00:37:09 +1100
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
Message-ID: <20150328133709.GE25453@ando.pearwood.info>

On Fri, Mar 27, 2015 at 04:13:46PM -0700, Andrew Barnert wrote:
> On Mar 27, 2015, at 06:22, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
[...]
> > The Python equivalent of a C Struct.
> 
> But a C struct is not a subtype of, or substitutable for, a C array. 
> It's not indexable. And the same is true with the equivalents in other 
> languages. In fact, the dichotomy between struct--heterogeneous 
> collection of fixed named fields--and array--homogeneous collection of 
> indexed fields--goes back way before C. So, if you want the equivalent 
> of a C struct, there's no reason to make it an iterable in Python.

Joao said "The Python equivalent of a C struct", not "a C struct". 
Python is not C, and Python data types are not limited to what C does. 
Python strings aren't limited to C null-delimited strings, and Python 
ints aren't limited to what C ints can do.

I think the original thread was quite explicit about what is wanted: 
something like a mutable equivalent to namedtuple. Namedtuples are used 
in Python where C would use a struct, or Pascal a record, except that 
namedtuples (being tuples) are immutable. I think it's quite reasonable 
to want a mutable version.

Effectively, namedtuple is just a convenience function for wrapping up a 
bunch of nice-to-have but not essential functionality around an 
immutable struct. Python got by with unnamed tuples for over a decade, 
so it's not like we *have* to have namedtuples. But having got them, 
would we go back to using regular tuples as a struct? Hell no. Having 
named fields is so much better.



> And a class already is the Python of a C struct, it's just that it can 
> do _more_ than a C struct.

This is why it is unfair to insist that a Python equivalent of a C 
struct be limited to what C structs do.


> > Just that.
> > An easy to create class, with named fields,
> 
> Which is easy to do: just create a class, and create its fields in the 
> __init__ method (or, in some cases, it's acceptable to use class 
> attributes as "default values" for instance attributes).

If this is so easy, why we have namedtuple *and* SimpleNamespace 
in the standard library. Are they both mistakes?

SimpleNamespace is especially interesting. The docs say:

"However, for a structured record type use namedtuple() instead."

https://docs.python.org/3/library/types.html#types.SimpleNamespace


which is great if you want an *immutable* structured record type, but 
not if you want a mutable one.

Which brings us back to where this thread started: a request for a 
mutable version of namedtuple. That's trickier than namedtuple, because 
we don't have a mutable version of a tuple to inherit from. Lists won't 
do the job, because they have a whole lot of functionality that are 
inappropriate, e.g. sort, reverse, pop methods.

That makes it harder to create a mutable structured record type, not 
simpler.

Think about the functional requirements:

- it should be semantically a struct, not a list or array;

- with a fixed set of named fields;

- fields should be ordered: a record with fields foo and bar is not the 
same as a record with fields bar and foo;

- accessing fields by index would be a Nice To Have, but not essential;

- but iteration is essential, for sequence unpacking;

- values in the fields must be mutable;

- it should support equality, but not hashing (since it is mutable);

- it must have a nice repr and/or str;

- being mutable, it may directly or indirectly contain a reference to 
itself (e.g. x.field = x) so it needs to deal with that correctly;

- support for pickle;

- like namedtuple, it may benefit from a handful of methods such as 
'_asdict', '_fields', '_make', '_replace' or similar.


Does this sound easy to write? Well, sure, in the big picture, it's 
hardly a 100,000 line application. But it's not a trivial class.



-- 
Steve

From rob.cliffe at btinternet.com  Sat Mar 28 15:56:02 2015
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Sat, 28 Mar 2015 14:56:02 +0000
Subject: [Python-ideas] Format character to center strings
Message-ID: <5516C102.1040608@btinternet.com>

As a string can be left-justified (within a given width) using e.g.
     '%-20s' % s
and right-justified using
     '%+20s' % s    #    or    '%20s' % s
why not allow a string to be centered using
     '%=20s' %s

Rob Cliffe

From mrocklin at gmail.com  Sat Mar 28 17:53:48 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Sat, 28 Mar 2015 09:53:48 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
Message-ID: <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>

Responding to comments off list:

I'm not referring to C-style preprocessor macros, I'm referring to macros
historically found in functional languages and commonly found in many
user-targeted languages built in the last few years.

The goal is to create things that look like functions but have access to
the expression that was passed in.

Some examples where this is useful:

    plot(year, miles / gallon)  # Plot with labels determined by
input-expressions, e.g. miles/gallon

    assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
assertRaises function, not before

    run_concurrently(f(x), f(y), f(z))  # Run f three times in three
threads controlled by run_concurrently

Generally one constructs something that looks like a function but, rather
than receiving a pre-evaluated input, receives a syntax tree along with the
associated context.  This allows that function-like-thing to manipulate the
expression and to control the context in which the evaluation occurs.

There are lots of arguments against this, mostly focused around potential
misuse.  I'm looking for history of such arguments and for a general "Yes,
this is theoretically possible" or "Not a chance in hell" from the
community.  Both are fine.

Cheers,
Matthew

On Fri, Mar 27, 2015 at 4:24 PM, Luciano Ramalho <luciano at ramalho.org>
wrote:

> On Fri, Mar 27, 2015 at 8:19 PM, Matthew Rocklin <mrocklin at gmail.com>
> wrote:
> > I want Python to have macros.  This is obviously a hard sell.  I'm
> willing
> > to do some legwork to demonstrate value.
>
> You're probably aware of this "prior art", but anyway, it's worth a
> link. Looks solid to me:
>
> https://github.com/lihaoyi/macropy
>
> Cheers,
>
> Luciano
>
> --
> Luciano Ramalho
> |  Author of Fluent Python (O'Reilly, 2015)
> |     http://shop.oreilly.com/product/0636920032519.do
> |  Professor em: http://python.pro.br
> |  Twitter: @ramalhoorg
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150328/ac9905cf/attachment.html>

From flying-sheep at web.de  Sat Mar 28 18:07:32 2015
From: flying-sheep at web.de (Philipp A.)
Date: Sat, 28 Mar 2015 17:07:32 +0000
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <5516C102.1040608@btinternet.com>
References: <5516C102.1040608@btinternet.com>
Message-ID: <CAN8d9g=muUrBmPLhtd2sh2==90kWEWVoB++iwdRPXOekj3C+Kg@mail.gmail.com>

because you should use '{:^20}'.format(s) instead of the obsolete % stuff.

Rob Cliffe <rob.cliffe at btinternet.com> schrieb am Sa., 28. M?rz 2015 um
16:01 Uhr:

> As a string can be left-justified (within a given width) using e.g.
>      '%-20s' % s
> and right-justified using
>      '%+20s' % s    #    or    '%20s' % s
> why not allow a string to be centered using
>      '%=20s' %s
>
> Rob Cliffe
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150328/ad0c3545/attachment.html>

From dwblas at gmail.com  Sat Mar 28 18:11:49 2015
From: dwblas at gmail.com (David Blaschke)
Date: Sat, 28 Mar 2015 10:11:49 -0700
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <5516C102.1040608@btinternet.com>
References: <5516C102.1040608@btinternet.com>
Message-ID: <CAFx9zocxNFD_qPDhV+aw8VoJ4aP5RF7w5phDuydkGGU3caHasg@mail.gmail.com>

Centered=same amount of white space on the left and to the right of
the text.  This depends on the font being used, especially whether it
is a fixed or proportional font.  So you would possibly have to also
specify the font face and font size being used.  Programming is not
like a word processor where the font is specified within each
document.  Run this program to see the display difference between two
"sentences" that are the same length, and between proportional and
fixed fonts.

#! /usr/bin/python

try:
       import Tkinter as tk     ## Python 2.x
except ImportError:
       import tkinter as tk     ## Python 3.x

class DifferentFonts():
    def __init__(self):
        self.top=tk.Tk()
        lit_1="Display lots of i's and l's-iiilll"
        lit_2="This is the same length as 1-wwwqq"
        ctr = 0
        for each_font in (('Verdana', 12),
                          ('Fixed', 12)):
            tk.Label(self.top, text=each_font[0]+"-"*30,
font=each_font).grid(row=ctr)
            ctr += 1
            tk.Label(self.top, text=lit_1,
font=each_font).grid(row=ctr, sticky="w")
            ctr += 1
            tk.Label(self.top, text=lit_2,
font=each_font).grid(row=ctr, sticky="w")
            ctr += 1

        tk.Button(self.top, text="Quit", bg="orange",
               command=self.top.quit).grid(row=20)

        self.top.mainloop()

DifferentFonts()


On 3/28/15, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
> As a string can be left-justified (within a given width) using e.g.
>      '%-20s' % s
> and right-justified using
>      '%+20s' % s    #    or    '%20s' % s
> why not allow a string to be centered using
>      '%=20s' %s
>
> Rob Cliffe
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Only when you actually get to the state where there is neither
delusion nor enlightenment are you finally comfortable...Foyan

From alexander.belopolsky at gmail.com  Sat Mar 28 18:12:16 2015
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Sat, 28 Mar 2015 13:12:16 -0400
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <CAN8d9g=muUrBmPLhtd2sh2==90kWEWVoB++iwdRPXOekj3C+Kg@mail.gmail.com>
References: <5516C102.1040608@btinternet.com>
 <CAN8d9g=muUrBmPLhtd2sh2==90kWEWVoB++iwdRPXOekj3C+Kg@mail.gmail.com>
Message-ID: <CAP7h-xYt931U9+4hXTwBxtMHUgBvta6hJgsZ_b0VVLCHvj+ymQ@mail.gmail.com>

On Sat, Mar 28, 2015 at 1:07 PM, Philipp A. <flying-sheep at web.de> wrote:

> because you should use '{:^20}'.format(s) instead of the obsolete % stuff.
>

"% stuff" is not "obsolete", but it follows C standard by design and any
new features that do not come from C will likely be rejected.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150328/c8d00c31/attachment.html>

From steve at pearwood.info  Sat Mar 28 18:26:24 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Mar 2015 04:26:24 +1100
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
Message-ID: <20150328172624.GG25453@ando.pearwood.info>

On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote:
[...]
> The goal is to create things that look like functions but have access to
> the expression that was passed in.

>     assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
> assertRaises function, not before

> Generally one constructs something that looks like a function but, rather
> than receiving a pre-evaluated input, receives a syntax tree along with the
> associated context.  This allows that function-like-thing to manipulate the
> expression and to control the context in which the evaluation occurs.

How will the Python compiler determine that assertRaises should receive 
the syntax tree rather than the evaluated 1/0 expression (which of 
course will raise)? The information that assertRaises is a "macro" is 
not available at compile time.

I really like the idea of delaying the evaluation of certain expressions 
until a time of the caller's choosing, but I don't see how to make that 
work automatically. Of course we can do it manually by wrapping the 
expression in a function, or by writing it as a string and compiling it 
with compile() for later eval'ing, but the sort of thing you show above 
strikes me as fundamentally incompatible with Python's execution model.

Happy to be proven wrong though.


-- 
Steve

From steve at pearwood.info  Sat Mar 28 18:37:51 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Mar 2015 04:37:51 +1100
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <CAFx9zocxNFD_qPDhV+aw8VoJ4aP5RF7w5phDuydkGGU3caHasg@mail.gmail.com>
References: <5516C102.1040608@btinternet.com>
 <CAFx9zocxNFD_qPDhV+aw8VoJ4aP5RF7w5phDuydkGGU3caHasg@mail.gmail.com>
Message-ID: <20150328173751.GH25453@ando.pearwood.info>

On Sat, Mar 28, 2015 at 10:11:49AM -0700, David Blaschke wrote:
> Centered=same amount of white space on the left and to the right of
> the text.  This depends on the font being used, especially whether it
> is a fixed or proportional font.

You're over-thinking it. Python already offers two ways of centering 
strings, with the distance measured in number-of-characters rather than 
pixels or points.

py> "Hello world!".center(20)
'    Hello world!    '
py> "{:^20}".format("Goodbye everyone!")
' Goodbye everyone!  '

Left and right justification also pad the string to a fixed width in 
characters, not in pixels.


-- 
Steve

From ericsnowcurrently at gmail.com  Sat Mar 28 19:11:34 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 28 Mar 2015 12:11:34 -0600
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <20150328133709.GE25453@ando.pearwood.info>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <20150328133709.GE25453@ando.pearwood.info>
Message-ID: <CALFfu7B9fs_jg=UipUaorm=KA0irZg7QLUqn5PPEPRzGAdjKwQ@mail.gmail.com>

On Sat, Mar 28, 2015 at 7:37 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Effectively, namedtuple is just a convenience function for wrapping up a
> bunch of nice-to-have but not essential functionality around an
> immutable struct. Python got by with unnamed tuples for over a decade,
> so it's not like we *have* to have namedtuples. But having got them,
> would we go back to using regular tuples as a struct? Hell no. Having
> named fields is so much better.

+1, though it doesn't *necessarily* follow that a mutable equivalent
is a good idea.  This (and related threads) imply there is at least
some support for the new type in principle.  I haven't followed the
threads too closely so I've missed any mention of solid pythonic use
cases that would give the idea much more solid footing.  However, I
have seen references to prior art on the cheeseshop which may be used
to provide harder evidence (both of support and of solid use cases).

Regardless, I concur that there are many cases where types and
functions have been added to the stdlib that weren't strictly
necessary.  Perhaps if those proposals had come from someone else or
when the mood on python-dev was different then they would not have
been added.  That is what has happened with numerous other
we-have-gotten-by-without-it-so-why-add-it ideas (which may also have
proven themselves as namedtuple has).

Ultimately we have to be careful in this space because, as Raymond
often reminds us, it really is important to make the effort to keep
Python small enough to fit in people's brains (and in *some* regard
we've probably failed there already).  With the great efforts in the
last year to improve packaging, the cheeseshop is increasingly the
better place for new types and helpers to live.

With that in mind, perhaps we should start adding a section to the
bottom of relevant docs that contains links to vetted PyPI packages
(and recipes) that provide extended capabilities.  We've done this
already in a few select places (e.g. the 3.2+ namedtuple docs).

> If this is so easy, why we have namedtuple *and* SimpleNamespace
> in the standard library. Are they both mistakes?
>
> SimpleNamespace is especially interesting. The docs say:
>
> "However, for a structured record type use namedtuple() instead."
>
> https://docs.python.org/3/library/types.html#types.SimpleNamespace

As the person who wrote that I'll point out that I added it to help
make the distinction clearer between the two.  At the time there were
concerns about the similarities and with users getting confused about
which to use.  I will argue that "record type" implies an archive of
data, ergo immutable.  "Structured" refers to being ordered and having
attribute access.  IMHO that statement is clear and helpful, but if it
has proven otherwise we should consider improving it.

In contrast, I see the proposal here as somewhat of a middle ground.
Folks are looking for a factory mechanism that produces classes with
slots and have both iteration (for unpacking) and index lookup.  So
something like this:

class FixedClassMeta(type):
    # Ideally this would be a "classonly" method (like classmethod but
    # class-only) method on FixedClass and not need a metaclass.
    def subclass(base, name, *fields):
        # XXX validate fields first
        args = ', '.join(fields)
        body = '\n    '.join('self.{0} = {0}'.format(f) for f in fields)
        code = """def __init__(self, {}):\n    {}""".format(args, body)
        ns = {}
        exec(code, ns)
        class X(base):
            __slots__ = fields
            __init__ = ns['__init__']
        X.__name__ = name
        X.__qualname__ = X.__qualname__.replace('X', name, 1)
        X.__doc__ = """..."""
        return X


class FixedClass(metaclass=FixedClassMeta):
    __slots__ = ()
    def __repr__(self):
        items = ("{}={!r}".format(f, getattr(self, f)) for f in self.__slots__)
        return "{}({})".format(self.__class__.__name__, ', '.join(items))
    def __iter__(self):  # for unpacking
        return (getattr(self, f) for f in self.__slots__)
    def __getitem__(self, index):
        field = self.__slots__[index]
        try:
            return getattr(self, field)
        except AttributeError:
            raise IndexError(index)
    # Index lookup exists for convenience, but assignment & deletion
    # are not in scope.


def fixedClass(name, field_names):
    """A factory that produces classes with fixed, ordered attributes.

    The returned class has __slots__ set to the field names, as well as
    __iter__ (for unpacking) and __getitem__ implemented.

    """
    if isinstance(field_names, str):
        fields = field_names.replace(',', ' ').split()
    else:
        fields = field_names
    return FixedClass.subclass(name, *fields)


That said, I'm still not clear on what the use cases are.

> Which brings us back to where this thread started: a request for a
> mutable version of namedtuple. That's trickier than namedtuple, because
> we don't have a mutable version of a tuple to inherit from. Lists won't
> do the job, because they have a whole lot of functionality that are
> inappropriate, e.g. sort, reverse, pop methods.
>
> That makes it harder to create a mutable structured record type, not
> simpler.
>
> Think about the functional requirements:
>
> - it should be semantically a struct, not a list or array;

This is the key point.  It is a fixed-size class with iteration for
unpacking and index lookup for convenience.  A full-fledged mutable
namedtuple doesn't make sense (without clear use cases).

>
> - with a fixed set of named fields;
>
> - fields should be ordered: a record with fields foo and bar is not the
> same as a record with fields bar and foo;

Ah, my example above would have to grow __eq__ then.

>
> - accessing fields by index would be a Nice To Have, but not essential;

Exactly.  Not Essential.

>
> - but iteration is essential, for sequence unpacking;

This brings to mind a different proposal that has come up in the past
(a separate "dunder" method for unpacking).  Iteration seems out of
place for here, but we need it for sequence unpacking.

>
> - values in the fields must be mutable;
>
> - it should support equality, but not hashing (since it is mutable);
>
> - it must have a nice repr and/or str;
>
> - being mutable, it may directly or indirectly contain a reference to
> itself (e.g. x.field = x) so it needs to deal with that correctly;

Ah, yes.  "RuntimeError: maximum recursion depth exceeded". :)

>
> - support for pickle;
>
> - like namedtuple, it may benefit from a handful of methods such as
> '_asdict', '_fields', '_make', '_replace' or similar.

Perhaps.  I think there are a few things we can learn from namedtuple
that can be applied for this hypothetical new type/factory.

And to add to your list:

- performance should be a consideration since the apparent use cases
relate to handling many of these as "records".

Again, I'm not sold on the benefit of this over the existing
alternatives.  For records use namedtuple (with the _replace method
for "mutation").

-eric

From breamoreboy at yahoo.co.uk  Sat Mar 28 19:11:44 2015
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Sat, 28 Mar 2015 18:11:44 +0000
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <5516C102.1040608@btinternet.com>
References: <5516C102.1040608@btinternet.com>
Message-ID: <mf6qt4$r6b$1@ger.gmane.org>

On 28/03/2015 14:56, Rob Cliffe wrote:
> As a string can be left-justified (within a given width) using e.g.
>      '%-20s' % s
> and right-justified using
>      '%+20s' % s    #    or    '%20s' % s
> why not allow a string to be centered using
>      '%=20s' %s
>
> Rob Cliffe
>

This http://bugs.python.org/issue23624 was rejected so I see two chances 
of this happening, zero or none.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From ericsnowcurrently at gmail.com  Sat Mar 28 19:14:02 2015
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 28 Mar 2015 12:14:02 -0600
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <CALFfu7B9fs_jg=UipUaorm=KA0irZg7QLUqn5PPEPRzGAdjKwQ@mail.gmail.com>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <20150328133709.GE25453@ando.pearwood.info>
 <CALFfu7B9fs_jg=UipUaorm=KA0irZg7QLUqn5PPEPRzGAdjKwQ@mail.gmail.com>
Message-ID: <CALFfu7BZ++3ePPXnXfO3wmhbz8Xwpyr5PXgNLfxpROHT8VZ9HA@mail.gmail.com>

On Sat, Mar 28, 2015 at 12:11 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Mar 28, 2015 at 7:37 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> - but iteration is essential, for sequence unpacking;
>
> This brings to mind a different proposal that has come up in the past
> (a separate "dunder" method for unpacking).

What ever became of that proposal?  If I recall correctly, it had
moderate support but the champion wasn't able to continue pursuing it.

-eric

From abarnert at yahoo.com  Sat Mar 28 19:57:46 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Mar 2015 11:57:46 -0700
Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple
In-Reply-To: <20150328133709.GE25453@ando.pearwood.info>
References: <CALxg4FVVuwpT7O-cmOY=k9vBQhPpDT+oa-gMDm-K-v3Cq2bKvw@mail.gmail.com>
 <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com>
 <CAB1GNpSRoFMzBA-4o0LSdpaiF4tjqoWComO28xRCKAHQAMgdKA@mail.gmail.com>
 <CAPOE21RPB2wacQFW0Z3K-4Ldc3e7VdgmvY2gUt4HkvVVVt2vfg@mail.gmail.com>
 <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com>
 <CAH0mxTTBKb9_EtnxgZDROnniZDdTfhh5EJ=WAFZJEbrEYEBKmg@mail.gmail.com>
 <F70CA51B-8A1A-4B6F-8C4A-9FAC93967D60@yahoo.com>
 <20150328133709.GE25453@ando.pearwood.info>
Message-ID: <7694034E-55FA-423D-A953-955D0B6215E0@yahoo.com>

On Mar 28, 2015, at 06:37, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Fri, Mar 27, 2015 at 04:13:46PM -0700, Andrew Barnert wrote:
>> On Mar 27, 2015, at 06:22, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
> [...]
>>> The Python equivalent of a C Struct.
>> 
>> But a C struct is not a subtype of, or substitutable for, a C array. 
>> It's not indexable. And the same is true with the equivalents in other 
>> languages. In fact, the dichotomy between struct--heterogeneous 
>> collection of fixed named fields--and array--homogeneous collection of 
>> indexed fields--goes back way before C. So, if you want the equivalent 
>> of a C struct, there's no reason to make it an iterable in Python.
> 
> Joao said "The Python equivalent of a C struct", not "a C struct". 
> Python is not C, and Python data types are not limited to what C does. 
> Python strings aren't limited to C null-delimited strings, and Python 
> ints aren't limited to what C ints can do.

Sure, but nobody just invents random new features to add to int and then justifies them by saying "I want the equivalent of a C int" even though C int doesn't have those features. People invent features (like bit_length) to solve actual use cases, and justify them based on those use cases.

Multiple people have asked "what do you want this for?", and the best answer anyone's given has been "the equivalent of a C struct". (That, and to prematurely and badly optimize memory usage.)

Even worse, when I ask why specifically anyone wants this thing to be iterable, the answer is "to be the equivalent of a C struct", and that doesn't answer the question.

[snip]

>> And a class already is the Python of a C struct, it's just that it can 
>> do _more_ than a C struct.
> 
> This is why it is unfair to insist that a Python equivalent of a C 
> struct be limited to what C structs do.

I'm not saying a record type shouldn't be allowed to have any features that C structs don't, just that equivalency with C structs isn't an argument for features that C structs don't have.

Some of the extra features are so obviously desirable that they probably don't need any argument--if you're going to build this thing, having a nice repr or not breaking pickle seems hard to argue against. But iterability is not that kind of obvious win.

Also, how is it "unfair" to suggest that this thing should be limited in some ways? For example, two instances of the same class can have completely different fields; presumably two instances of the same record type really shouldn't. There's no reason it _couldn't_ be completely open like a general class, it's just that you usually don't want it to be. Similarly, there's no reason it couldn't be a sequence, but I don't think you usually want it to be.

[snip]

> Which brings us back to where this thread started: a request for a 
> mutable version of namedtuple. That's trickier than namedtuple, because 
> we don't have a mutable version of a tuple to inherit Sent from my iPhone
> do the job, because they have a whole lot of functionality that are 
> inappropriate, e.g. sort, reverse, pop methods.
> 
> That makes it harder to create a mutable structured record type, not 
> simpler.

Sure. And that's the problem. If you want something that's "just like a sequence whose elements can be replaced but whose shape is fixed, except that the elements are also named", you run into the problem that Python doesn't have such a sequence type. It's a perfectly coherent concept, and there's no reason you could design a language around immutable, fixed-shape-mutable, and mutable-shape sequences instead of just the first and last, but that's not the way Python was designed. Should that be changed? Or is the only use for such a type to underlie this new type?

> Think about the functional requirements:
> 
> - it should be semantically a struct, not a list or array;
> 
> - with a fixed set of named fields;
> 
> - fields should be ordered: a record with fields foo and bar is not the 
> same as a record with fields bar and foo;

Note that namedtuples are nominatively typed, not structurally--a record with fields foo and bar is not necessarily the same as another record with fields foo and bar. Ordering doesn't enter into it; they were defined separately, so they're separate types. Do you want the same behavior here, or the behavior your description implies instead?

> - accessing fields by index would be a Nice To Have, but not essential;

Why would that be nice to have? The record/sequence dichotomy has been fundamental to the design of languages since the earliest days, and it's still there in almost all languages. Maybe every language in the world is wrong--but if so, surely you can explain why?

For structseq, there was a good reason: a stat result is a 7-tuple as well as being a record with 13-odd fields, because there was a pre-existing mass of code that used stat results as 7-tuples, but people also wanted to be able to access the newer or not-100%-portable fields. That's a great use case. And people have used structseq in other similar examples to migrate users painlessly from an early API that turned out to be too simple and limited. And namedtuple gives you a way to write APIs in a similar style that previously could only be (easily) written with a C extension, which is an obvious win. That's clearly not the case here--nobody has existing APIs that use a fixed-length but mutable sequence that they want to expand into something more flexible, because Python doesn't come with such a sequence type.

Of course that's not the only use anyone's ever found for, respectively, structseq and namedtuple--e.g., converting to namedtuple turns out to be handy for cases where you want a record but some external API like SQL gives you a sequence, and that would probably be a good enough justification for namedtuple too. But what is the use that justifies this addition? (For example, if you need to take SQL rows as a sequence, mutate them by name, and then do something I can't imagine with them that requires them to still be a sequence, that would be a pretty good answer.)

> - but iteration is essential, for sequence unpacking;

Again, why is that essential?

TOOWTDI isn't an iron-clad rule, but it's a good rule of thumb; adding a second way to access the members of a record that's both unique to Python and less Pythonic seems like a bad idea, unless there's some good reason that overbalances it in the other direction. Think of stat code: it's a lot more readable when you access the fields by name instead of by unpacking. Why wouldn't the same be true for, say, a Person record, or an Address record, or an ImageHeader record, or almost anything else you can imagine?

(I can think of one particular special case where it might be nice: small, homogenous, essentially-sequence-like records like a Vector or Point or... Well, really just a Vector or Point. And they're clearly special. Both in C and in Python, you're often torn between storing them as an array or as a sequence, and you'll find different apps doing it each way. That isn't true for a Person or Address etc.)

> - values in the fields must be mutable;
> 
> - it should support equality, but not hashing (since it is mutable);
> 
> - it must have a nice repr and/or str;
> 
> - being mutable, it may directly or indirectly contain a reference to 
> itself (e.g. x.field = x) so it needs to deal with that correctly;
> 
> - support for pickle;
> 
> - like namedtuple, it may benefit from a handful of methods such as 
> '_asdict', '_fields', '_make', '_replace' or similar.
> 
> 
> Does this sound easy to write? Well, sure, in the big picture, it's 
> hardly a 100,000 line application. But it's not a trivial class.
> 
> 
> 
> -- 
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rob.cliffe at btinternet.com  Sat Mar 28 19:59:01 2015
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Sat, 28 Mar 2015 18:59:01 +0000
Subject: [Python-ideas] Format character to center strings
In-Reply-To: <mf6qt4$r6b$1@ger.gmane.org>
References: <5516C102.1040608@btinternet.com> <mf6qt4$r6b$1@ger.gmane.org>
Message-ID: <5516F9F5.2020100@btinternet.com>



On 28/03/2015 18:11, Mark Lawrence wrote:
> On 28/03/2015 14:56, Rob Cliffe wrote:
>> As a string can be left-justified (within a given width) using e.g.
>>      '%-20s' % s
>> and right-justified using
>>      '%+20s' % s    #    or    '%20s' % s
>> why not allow a string to be centered using
>>      '%=20s' %s
>>
>> Rob Cliffe
>>
>
> This http://bugs.python.org/issue23624 was rejected so I see two 
> chances of this happening, zero or none.
>
Except that that referred to a proposed change that would break existing 
code.  This is a proposed new feature which wouldn't.  (And for what 
it's worth, I too would prefer it to add an odd pad character on the right.)

From abarnert at yahoo.com  Sat Mar 28 20:50:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Mar 2015 12:50:09 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <20150328172624.GG25453@ando.pearwood.info>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
Message-ID: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>

On Mar 28, 2015, at 10:26, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote:
>> [...]
>> The goal is to create things that look like functions but have access to
>> the expression that was passed in.
> 
>>    assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
>> assertRaises function, not before
> 
>> Generally one constructs something that looks like a function but, rather
>> than receiving a pre-evaluated input, receives a syntax tree along with the
>> associated context.  This allows that function-like-thing to manipulate the
>> expression and to control the context in which the evaluation occurs.
> 
> How will the Python compiler determine that assertRaises should receive 
> the syntax tree rather than the evaluated 1/0 expression (which of 
> course will raise)? The information that assertRaises is a "macro" is 
> not available at compile time.

Well, it _could_ be available. At the time you're compiling a scope (a function, class, or top-level module code), if it uses an identifier that's the name of a macro in scope, the compiler expands the macro instead of compiling in a function call.

Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible.

Of course that doesn't mean it's a good idea. It would require some pretty significant changes that may not be immediately obvious (e.g., the whole .pyc mechanism no longer works as-is if you can import macros from other modules, which isn't an issue for __future__ statements...). And debugging could be a nightmare--you now have to know what names are defined at call time, and also what names were defined at definition time, to trace through a function. And so on.

> I really like the idea of delaying the evaluation of certain expressions 
> until a time of the caller's choosing, but I don't see how to make that 
> work automatically. Of course we can do it manually by wrapping the 
> expression in a function,

That doesn't work for all uses of macros--a macro can swap two variables, or break or return, etc., and a higher-order function whose arguments are delayed by wrapping them in a function can't do that. But it does work for _many_ uses of macros, like this one.

And if you look at languages with light lambda syntax, a lot of things that you'd naturally write as a macro in Lisp, you instead naturally write as a plain higher-order function, and the result is often more readable, not less. (Explicit is better than implicit, but you sometimes don't realize it if the explicitness forces you to write more boilerplate than actual code...) Consider:

    assertRaises(ZeroDivisionError, :1/0) 

That ":1/0" means the same thing as "lambda: 1/0". And now "wrapping the expression in a function" doesn't seem so bad.

If such a light-lambda syntax reduced the desire for macros down to the point where it could be ignored, and if that desire weren't _already_ low enough that it can be ignored, it would be worth adding. I think the second "if" is where it fails, not the first, but I could be wrong.




From jheiv at jheiv.com  Sat Mar 28 23:35:17 2015
From: jheiv at jheiv.com (James Edwards)
Date: Sat, 28 Mar 2015 18:35:17 -0400
Subject: [Python-ideas] "Loose" descriptors
Message-ID: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>

(I apologize in advance if this was covered previously or elsewhere; a
fairly thorough search didn't yield anything.)

Currently, we can implement a descriptor as follows:

    class VerboseDescriptor(object):
        def __init__(self, init):
            self.val = init

        def __get__(self, obj, typ=None):
            print("Get is %s" % (self.val))
            return self.val

        def __set__(self, obj, val):
            print("Set to %s" % (val))
            self.val = val

    class Holder(object):
        val = VerboseDescriptor(2)


    foo = Holder()
    _ = foo.val
    foo.val = 4

Which of course works as expected, displaying:

    Get is 2
    Set to 4

But we can't achieve the same indirection without a "Holder"-type class.

That is, we can't do the following:

    class VerboseDescriptor(object):
        def __init__(self, init):
            self.val = init

        def __get__(self, obj, typ=None):
            print("Get is %s" % (self.val))
            return self.val

        def __set__(self, obj, val):
            print("Set to %s" % (val))
            self.val = val


    val = VerboseDescriptor(2)
    _ = val
    val = 4

Or rather, we can, but what might be expected (the same functionality
/ output), is not what occurs.

My understanding of the resolution of the first example is that python
looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it
finds it, identifies that `Holder.val` implements `__get__`, and
handles the indirection accordingly.

So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`.

But my question is would it make sense to be able to do this sort of
indirection for "loose" objects also, not just attributes of classes
or instances?  In other words, could descriptors be extended so that
they perform the same sort of indirection in the second example as the
first?

I envision (but with no real knowledge of the implications of this)
that just as python identifies that `val` in `Holder.__dict__`
implements `__get__`, python could similarly identify that `val` in
`globals()` (or `sys.modules[__name__].__dict__`, or wherever is more
appropriate) implements `__get__` and handle the indirection.

Of course, a syntax issue would arise if this were possible -- notably
the class definition of `Holder` would break.  So some syntax
adjustment would have to be made.  But is this reasonable / of
interest / attainable?

Thanks for your time and consideration.

From rosuav at gmail.com  Sat Mar 28 23:45:18 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Mar 2015 09:45:18 +1100
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
Message-ID: <CAPTjJmpUG42eMjv4W9NRHP+ODhTma1Y0S=50ty3LOKqsLxdLtA@mail.gmail.com>

On Sun, Mar 29, 2015 at 6:50 AM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible.

To be technically accurate, __future__ directives do not affect
"subsequent compilation" - they affect the compilation of the one
module they're at the top of, you can't do this:

def func1(f):
    print "Hello, world!"
    f("Haha")

from __future__ import print_function

def func2():
    func1(print)

ChrisA

From greg.ewing at canterbury.ac.nz  Sat Mar 28 23:51:31 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 29 Mar 2015 11:51:31 +1300
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
Message-ID: <55173073.9070600@canterbury.ac.nz>

James Edwards wrote:
> I envision (but with no real knowledge of the implications of this)
> that just as python identifies that `val` in `Holder.__dict__`
> implements `__get__`, python could similarly identify that `val` in
> `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more
> appropriate) implements `__get__` and handle the indirection.

This would slow down *all* lookups of global names in
order to support a rarely-used feature. Since global
name lookup is critical performance-wise, this is
likely to meet a lot of resistance.

Similar arguments have been made concerning the
support of descriptors in module namespaces, which
is a closely related idea.

-- 
Greg



From steve at pearwood.info  Sun Mar 29 03:51:44 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Mar 2015 12:51:44 +1100
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
Message-ID: <20150329015144.GJ25453@ando.pearwood.info>

On Sat, Mar 28, 2015 at 12:50:09PM -0700, Andrew Barnert wrote:
> On Mar 28, 2015, at 10:26, Steven D'Aprano <steve at pearwood.info> wrote:
> > 
> >> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote:
> >> [...]
> >> The goal is to create things that look like functions but have access to
> >> the expression that was passed in.
> > 
> >>    assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
> >> assertRaises function, not before
> > 
> >> Generally one constructs something that looks like a function but, rather
> >> than receiving a pre-evaluated input, receives a syntax tree along with the
> >> associated context.  This allows that function-like-thing to manipulate the
> >> expression and to control the context in which the evaluation occurs.
> > 
> > How will the Python compiler determine that assertRaises should receive 
> > the syntax tree rather than the evaluated 1/0 expression (which of 
> > course will raise)? The information that assertRaises is a "macro" is 
> > not available at compile time.
> 
> Well, it _could_ be available. At the time you're compiling a scope (a 
> function, class, or top-level module code), if it uses an identifier 
> that's the name of a macro in scope, the compiler expands the macro 
> instead of compiling in a function call.

Perhaps I trimmed out too much of Matthew's comment, but he did say he 
isn't talking about C-style preprocessor macros, so I think that if you 
are imagining "expanding the macro" C-style, you're barking up the wrong 
tree. From his description, I don't think Lisp macros are quite the 
right description either. (As always, I welcome correction if I'm the 
one who is mistaken.)

C macros are more or less equivalent to a source-code rewriter: if you 
define a macro for "foo", whenever the compiler sees a token "foo", it 
replaces it with the body of the macro. More or less.

Lisp macros are different, and more powerful:

http://c2.com/cgi/wiki?LispMacro
http://cl-cookbook.sourceforge.net/macros.html

but I don't think that's what Matthew wants either. The critical phrase 
is, I think:

"one constructs something that looks like a function but, rather than 
receiving a pre-evaluated input, receives a syntax tree along with the 
associated context"

which I interpret in this way:

Suppose we have this chunk of code creating then using a macro:

    macro mymacro(expr):
        # process expr somehow

    mymacro(x + 1)


That is *roughly* equivalent (ignoring the part about context) to what 
we can do today:

    def myfunction(expr):
        assert isinstance(expr, ast.Expression)
        # process expr somehow

    tree = ast.parse('x + 1', mode='eval')
    myfunction(tree)


The critical difference being that instead of the author writing code to 
manually generate the syntax tree from a string at runtime, the compiler 
automatically generates the tree from the source code at compile time.

This is why I think that it can't be done by Python. What should the 
compiler do here?

callables = [myfunction, mymacro]
random.shuffle(callables)
for f in callables:
    f(x + 1)


If that strikes you as too artificial, how about a simpler case?

from mymodule import name
name(x + 1)


If `name` will refer to a function at runtime, the compiler needs to 
generate code which evaluates x+1 and passes the result to `name`; but 
if `name` will refer to a macro, the compiler needs to generate an ast 
(plus context) and pass it without evaluating it. To do that, it needs 
to know at compile-time which objects are functions and which are 
macros, but that is not available until runtime.

But we might be able to rescue this proposal by dropping the requirement 
that the compiler knows when to pass the syntax tree and when to 
evaluate it. Suppose instead we had a lightweight syntax for generating 
the AST plus grabbing the current context:

    x = 23
    spam(x + 1, !(x+1))  #  macro syntax !( ... )


Now the programmer is responsible for deciding when to use an AST and 
when to evaluate it, not the compiler, and "macros" become regular 
functions which just happen to expect an AST as their argument.


[...]
> If such a light-lambda syntax reduced the desire for macros down to 
> the point where it could be ignored, and if that desire weren't 
> _already_ low enough that it can be ignored, it would be worth adding. 
> I think the second "if" is where it fails, not the first, but I could 
> be wrong.

I presume that Matthew wants the opportunity to post-process the AST, 
not merely evaluate it. If all you want is to wrap some code an an 
environment in a bundle for later evaluation, you are right, a function 
will do the job. But it's hard to manipulate byte code, hence the desire 
for a syntax tree.


-- 
Steve

From rosuav at gmail.com  Sun Mar 29 04:13:41 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 29 Mar 2015 13:13:41 +1100
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info>
Message-ID: <CAPTjJmq=TZUAjSi-oJrUU2WdC2DZtW_CEJsTG52_UnD++qsY1w@mail.gmail.com>

On Sun, Mar 29, 2015 at 12:51 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> But we might be able to rescue this proposal by dropping the requirement
> that the compiler knows when to pass the syntax tree and when to
> evaluate it. Suppose instead we had a lightweight syntax for generating
> the AST plus grabbing the current context:
>
>     x = 23
>     spam(x + 1, !(x+1))  #  macro syntax !( ... )
>
>
> Now the programmer is responsible for deciding when to use an AST and
> when to evaluate it, not the compiler, and "macros" become regular
> functions which just happen to expect an AST as their argument.

This is actually a lot more plausible than most of the other theories
(except the preprocessor, but anyone can do that, and it's not
necessarily going to help you any). If the magic macro operator is
given precedence equivalent to lambda, it would often be possible to
do it without the parens, too. In the same way that "lambda: expr"
yields an object (a function) rather than evaluating its argument, the
macro syntax would yield an object (an AST tree or somesuch) without
actually evaluating the expression.

ChrisA

From steve at pearwood.info  Sun Mar 29 04:17:07 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 29 Mar 2015 13:17:07 +1100
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <55173073.9070600@canterbury.ac.nz>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
 <55173073.9070600@canterbury.ac.nz>
Message-ID: <20150329021707.GK25453@ando.pearwood.info>

On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote:
> James Edwards wrote:
> >I envision (but with no real knowledge of the implications of this)
> >that just as python identifies that `val` in `Holder.__dict__`
> >implements `__get__`, python could similarly identify that `val` in
> >`globals()` (or `sys.modules[__name__].__dict__`, or wherever is more
> >appropriate) implements `__get__` and handle the indirection.
> 
> This would slow down *all* lookups of global names in
> order to support a rarely-used feature. Since global
> name lookup is critical performance-wise, this is
> likely to meet a lot of resistance.
> 
> Similar arguments have been made concerning the
> support of descriptors in module namespaces, which
> is a closely related idea.

I don't think it's a closely related idea, I think it is *exactly* the 
same idea. If I've understood him correctly, James wants to use 
descriptors in global variables, i.e. module namespaces, not just class 
attributes.

You are right that making this the default behaviour would cause 
slow-down of all module-level lookups for very rare advantage, but what 
if we moved the descriptor logic into the global __dict__ instead of the 
compiler? Then, in thory at least, supporting the descriptor protocol 
for global variables will be opt-in on a per-module basis.

Here's a sketch of how you might do it.

class DescriptorDict(dict):
    def __getitem__(self, key):
        obj = super().__getitem__(key)
        if hasattr(type(obj), '__get__'):
            # I'm not sure what to pass as instance here.
            return type(obj).__get__(obj, instance)
        return obj


__dict__ = DescriptorDict(__dict__)


I'm sure there are complications I haven't even begun to imagine, but 
here are a few obvious ones:

(1) Can modules use a dict subclass for __dict__? 
(2) Can you replace the __dict__ on the fly like this?
(3) If not, how do you bootstrap the module to have a magic dict instead 
    of an ordinary dict?
(4) The descriptor protocol expects to pass the instance and/or class 
    to the __get__ method, as well as `self`. self is the descriptor 
    itself; what is instance? The module?


I don't know if it can be done, but if it could be, it would certainly 
open the doors for a lot of interesting experiments.



-- 
Steve

From dw+python-ideas at hmmz.org  Sun Mar 29 04:34:05 2015
From: dw+python-ideas at hmmz.org (David Wilson)
Date: Sun, 29 Mar 2015 02:34:05 +0000
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
Message-ID: <20150329023405.GA7521@k3>

On Sat, Mar 28, 2015 at 06:35:17PM -0400, James Edwards wrote:

> But my question is would it make sense to be able to do this sort of
> indirection for "loose" objects also, not just attributes of classes
> or instances?  In other words, could descriptors be extended so that
> they perform the same sort of indirection in the second example as the
> first?

FWIW you could probably experiment with this using PyPy's object spaces
feature without modifying the interpreter:

    http://pypy.readthedocs.org/en/latest/objspace-proxies.html

The 'exec' statement also indirectly supports a restrictive version of
this, by allowing dict subclasses to be provided for globals/locals.
Naturally the effect wouldn't carry over to function calls invoked by
the statement.

As a general comment, the ability to overload scope load/store seems
like an expert feature of questionable applicability. It's fun to play
with, but I'd hope to never encounter it in code intended for
maintainability.


David

From abarnert at yahoo.com  Sun Mar 29 05:58:25 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Mar 2015 20:58:25 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAPTjJmpUG42eMjv4W9NRHP+ODhTma1Y0S=50ty3LOKqsLxdLtA@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <CAPTjJmpUG42eMjv4W9NRHP+ODhTma1Y0S=50ty3LOKqsLxdLtA@mail.gmail.com>
Message-ID: <006D5417-D16B-41A7-8B6E-0395FEDA2D58@yahoo.com>

On Mar 28, 2015, at 15:45, Chris Angelico <rosuav at gmail.com> wrote:
> 
> On Sun, Mar 29, 2015 at 6:50 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible.
> 
> To be technically accurate, __future__ directives do not affect
> "subsequent compilation" - they affect the compilation of the one
> module they're at the top of, you can't do this:
> 
> def func1(f):
>    print "Hello, world!"
>    f("Haha")
> 
> from __future__ import print_function
> 
> def func2():
>    func1(print)

Except at the interactive interpreter, where you can write exactly that, and it will do exactly what you'd want/expect.


From abarnert at yahoo.com  Sun Mar 29 07:29:29 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Mar 2015 22:29:29 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info>
Message-ID: <9834EACC-EEA3-4F1B-B20D-4BD6F9399B92@yahoo.com>

On Mar 28, 2015, at 18:51, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Sat, Mar 28, 2015 at 12:50:09PM -0700, Andrew Barnert wrote:
>>> On Mar 28, 2015, at 10:26, Steven D'Aprano <steve at pearwood.info> wrote:
>>> 
>>>> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote:
>>>> [...]
>>>> The goal is to create things that look like functions but have access to
>>>> the expression that was passed in.
>>> 
>>>>   assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
>>>> assertRaises function, not before
>>> 
>>>> Generally one constructs something that looks like a function but, rather
>>>> than receiving a pre-evaluated input, receives a syntax tree along with the
>>>> associated context.  This allows that function-like-thing to manipulate the
>>>> expression and to control the context in which the evaluation occurs.
>>> 
>>> How will the Python compiler determine that assertRaises should receive 
>>> the syntax tree rather than the evaluated 1/0 expression (which of 
>>> course will raise)? The information that assertRaises is a "macro" is 
>>> not available at compile time.
>> 
>> Well, it _could_ be available. At the time you're compiling a scope (a 
>> function, class, or top-level module code), if it uses an identifier 
>> that's the name of a macro in scope, the compiler expands the macro 
>> instead of compiling in a function call.
> 
> Perhaps I trimmed out too much of Matthew's comment, but he did say he 
> isn't talking about C-style preprocessor macros, so I think that if you 
> are imagining "expanding the macro" C-style, you're barking up the wrong 
> tree.

No, I'm not imagining C style expansion. What I'm imagining is closer to Lisp-style, and closer still to Dylan-style.* I didn't want to get into details because they're complicated, and there even may be multiple complicated ways to do things that have to be chosen from, and none of that is relevant to your question. But if you're curious, let me give a more specific explanation:

A macro is compiled by transforming its AST into AST-transforming bytecode, similar to a function, but then instead of embedding a BUILD_FUNCTION opcode into the defining scope's bytecode, you do that, and _also_ effectively call it in the current scope and bind the macro to the name there.**

A macro is expanded by parsing the arguments into ASTs, calling the AST-transforming function on those ASTs, and substituting the result into the tree at the point of the macro call.*** You don't need to explicitly "pass" a context; the context in which the function is called (the compile-time scope, etc.) is implicitly available, just as for runtime functions.

As I mentioned before, there are a number of additional issues you'd have to resolve (again, think import, and .pyc files, for an example), some of which may make the feature undesirable once you think them through, but I don't think any of them are relevant to your question, and I think something like this design is what he was asking about.

* In Lisp, because there is no syntax and hence no separate step to turn a parenthesized token stream into an AST, it's ambiguous at which stage--before or after that non-existent step--macros are applied. In languages with syntax and grammars, the usual answer is to do it after parsing to AST. You can conceptually define macros at any stage in the pipeline, but that's where they turn out to be most useful. Also, I'm ignoring the issue of hygiene, but I think most people want macros to be hygienic by default and unhygienic only on explicit demand, rather than what Lisp does.

** The details here are tricky because the compiler's notion of "current scope" isn't defined by the language anyway and doesn't correspond to anything defined at runtime, but the intuitive idea is clear: while compiling a module or other scope, if you reach a def or class, a compiler has to do the equivalent of recursively calling itself or pushing a scope onto a stack manually; that stack defines the compile-time scope. So the name bound to the macro goes away when you exit that recursive call/pop that scope from the stack. The compiler currently keeps track of all variables assigned to in the current scope to determine local variables and closures; I believe macro assignments can be piled on top of that. If not, this is something completely new that you have to bolt on.

*** In most languages with syntax and macros, a macro can only take an expression, and must return an expression--which is fine for most of those languages, where almost everything (flow control, variable binding, etc.) is an expression, but not so much in Python, where many of those things can only be done in a statement. But allowing a statement or an expression (or any arbitrary AST node) and allowing a macro to likewise "return" either (or any) type isn't straightforward once you think through some examples.

> From his description, I don't think Lisp macros are quite the 
> right description either. (As always, I welcome correction if I'm the 
> one who is mistaken.)
> 
> C macros are more or less equivalent to a source-code rewriter: if you 
> define a macro for "foo", whenever the compiler sees a token "foo", it 
> replaces it with the body of the macro. More or less.
> 
> Lisp macros are different, and more powerful:
> 
> http://c2.com/cgi/wiki?LispMacro
> http://cl-cookbook.sourceforge.net/macros.html
> 
> but I don't think that's what Matthew wants either. The critical phrase 
> is, I think:
> 
> "one constructs something that looks like a function but, rather than 
> receiving a pre-evaluated input, receives a syntax tree along with the 
> associated context"
> 
> which I interpret in this way:
> 
> Suppose we have this chunk of code creating then using a macro:
> 
>    macro mymacro(expr):
>        # process expr somehow
> 
>    mymacro(x + 1)
> 
> 
> That is *roughly* equivalent (ignoring the part about context) to what 
> we can do today:
> 
>    def myfunction(expr):
>        assert isinstance(expr, ast.Expression)
>        # process expr somehow
> 
>    tree = ast.parse('x + 1', mode='eval')
>    myfunction(tree)
> 
> 
> The critical difference being that instead of the author writing code to 
> manually generate the syntax tree from a string at runtime, the compiler 
> automatically generates the tree from the source code at compile time.
> 
> This is why I think that it can't be done by Python. What should the 
> compiler do here?
> 
> callables = [myfunction, mymacro]
> random.shuffle(callables)
> for f in callables:
>    f(x + 1)

Well, that depends on how much of Python you want to be available at compile time. One possibility is that only the defmacro statement is executed at compile time. You may want to add imports to that. You may want to add assignments. You may want to add some large, ridiculously-complex, but well-defined subset of Python (I would ask you to think C++ constexpr rules, but that request might be construed as torture.) Or you may even want the entire language. And any of the above could be modified by making some or all of those constructs compile-time only when explicitly required (again, like constexpr).

Any of these is conceptually sensible (although none of them may be desirable...), and they give you different answers here. For example, if you only execute defmacro and import at compile time, then at compile time f is not expanded as a macro, it's just called as a function, which will probably raise a TypeError at runtime (because the current value of x + 1 is probably not an AST node...).

> If that strikes you as too artificial, how about a simpler case?
> 
> from mymodule import name
> name(x + 1)
> 
> 
> If `name` will refer to a function at runtime, the compiler needs to 
> generate code which evaluates x+1 and passes the result to `name`; but 
> if `name` will refer to a macro, the compiler needs to generate an ast 
> (plus context) and pass it without evaluating it. To do that, it needs 
> to know at compile-time which objects are functions and which are 
> macros, but that is not available until runtime.

This is exactly what I was referring to when I said that you need to make significant changes to other parts of the language, such as revising the import machinery and .pyc files, and that this problem does not apply to future statements.

I don't want to go over all the details in as much depth as the last question, so hopefully you'll just accept that the answer is the same: it's not available today, but it could be available, in a variety of different ways that you'd have to choose between, all of which would have different knock-on effects.

> But we might be able to rescue this proposal by dropping the requirement 
> that the compiler knows when to pass the syntax tree and when to 
> evaluate it. Suppose instead we had a lightweight syntax for generating 
> the AST plus grabbing the current context:
> 
>    x = 23
>    spam(x + 1, !(x+1))  #  macro syntax !( ... )

That's a lot closer to what MacroPy does. And notice that if the !() syntax were added to the grammar, MacroPy or something like it could be significantly simpler.* Which means it might be easier to integrate it directly into the builtin compiler--but also means it might be less desirable to do so, as leaving it as an externally-supplied import hook has all the benefits of externally-supplied modules in general. (However, there is still the disadvantage that you have to apply an import hook before importing, effectively meaning you can't use or define macros in your top-level script. If necessary, you could fix that as well with special syntax that must come before anything but comments and future statements that adds an import hook. This is the kind of thing I was talking about in my first message, about finding smaller changes to the language that make MacroPy or something like it simpler and/or more flexible.)

* I believe OCaml recently added something similar for similar purposes, but I haven't used it in a few years, so I may be misinterpreting what I saw skimming the what's new for the last major version.

> Now the programmer is responsible for deciding when to use an AST and 
> when to evaluate it, not the compiler, and "macros" become regular 
> functions which just happen to expect an AST as their argument.

No, not quite. What do you _do_ with the AST returned by the macro? And when do you do it? You still have to substitute it into the tree of the current compilation target in place of the macro call, which means it still has to be available at compile time. It does provide a simpler way to resolve some of the other issues, but it doesn't resolve the most fundamental one.

> [...]
>> If such a light-lambda syntax reduced the desire for macros down to 
>> the point where it could be ignored, and if that desire weren't 
>> _already_ low enough that it can be ignored, it would be worth adding. 
>> I think the second "if" is where it fails, not the first, but I could 
>> be wrong.
> 
> I presume that Matthew wants the opportunity to post-process the AST, 
> not merely evaluate it. If all you want is to wrap some code an an 
> environment in a bundle for later evaluation, you are right, a function 
> will do the job. But it's hard to manipulate byte code, hence the desire 
> for a syntax tree.

Sure, but most of the examples people give for wanting macros--including his example that you quoted--don't actually do anything that can't be done with a higher-order function. Which means they may not actually want macros, they just think they do.

Ask a Haskell lover why Haskell doesn't need macros, and he'll tell you that it's because you don't need them, you only think you do because of your Lisp prejudices. Of course that isn't 100% true,* but it's true enough that most people are happy without macros in Haskell. Similarly, while it would be even farther from 100% true in Python,** it might still be true enough that most people are happy without macros in Python. (Except, as I said, most people are _already_ happy without macros in Python, which means we may have an even simpler option: just do nothing.)

* For one reasonably well-known example (although I may be misremembering, so take this as a "this kind of thing" rather than "exactly this..."), if Haskell98 had macros, you could use them to simulate GADTs, which didn't exist until a later version of the language. For an equivalent example in Python: you could use macros to simulate with statements in Python 2.5. As long as non-silly cases for macros are rare enough, people are satisfied with evaluating them at language design time (a discussion on the list followed by an update to the language and a patch to GHCI/CPython) instead of compile time. :)

** The main reason it would be less true in Python is eager evaluation; a lazy language like Haskell (or, even better, a dataflow language) can replace even more uses of macros with HOFs than an eager language. For example, in Haskell, the equivalent of "def foo(x, y): return y if x else 0" doesn't need the value of y unless x is true, so it doesn't matter that it y is a value rather than an expression. But OCaml, for example, also doesn't have lazy evaluation, and people seem to have the same attitude toward macros there too. (Although it does have a powerful preprocessor, it's not that much different from what Python has with import hooks.)

Well, despite trying to skim over some parts, I still wrote a whole book here; apologies for that, to anyone who's still reading. :)

From abarnert at yahoo.com  Sun Mar 29 07:36:37 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Mar 2015 22:36:37 -0700
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <20150329021707.GK25453@ando.pearwood.info>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
 <55173073.9070600@canterbury.ac.nz>
 <20150329021707.GK25453@ando.pearwood.info>
Message-ID: <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com>

On Mar 28, 2015, at 19:17, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote:
>> James Edwards wrote:
>>> I envision (but with no real knowledge of the implications of this)
>>> that just as python identifies that `val` in `Holder.__dict__`
>>> implements `__get__`, python could similarly identify that `val` in
>>> `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more
>>> appropriate) implements `__get__` and handle the indirection.
>> 
>> This would slow down *all* lookups of global names in
>> order to support a rarely-used feature. Since global
>> name lookup is critical performance-wise, this is
>> likely to meet a lot of resistance.
>> 
>> Similar arguments have been made concerning the
>> support of descriptors in module namespaces, which
>> is a closely related idea.
> 
> I don't think it's a closely related idea, I think it is *exactly* the 
> same idea. If I've understood him correctly, James wants to use 
> descriptors in global variables, i.e. module namespaces, not just class 
> attributes.

Yeah, that's how I interpreted it too.

The obvious way to do this is to provide some way for a module to define not only an instance of the module type, but a subclass and an instance of that subclass. The subclass then provides a natural place for the descriptors to live (and a natural answer to what gets passed for inst and typ).

There was a proposal for something very much like that some time late last year, IIRC, which might be worth digging up.

(Modules that don't define a subclass could still be easily optimized for fast global lookup, because the compiler knows that the base module class doesn't have any descriptors.)

> You are right that making this the default behaviour would cause 
> slow-down of all module-level lookups for very rare advantage, but what 
> if we moved the descriptor logic into the global __dict__ instead of the 
> compiler? Then, in thory at least, supporting the descriptor protocol 
> for global variables will be opt-in on a per-module basis.
> 
> Here's a sketch of how you might do it.
> 
> class DescriptorDict(dict):
>    def __getitem__(self, key):
>        obj = super().__getitem__(key)
>        if hasattr(type(obj), '__get__'):
>            # I'm not sure what to pass as instance here.
>            return type(obj).__get__(obj, instance)
>        return obj
> 
> 
> __dict__ = DescriptorDict(__dict__)
> 
> 
> I'm sure there are complications I haven't even begun to imagine, but 
> here are a few obvious ones:
> 
> (1) Can modules use a dict subclass for __dict__? 
> (2) Can you replace the __dict__ on the fly like this?
> (3) If not, how do you bootstrap the module to have a magic dict instead 
>    of an ordinary dict?
> (4) The descriptor protocol expects to pass the instance and/or class 
>    to the __get__ method, as well as `self`. self is the descriptor 
>    itself; what is instance? The module?
> 
> 
> I don't know if it can be done, but if it could be, it would certainly 
> open the doors for a lot of interesting experiments.
> 
> 
> 
> -- 
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From techtonik at gmail.com  Fri Mar 27 14:52:22 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 16:52:22 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPTjJmpyqiNgWgMCnVHfWTEyFQ2C8Bs0gTfBOs3R1uhzMv6gaQ@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <CAPTjJmpyqiNgWgMCnVHfWTEyFQ2C8Bs0gTfBOs3R1uhzMv6gaQ@mail.gmail.com>
Message-ID: <CAPkN8xKX1imsRaLxU6Za3zme=mWq_KO0pCGspGaQ6jKbkX_EOA@mail.gmail.com>

On Mon, Mar 23, 2015 at 10:21 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Mar 23, 2015 at 5:08 PM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > Handling geo coordinates in most used formats is as important as
> > handling .zip archives in stdlib.
>
> That may or may not be the case, but if it is, the best way to argue
> the point is:
>
> 1) Find or create a third-party package on PyPI
> 2) Demonstrate that this package has a stable API
> 3) Demonstrate or pledge support for the package
> 4) Appeal for the package's inclusion in the stdlib
>
> You'll need to show that there's benefit to having it in the stdlib as
> opposed to on PyPI - demonstrate the problem with "pip install
> packagename" that means that you want it in the standard library. Just
> *stating* that there is a need isn't sufficient - you need to prove
> it.
>

Why the wave module is there? Or ioctl module? Are they so widely used?

This process is flawed. If anybody would sponsor the research on popularity
of stdlib, I believe that math will be excluded from top 50 as well. Will
stdlib
without math will still be ok?

The measure to estimate the usefullness of standard library is not in
download score, but in coverage across different areas. There thousands of
people downloading stuff from youtube, so should we include youtube-dl
instead of math?
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/1c7e80dc/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 15:30:09 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 17:30:09 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com>
Message-ID: <CAPkN8xLpL0fmyU1OZCVA-c78NxtK4tgTx4rA2Ohs=mUg541Q2Q@mail.gmail.com>

On Mon, Mar 23, 2015 at 11:29 AM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Mar 22, 2015, at 11:08 PM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> >
> > On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert
> > <abarnert at yahoo.com.dmarc.invalid> wrote:
> >> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >>>
> >>> On Sun, 22 Mar 2015 17:15:39 -0400
> >>> Alexander Walters
> >>> <tritium-list at sdamon.com> wrote:
> >>>>> On 3/21/2015 05:46, anatoly techtonik wrote:
> >>>>> I think it should be in stdlib.
> >>>>>
> >>>>> 11? 18? 0? S, 2? 42? 0? E   to  -11.3, 2.7  and back.
> >>>>>
> >>>>> This is from Wikipedia.
> >>>> Why does this need to be in the standard library?
> >>>
> >>> Because Anatoly needs it, of course :)
> >>
> >> Well, it would certainly be nice to have all of Wikipedia in the
> stdlib, but I wonder how that would affect Python's release schedule, and
> its use in resource-constrained systems.
> >>
> >> More seriously, if you're just looking to convert arcminutes and
> arcseconds to and from fractional degrees, it's exactly the same algorithm
> as converting minutes and seconds to fractional hours, which we already
> have in the datetime.timedelta class. If you can't figure out the
> complexity of multiplying and divmodding by 60 twice, you can borrow its
> code, or just wrap it and delegate. That seems trivial enough to do in any
> project that needs it. And that means projects that want
> degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just
> degrees and fractional arcminutes with no arcseconds (e.g., for marine
> navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US
> military scopes) can just as easily get what they need, instead of being
> forced to use the variation surveyors prefer.
> >
> > That's nice to know, but IIRC datetime is from the top 10 Python
> > modules that need a redesign. Things contained therein doesn't pass
> > human usability check, and are not used as a result.
>
> There are thousands of projects that actually do use it; thousands of
> people asking questions on StackOverflow, Python-tutor, etc. and getting
> actual answers; etc. So your theoretical argument that it's not usable
> doesn't really matter given that practically it is actually usable, and
> used.
>

The fact that thousands of people are asking question on SO about datetime
is the main indicator that the module fails usability check. If it passed,
there was no need to ask stuff. If docs were good, the rate of question was
much lower.

And when I say about top 10 modules that need a redesign, it is not
abstract.

http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html

> My story for that day is that I needed to process geodata from
> > Wikipedia to see the distribution over the Earth surface, so I thought
> > that if I know the Python it is not a problem. It appeared that the
> > problem is there, and the problem is recurring (last time I had GPS
> > points).
>
> I suspect that people who do a lot of geodata processing in Python already
> have third-party libs that they use, and would have no interest in a very
> small limited subset being duplicated in the stdlib.


Actually, these people are journalists and we'd like to teach them some
Python scripting. If regexps and file reading operations are there, it is
still a problem to do an interesting case without installing a lot of
stuff, meaning that they are not using Python, but some Python
distribution. Maybe you're right - every type Python user should have a
specialized Python distribution like Anaconda or WinPython, but the most
basic entrypoint should be in stdlib.

There should be evolution in learning, going from math -> SciPy, urlib2 ->
requests, etc. The Python should come with toys for most major uses of
scripting languages in modern world. It is already long past only "computer
science" stuff.


> > Python docs explain a lot about core OS concepts, but Python
> > is not only a systems language. I'd say that it is now being replaced
> > with Go for all system stuff,
>
> I don't know anyone outside Google who's using Go for any system (or
> sysadmin, which seems to be more likely what you're talking about) stuff.
> I'm sure there are exceptions, but the idea that Python is being
> universally replaced by Go in that area is so ridiculous that it's hard to
> believe you actually meant it seriously. It's like saying that Ruby is
> being replaced by Scala for all web server stuff just because Scala is the
> fastest-growing language in that area this week.


Well, maybe it is only my impression, but I haven't seen any wide used user
level app recently written in Python, and there is a plenty of tools
written in Go for the areas that were once Python domain - system tools.
Docker, LXD, Syncthing - these are the tools that I actually use almost
daily, and I am looking at https://github.com/limetext/lime right now for
some reason.

> and more attempts are made to use is as
> > a data processing language. But that is not possible if Python doesn't
> > have a basic concepts of data representation from the Real Life (RL)
> > (tm) in its stdlib.
>
> Most people I know who use Python as a serious data processing language
> don't care much what's in the stdlib because they're using numpy and/or
> pandas.


Yes, and that's sad that the Python still doesn't have the basic building
blocks of Pandas in stdlib. It is the proven de-facto popularity
achievement that is required for packages to enter stdlib, but that still
doesn't happen. Looks like the granularity of package is not sufficient for
designing next stdlib - the need is to descent into the level of structures.


> > Handling geo coordinates in most used formats is as important as
> > handling .zip archives in stdlib.
>
> Not even close. Zip archives are used in a ridiculously wide range of
> application areas; geoccordinates are used only in processing geocoordinate
> data (and there only in small enough scales that you don't need numpy or
> some specialized geodata library that I don't know about but that I'll bet
> exists).


There are indeed a lot of proliferation in geo standards, but for the basis
stuff I trust the Wikipedia choice, because the practicality beats purity.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/f28a08f8/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 15:38:30 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 17:38:30 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <550FDA07.5020704@egenix.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <550FDA07.5020704@egenix.com>
Message-ID: <CAPkN8xKvUoQfjrbU1dFRd04OZCg=OtyMHt8XioUB8dv8PrnrKg@mail.gmail.com>

On Mon, Mar 23, 2015 at 12:16 PM, M.-A. Lemburg <mal at egenix.com> wrote:

>
> BTW: If you expect the Python stdlib to make coffee, clean your house
> and drive your kids to school, you're probably looking at the
> wrong language - we only do batteries in the stdlib :-)
>

Well, I got your point, it is not about GPS module anymore, it is about
inability
of community decisions on the stuff that looks like highly subjective and
personal. I don't think I can get convincing data for that, because nobody
has the money to spend time on that data mining stuff, especially me.

So the final argument would need to be extended to do the research on
Python usage in non-computer science and system programming world
to see if GPS conversion module in standard library provides any value.

That means that we need to find the metrics that will be more useful than
just a PyPI download index.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/d7f39bf8/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 15:52:52 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 17:52:52 +0300
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPTjJmrD5t1nGR59yJ1UBqoOG314jei+Q8OCCXus1oSeAMfiwg@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <CAPTjJmpyqiNgWgMCnVHfWTEyFQ2C8Bs0gTfBOs3R1uhzMv6gaQ@mail.gmail.com>
 <CAPkN8xKX1imsRaLxU6Za3zme=mWq_KO0pCGspGaQ6jKbkX_EOA@mail.gmail.com>
 <CAPTjJmrD5t1nGR59yJ1UBqoOG314jei+Q8OCCXus1oSeAMfiwg@mail.gmail.com>
Message-ID: <CAPkN8xK396kYtRhST_8gMR_g2Ah5ePTN7-eZTDtGk_cqRBXgRA@mail.gmail.com>

On Fri, Mar 27, 2015 at 5:04 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sat, Mar 28, 2015 at 12:52 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > Why the wave module is there? Or ioctl module? Are they so widely used?
> >
> > This process is flawed. If anybody would sponsor the research on
> popularity
> > of stdlib, I believe that math will be excluded from top 50 as well. Will
> > stdlib
> > without math will still be ok?
> >
> > The measure to estimate the usefullness of standard library is not in
> > download score, but in coverage across different areas. There thousands
> of
> > people downloading stuff from youtube, so should we include youtube-dl
> > instead of math?
>
> The criteria for the *removal* from the stdlib would be insanely
> tight, compared to the criteria for *addition*. So some modules might
> now simply be grandfathered in (I suspect wave is one of them), due to
> the shift in ease of grabbing stuff off PyPI. But the math module is
> pretty important and fundamental, so it's not going anywhere.
>
> As to youtube-dl, though: Definitely not. No matter how popular it is,
> it still fails the criterion of release schedule synchronization - it
> needs to be agile enough to update as often as it needs, according to
> host site changes, and would be shackled rather badly by inclusion in
> the stdlib. (You probably don't want to get it from your OS repo
> either.)
>
> But none of this changes the recommendations I made earlier. You need
> to *demonstrate*, not simply state, a need; and one of the best ways
> to demonstrate support, API stability, usage, etc, is to stick a
> package up on PyPI. Not everything needs to be in the stdlib, but if
> you're going to argue that something should be, what better way than
> by pointing out what it's already doing?


There is now quite a big lump of data for me that in social systems (such
as community developed projects) the reputation is more important than
rationale, so how do you imagine a demonstration? Well, I have this:

https://en.wikipedia.org/wiki/List_of_solar_eclipses_in_the_21st_century
here is the list of all sun eclipses in the World. How would you show
them on map to paste into your student paper? You need to import the
table into your spreadsheet, then transform coordinates into floats and
then export the data for you map drawing tools. The problem -
spreadsheets don't convert coordinates. You need a script for that.
And that script you will need for much every data processing involving
data from Wikipedia.

Is that fair enough for demonstration of a need?

-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/8e5e3058/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 15:54:09 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 17:54:09 +0300
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
Message-ID: <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>

On Mon, Mar 23, 2015 at 8:21 PM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
> >
> > That's nice to know, but IIRC datetime is from the top 10 Python
> > modules that need a redesign. Things contained therein doesn't pass
> > human usability check, and are not used as a result.
>
>
> Where have you been when PEP 3108 was discussed?  I have not seen any
> other list of Python modules that needed a redesign, so I cannot tell
> what's on your top ten list.
>

http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/7ba9d8ae/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 16:05:52 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 18:05:52 +0300
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <mepnrr$ova$1@ger.gmane.org>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
 <mepnrr$ova$1@ger.gmane.org>
Message-ID: <CAPkN8xLRby2yKwydKC3OvTKxuzDi2vMjrqbDvHX6h_Bo9NjmOA@mail.gmail.com>

On Mon, Mar 23, 2015 at 9:59 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/23/2015 1:21 PM, Alexander Belopolsky wrote:
>
>>
>> On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik
>> <techtonik at gmail.com
>> <mailto:techtonik at gmail.com>> wrote:
>>  >
>>  > That's nice to know, but IIRC datetime is from the top 10 Python
>>  > modules that need a redesign. Things contained therein doesn't pass
>>  > human usability check, and are not used as a result.
>>
>
> ...
>
>> in what sense does it not "pass human usability check"?
>>
> ...
>
> Please don't feed the troll.  He is known for opinionated digs like the
> above and historically, nothing good comes from trying to discuss them.


Well, I admit that it takes a really bad mood and anger to write stuff
about anything, and it is not very constructive state of mind. The rest of
the things I could just probably silently enjoy (or spend time on something
else).

Whatever. To make anything constructive out of it, people need to accept
the "usability" and "user experience" as a study disciplines, and that the
studies could be and should be applied to Python (in this case to stdlib).
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/387dbafe/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 17:05:44 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 19:05:44 +0300
Subject: [Python-ideas] Python Research Insitute
Message-ID: <CAPkN8x+M+ALDmXOG1tJRJdp2B3LXTBwzLDnBuzErUiuxF17z4Q@mail.gmail.com>

I am well aware of https://www.blender.org/institute/ and I like it.

And recently I realized that my claims and accusations towards
PSF are just wrong. PSF is a fund - it should process the money
to ensure Python Community has at lest some degree of
independence from sponsors. It is not its role to register domains,
maintain servers, assign copyright to itself from contributors  - it
is not its role to do anything at all.

PSF waits for initiatives to appear and then supports them, but
the problem is - there are not many initiatives (except website,
which is well, <opinionated opinion stripped>). And that's the
problem that PSF is not solving, because it can not do this. It can
not just do what it wants - grab a bunch of people together to
solve a particular problem, provide them with a place to do stuff,
with access to funding from people interested in certain initiative,
direct contact with them, being interested people from outside
into the team and do other nasty things (like filming movies!).

PSF holds a lot of responsibility and therefore is not effective
for research works and experiments. This things that PSF can
not do, and nobody can, because economy doesn't allow this:

1.  Maintain a list of top 10 stdlib modules that need a redesign

The one is gathered there:
http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html

Try with Python community a usability study to find out "why
people feel these things need a redesign".

2. Do economic study on Python contributions

How many people could escape the corporate "slavery" to do
the necessary work for high quality patches. Are they allowed
to work on Python, do they feel stressed, etc. How do they feel
the economic model helps Python development, is there a way
for alternative models that help more people to jump in to make
good thing better.

3. Get data on Python services, develop participation tools

bugs.python.org is not only a GSoC project. There are a lot of
things that can be improved, should be improved and the issues
are not trivial.

4. Publish a research what will happen if Python 2.8 is released

Study the things like
https://gist.github.com/techtonik/574401cebf5e17c33f5b
with reports on income and outcomes that could actually be
referenced (unlike mails, which are good, but hard to filter).

5. Get data on Python usage, a set of areas and most popular use cases

This is came from Geo coordinates conversion module in stdlib.

>> Most people I know who use Python as a serious data processing language
don't care much what's in the stdlib because they're using numpy and/or
pandas.
> Yes, and that's sad that the Python still doesn't have the basic building
blocks of Pandas in stdlib.

Study the idea of making a stlib that consists from modules that cover the
most common scenarios of using the scripting language, provide a "toy
modules" that cover these common scenarios - a mini demos of more powerful
tools, like a "swiss-knives toolbelt".

6. Get a tools on measuring Python stdlib usage

Build a linked semantic AST from all packages on PyPI (vector data like
OpenStreetMap, but from the language level objects). This will allow to
grep how many times a particular module is used, and allow people to use
that database to extend into their own research (like trying to guess
context and usage scenarios for stdlib calls).

7. Record a summaries of Python evolution, build media materials

Visual information requires much much much less time to be consumed, but
requires a significant amount of time to produce. PyCon talks are
exceptionally good for video material, but for the community coordination,
a lot of HTML5 and dynamic design works needs to be done.

PSF won't be able to make it interesting to HTML5 folks, because it is
solely about Python itself, and the object of Python Research Institute
include everything that goes side by side and influences the development of
the project and community around it. From "cultural aspects of Monty Python
in context of computer language programming" to "feasibility of designing
an operating system in stackless" and "test of new economic models for
sustainable Python development" etc.

I just can't stop with ideas - research how appearance of startup scene
affected open source contributions, research why open source project die
and how to prevent that, what happened with former Python contributors and
where their interest is now, how other languages compare to Python, what
are the key problems of converting Python code to other languages, how to
deal with complexity etc., what this "agile" is about and could it be
useful for Python development etc.


Please, tell me if that is a worthy utopia and add your ideas.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/2f4a8696/attachment-0001.html>

From techtonik at gmail.com  Fri Mar 27 17:13:48 2015
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 27 Mar 2015 19:13:48 +0300
Subject: [Python-ideas] Top 10 Python modules that need a redesign Was:
 Geo coordinates conversion in stdlib
In-Reply-To: <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>
References: <CAP7h-xY_YwX3jDNxpNi3LgH6f47EDAU1zeeUBjXzK4L2b2WN2Q@mail.gmail.com>
 <CAPkN8xLx9zZXn6P0X+PpJ8va5fK5coBcD3==iJ3esVkK+e4UyQ@mail.gmail.com>
 <CAP7h-xZihNbWvcFtH-GyNJNS2patEp-FSOZ3c5ChZxhKDazoSQ@mail.gmail.com>
Message-ID: <CAPkN8xLTiaz7V_-TFXJVnJOp9SUbHAbq2kxk50+8MkbK4PD2BQ@mail.gmail.com>

On Fri, Mar 27, 2015 at 6:21 PM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
>
>> Where have you been when PEP 3108 was discussed?  I have not seen any
>>> other list of Python modules that needed a redesign, so I cannot tell
>>> what's on your top ten list.
>>>
>>
>> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html
>>
>
> Interesting.  I did not see this back in the day.  The top entry (urllib
> and friends) makes sense and there were heavily redesigned in Python 3.  I
> am surprised that distutils redesign got less support than logging and
> datetime.  I suspect that this may have to do with logging and datetime
> APIs not being PEP8 compliant.  Popular votes tend to exaggerate the
> importance of trivial things such as the spelling of class and method names
> and brush off more subtle, but important design flaws.
>

I don't think PEP8 matters at all. Python is not PHP - it has namespaces
and consistency between modules never was so important issue for me at
least, and I am pretty picky. But it is only a guess. I let data change my
mind if there will be some.

My theory is that these modules are just full of warts for specific but
very common scenarios. Here is something that can be used as an example
that it is not about PEP8
https://code.google.com/p/rainforce/wiki/WartsOfPython#measure_time And it
takes a lot of energy to collect something like that for the reference.
Usually you're just left a feeling that something suxx and this feeling is
reinforced every time you hit the wart again and again. It's like AI
learning.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150327/72199633/attachment.html>

From jsbfox at gmail.com  Sun Mar 29 19:04:12 2015
From: jsbfox at gmail.com (Dmitry Kazakov)
Date: Sun, 29 Mar 2015 18:04:12 +0100
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
Message-ID: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>

I propose adding a context manager to contextlib module, which would
restore the stream position of its argument at exit:

    with contextlib.closing(file_like_object) as file:
        file.read(1)
        old = file.tell()
        with contextlib.keep_stream_position(file) as f:
            assert file is f
            assert file_like_object is f
            # Do something with the file(-like) object e.g. feed it
            # to `PIL.Image.open` (will consume the whole stream)

        assert file.tell() == old

This CM would allow us to reuse streams efficiently, without requiring
to call seek() explicitly. I will open a new issue and submit a patch
in case of favorable responses.

From ron3200 at gmail.com  Sun Mar 29 19:21:00 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Sun, 29 Mar 2015 13:21:00 -0400
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info>
Message-ID: <mf9c9t$b87$1@ger.gmane.org>



On 03/28/2015 09:51 PM, Steven D'Aprano wrote:
> But we might be able to rescue this proposal by dropping the requirement
> that the compiler knows when to pass the syntax tree and when to
> evaluate it. Suppose instead we had a lightweight syntax for generating
> the AST plus grabbing the current context:
>
>      x = 23
>      spam(x + 1, !(x+1))  #  macro syntax !( ... )
>
>
> Now the programmer is responsible for deciding when to use an AST and
> when to evaluate it, not the compiler, and "macros" become regular
> functions which just happen to expect an AST as their argument.

Something related to this that I've wanted to experiment with, but is hard 
to do in python to be able to split a function signature and body, and be 
able to use them independently. (But in a well defined way.)


A signature object could have a default body that returns the closure.

And a body (or code) could have a default signature that *takes* a namespace.


Then a function becomes ...

      code(sig(...))    <--->   function(...)



The separate parts could be created with a decorator.

     @signature
     def sig_x(x): pass

     @code
     def inc_x(): x += 1

     @code
     def dec_x(): x -= 1


In most cases it's best to think of applying code bodies to names spaces.

     names = sig_x(0)
     inc_x(names)
     dec_x(names)

That is nicer than continuations as each code block is a well defined unit 
that executes to completion and doesn't require suspending the frame.

(Yes, it can be done with dictionaries, but that wouldn't give the macro 
like functionality (see below) this would.  And there may be other benifits 
to having it at a lower more efficient level.)


To allow macro like ability a code block needs to be executable in the 
current scope.  That can be done just by doing...

     code(locals())      #  Dependable?


And sugar to do that could be...

     if x < 10:
        ^^ inc_x   #just an example syntax.
     else:
        ^^ dec_x   # Note the ^^ looks like the M in Macro. ;-)


Possibly the decorators could be used with lambda directly to get inline 
functionality.

     code(lambda : x + 1)


And a bit of sugar to shorten the common uses if needed.

     spam(x + 1, code(lambda : x + 1))

     spam(x + 1, ^^: x + 1)


Cheers,
    Ron







































From skip.montanaro at gmail.com  Sun Mar 29 23:00:28 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sun, 29 Mar 2015 16:00:28 -0500
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
Message-ID: <CANc-5Uw0CDLG9sbGdtDyJ36GJz8g6qiZ26TKYqtfMPipd9c0HA@mail.gmail.com>

Interesting idea. It should raise an exception as early as possible if the
stream doesn't support seeking...

Skip
On Mar 29, 2015 12:04 PM, "Dmitry Kazakov" <jsbfox at gmail.com> wrote:

> I propose adding a context manager to contextlib module, which would
> restore the stream position of its argument at exit:
>
>     with contextlib.closing(file_like_object) as file:
>         file.read(1)
>         old = file.tell()
>         with contextlib.keep_stream_position(file) as f:
>             assert file is f
>             assert file_like_object is f
>             # Do something with the file(-like) object e.g. feed it
>             # to `PIL.Image.open` (will consume the whole stream)
>
>         assert file.tell() == old
>
> This CM would allow us to reuse streams efficiently, without requiring
> to call seek() explicitly. I will open a new issue and submit a patch
> in case of favorable responses.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150329/5ff5c3d0/attachment.html>

From skip.montanaro at gmail.com  Sun Mar 29 23:01:25 2015
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Sun, 29 Mar 2015 16:01:25 -0500
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
Message-ID: <CANc-5UzkkDmvGJWf2Od_0Qc5Q1zP-Fnq=ySJb5Ae0rzYYHBnRw@mail.gmail.com>

Ugh. Apologies for the top post. Stupid default in Gmail.

S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150329/9050866b/attachment-0001.html>

From toddrjen at gmail.com  Sun Mar 29 23:15:47 2015
From: toddrjen at gmail.com (Todd)
Date: Sun, 29 Mar 2015 23:15:47 +0200
Subject: [Python-ideas] Geo coordinates conversion in stdlib
In-Reply-To: <CAPkN8xLpL0fmyU1OZCVA-c78NxtK4tgTx4rA2Ohs=mUg541Q2Q@mail.gmail.com>
References: <CAPkN8xLdkmTWk_VC88wXrhN0LjS-Aw2Q-jjxL0Ay5F5dM73z-Q@mail.gmail.com>
 <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol>
 <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com>
 <CAPkN8xLQSJe_4hs8AKQ6RcZD0eCk0L8zyN=brzCyvxtG9zD+bA@mail.gmail.com>
 <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com>
 <CAPkN8xLpL0fmyU1OZCVA-c78NxtK4tgTx4rA2Ohs=mUg541Q2Q@mail.gmail.com>
Message-ID: <CAFpSVp+23TacReJjC++xJ0i7b7wXiQdytT5+CHTjBYF9T0FDnw@mail.gmail.com>

On Mar 29, 2015 6:59 PM, "anatoly techtonik" <techtonik at gmail.com> wrote:
>
> On Mon, Mar 23, 2015 at 11:29 AM, Andrew Barnert <abarnert at yahoo.com>
wrote:
>> Most people I know who use Python as a serious data processing language
don't care much what's in the stdlib because they're using numpy and/or
pandas.
>
>
> Yes, and that's sad that the Python still doesn't have the basic building
blocks of Pandas in stdlib. It is the proven de-facto popularity
achievement that is required for packages to enter stdlib, but that still
doesn't happen. Looks like the granularity of package is not sufficient for
designing next stdlib - the need is to descent into the level of structures.

It isn't just granularity, it is pace of development. Once something is in
stdlib,  is pretty much impossible to make substantial changes to it. Even
if all the other criteria were met, the pace of development of projects
like numpy and pandas is far too fast for stdlib.  Numpy devs, at least,
have explicitly stated they do not want numpy in the stdlib for precisely
this reason.

"stdlib is where packages go to die". I don't know who originally said
that, but it is very true.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150329/0e9eef4c/attachment.html>

From ron3200 at gmail.com  Sun Mar 29 23:47:59 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Sun, 29 Mar 2015 17:47:59 -0400
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <mf9c9t$b87$1@ger.gmane.org>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
Message-ID: <mf9ruh$qlp$1@ger.gmane.org>



On 03/29/2015 01:21 PM, Ron Adam wrote:
> To allow macro like ability a code block needs to be executable in the
> current scope.  That can be done just by doing...
>
>      code(locals())      #  Dependable?

Just to clarify, this should have been...

       code_obj(locals())

Since I used "code" as the decorator name, this may have been a bit confusing.

Cheers,
    Ron


From timothy.c.delaney at gmail.com  Mon Mar 30 00:01:52 2015
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 30 Mar 2015 09:01:52 +1100
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
Message-ID: <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>

On 30 March 2015 at 04:04, Dmitry Kazakov <jsbfox at gmail.com> wrote:

> I propose adding a context manager to contextlib module, which would
> restore the stream position of its argument at exit:
>
>     with contextlib.closing(file_like_object) as file:
>         file.read(1)
>         old = file.tell()
>         with contextlib.keep_stream_position(file) as f:
>             assert file is f
>             assert file_like_object is f
>             # Do something with the file(-like) object e.g. feed it
>             # to `PIL.Image.open` (will consume the whole stream)
>
>         assert file.tell() == old
>
> This CM would allow us to reuse streams efficiently, without requiring
> to call seek() explicitly. I will open a new issue and submit a patch
> in case of favorable responses.
>

I like the idea. Might need some bikeshedding on the name (maybe
restore_position?).

To test for seeking, I would suggest having a seek to current position
during construction. Not perfect (e.g. there may be streams that can only
seek forwards, not backwards) but I think it's the best non-destructive
option.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/9c78af4c/attachment.html>

From jheiv at jheiv.com  Sat Mar 28 23:27:49 2015
From: jheiv at jheiv.com (jedwards)
Date: Sat, 28 Mar 2015 15:27:49 -0700 (PDT)
Subject: [Python-ideas] "Loose" descriptors?
Message-ID: <f904e64a-0c51-4e07-8141-c2e95cbd359d@googlegroups.com>

(I apologize in advance if this was covered previously or elsewhere; a 
fairly thorough search didn't yield anything.)

Currently, we can implement a descriptor as follows:

class VerboseDescriptor(object):
    def __init__(self, init):
        self.val = init

    def __get__(self, obj, typ=None):
        print("Get is %s" % (self.val))
        return self.val

    def __set__(self, obj, val):
        print("Set to %s" % (val))
        self.val = val

class Holder(object):
    val = VerboseDescriptor(2)


foo = Holder()
_ = foo.val
foo.val = 4

Which of course works as expected, displaying:

> Get is 2
> Set to 4


But we can't achieve the same indirection without a "Holder"-type class.

That is, we can't do the following:

class VerboseDescriptor(object):
    def __init__(self, init):
        self.val = init

    def __get__(self, obj, typ=None):
        print("Get is %s" % (self.val))
        return self.val

    def __set__(self, obj, val):
        print("Set to %s" % (val))
        self.val = val


val = VerboseDescriptor(2)
_ = val
val = 4

Or rather, we can, but what might be expected (the same functionality / 
output), is not what occurs.

My understanding of the resolution of the first example is that python 
looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it 
finds it, identifies that `Holder.val` implements `__get__`, and handles 
the indirection accordingly.

So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`.

But my question is would it make sense to be able to do this sort of 
indirection for "loose" objects also, not just attributes of classes or 
instances?  In other words, could descriptors be extended so that they 
perform the same sort of indirection in the second example as the first?

I envision (but with no real knowledge of the implications of this) that 
just as python identifies that `val` in `Holder.__dict__` implements 
`__get__`, python could similarly identify that `val` in `globals()` (or 
`sys.modules[__name__].__dict__`, or wherever is more appropriate) 
implements `__get__` and handle the indirection.

Of course, a syntax issue would arise if this were possible -- notably the 
class definition of `Holder` would break.  So some syntax adjustment would 
have to be made.  But is this reasonable / of interest / attainable?

Thanks for your time and consideration.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150328/c57466dc/attachment-0001.html>

From jheiv at jheiv.com  Mon Mar 30 01:19:10 2015
From: jheiv at jheiv.com (James Edwards)
Date: Sun, 29 Mar 2015 19:19:10 -0400
Subject: [Python-ideas] "Loose" descriptors?
In-Reply-To: <f904e64a-0c51-4e07-8141-c2e95cbd359d@googlegroups.com>
References: <f904e64a-0c51-4e07-8141-c2e95cbd359d@googlegroups.com>
Message-ID: <CANbgQ_ErcNhjCUGOH5R2EAi_67fPnYbF1noQVAeF+s-Mz_0ryg@mail.gmail.com>

I sincerely apologize for the duplicate, google-groups decided to push
the email to the list as well, a day later.  I should have stuck with
mailman.

On Sat, Mar 28, 2015 at 6:27 PM, jedwards <jheiv at jheiv.com> wrote:
> (I apologize in advance if this was covered previously or elsewhere; a
> fairly thorough search didn't yield anything.)
>
> Currently, we can implement a descriptor as follows:
>
> class VerboseDescriptor(object):
>     def __init__(self, init):
>         self.val = init
>
>     def __get__(self, obj, typ=None):
>         print("Get is %s" % (self.val))
>         return self.val
>
>     def __set__(self, obj, val):
>         print("Set to %s" % (val))
>         self.val = val
>
> class Holder(object):
>     val = VerboseDescriptor(2)
>
>
> foo = Holder()
> _ = foo.val
> foo.val = 4
>
> Which of course works as expected, displaying:
>>
>> Get is 2
>> Set to 4
>
>
> But we can't achieve the same indirection without a "Holder"-type class.
>
> That is, we can't do the following:
>
> class VerboseDescriptor(object):
>     def __init__(self, init):
>         self.val = init
>
>     def __get__(self, obj, typ=None):
>         print("Get is %s" % (self.val))
>         return self.val
>
>     def __set__(self, obj, val):
>         print("Set to %s" % (val))
>         self.val = val
>
>
> val = VerboseDescriptor(2)
> _ = val
> val = 4
>
> Or rather, we can, but what might be expected (the same functionality /
> output), is not what occurs.
>
> My understanding of the resolution of the first example is that python looks
> for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it,
> identifies that `Holder.val` implements `__get__`, and handles the
> indirection accordingly.
>
> So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`.
>
> But my question is would it make sense to be able to do this sort of
> indirection for "loose" objects also, not just attributes of classes or
> instances?  In other words, could descriptors be extended so that they
> perform the same sort of indirection in the second example as the first?
>
> I envision (but with no real knowledge of the implications of this) that
> just as python identifies that `val` in `Holder.__dict__` implements
> `__get__`, python could similarly identify that `val` in `globals()` (or
> `sys.modules[__name__].__dict__`, or wherever is more appropriate)
> implements `__get__` and handle the indirection.
>
> Of course, a syntax issue would arise if this were possible -- notably the
> class definition of `Holder` would break.  So some syntax adjustment would
> have to be made.  But is this reasonable / of interest / attainable?
>
> Thanks for your time and consideration.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From njs at pobox.com  Mon Mar 30 01:23:17 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 29 Mar 2015 16:23:17 -0700
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
 <55173073.9070600@canterbury.ac.nz>
 <20150329021707.GK25453@ando.pearwood.info>
 <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com>
Message-ID: <CAPJVwBmGx1vCD__pM4z0Uz1ANkA0f3En8-smxDKq1LDkOQsSuw@mail.gmail.com>

On Sat, Mar 28, 2015 at 10:36 PM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Mar 28, 2015, at 19:17, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>>> On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote:
>>> James Edwards wrote:
>>>> I envision (but with no real knowledge of the implications of this)
>>>> that just as python identifies that `val` in `Holder.__dict__`
>>>> implements `__get__`, python could similarly identify that `val` in
>>>> `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more
>>>> appropriate) implements `__get__` and handle the indirection.
>>>
>>> This would slow down *all* lookups of global names in
>>> order to support a rarely-used feature. Since global
>>> name lookup is critical performance-wise, this is
>>> likely to meet a lot of resistance.
>>>
>>> Similar arguments have been made concerning the
>>> support of descriptors in module namespaces, which
>>> is a closely related idea.
>>
>> I don't think it's a closely related idea, I think it is *exactly* the
>> same idea. If I've understood him correctly, James wants to use
>> descriptors in global variables, i.e. module namespaces, not just class
>> attributes.
>
> Yeah, that's how I interpreted it too.
>
> The obvious way to do this is to provide some way for a module to define not only an instance of the module type, but a subclass and an instance of that subclass. The subclass then provides a natural place for the descriptors to live (and a natural answer to what gets passed for inst and typ).
>
> There was a proposal for something very much like that some time late last year, IIRC, which might be worth digging up.

It got implemented. In 3.5, you can do:

# mymodule.py:
class MyModuleType(types.ModuleType):
    @property
    def someattr(self):
        print("Fetching someattr!")
        return 1

sys.modules[__name__].__class__ = MyModuleType

and now accessing mymodule.someattr will print a message.

This is fast -- it doesn't add any overhead for regular module
attribute lookups. OTOH it only works for module attribute lookups,
not for direct accesses to the global dict. So code executing inside
the module namespace can't write a bare "someattr" to refer to the
property; only external users of the module see "someattr".

There's some code to make this available on all versions of CPython here:
  https://github.com/njsmith/metamodule/
which I'll clean up slightly and put up on pypi Any Day Now...

-n

-- 
Nathaniel J. Smith -- http://vorpus.org

From abarnert at yahoo.com  Mon Mar 30 02:36:00 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 29 Mar 2015 17:36:00 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <mf9c9t$b87$1@ger.gmane.org>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
Message-ID: <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com>

> On Mar 29, 2015, at 10:21, Ron Adam <ron3200 at gmail.com> wrote:
> 
> 
> 
>> On 03/28/2015 09:51 PM, Steven D'Aprano wrote:
>> But we might be able to rescue this proposal by dropping the requirement
>> that the compiler knows when to pass the syntax tree and when to
>> evaluate it. Suppose instead we had a lightweight syntax for generating
>> the AST plus grabbing the current context:
>> 
>>     x = 23
>>     spam(x + 1, !(x+1))  #  macro syntax !( ... )
>> 
>> 
>> Now the programmer is responsible for deciding when to use an AST and
>> when to evaluate it, not the compiler, and "macros" become regular
>> functions which just happen to expect an AST as their argument.
> 
> Something related to this that I've wanted to experiment with, but is hard to do in python to be able to split a function signature and body, and be able to use them independently. (But in a well defined way.)

Almost everything you're asking for is already there.

A function object contains, among other things, a sequence of closure cells, a local and global environment, default parameter values, and a code object.

A code object contains, among other things, parameter names, a count of locals, and a bytecode string.

You can see the attributes of these objects at runtime, and the inspect module docs describe what they mean. You can also construct these objects at runtime by using their constructors (you have to use types.FunctionType and types.CodeType; the built-in help can show you the parameters).

You can also compile source code (or an AST) to a code object with the compile function.

You can call a code object with the exec function, which takes a namespace (or, optionally, separate local and global namespaces--and, in a slightly hacky way, you can also override the builtin namespace).

There are also Signature objects in the inspect module, but they're not "live" usable objects, they're nicely-organized-for-human-use representations of the signature of a function. So practically you'd use a dummy function object or just a dict or something, and create a new function from its attributes/members and the new code.

So, except for minor spelling differences, that's exactly what you're asking for, and it's already there.

But if I've guessed right about _why_ you want this, it doesn't do what you'd want it to, and I don't think there's any way it could.

Bytecode accesses locals (including arguments), constants, and closure cells by index from the frame object, not by name from a locals dict (although the frame has one of those as well, in case you want to debug or introspect, or call locals()). So, when you call a function, Python sets up the frame object, matching positional and keyword arguments (and default values in the function object) up to parameters and building up the sequence of locals. The frame is also essential for returning and uncaught exceptions (it has a back pointer to the calling frame).

The big thing you can't do directly is to create new closure cells programmatically from Python. The compiler has to know which of your locals will be used as closure variables by any embedded functions; it then stores these specially within your code object, so the MAKE_CLOSURE bytecode that creates a function object out of each embedded function can create matching closure cells to store in the embedded function object. This is the part that you need to add into what Python already has, and I'm not sure there's a clean way to do it.

But you really should learn how all the existing stuff works (the inspect docs, the dis module, and the help for the constructors in the types module are actually sufficient for this, without having to read the C source code, in 3.4 and later) and find out for yourself, because if I'm wrong, you may come up with something cool. (Plus, it's fun and useful to learn about.)

That only gets you through the first half of your message--enough to make inc_x work as a local function (well, almost--without a "nonlocal x" statement it's going to compile x as a local variable rather than a closure variable, and however you execute it, you're just going to get UnboundLocalError).

What about the second part, where you execute code in an existing frame?

That's even trickier. A frame already has its complete lists of locals, cells, and constants at construction time. If your code objects never used any new locals or constants, and never touched any nonlocal variables that weren't already touched by the calling function, all you need is some kind of "relocation" step that remaps the indices compiled into the bytecode into the indices in the calling frame (there's enough info in the code objects to do the mapping; for actually creating the relocated bytecode from the original you'll want something like the byteplay module, which unfortunately doesn't exist for 3.4--although I have an incomplete port that might be good enough to play with if you're interested).

You can almost get away with the "no new locals or nonlocal cells" part, but "no new constants" is pretty restrictive. For example, if you compile inc_x into a fragment that can be executed inline, the number 1 is going to be constant #0 in its code object. And now, you try to "relocate" it to run in a frame with a different code object, and (unless that different code object happened to refer to 1 as a constant as well) there's nothing to match it up to.

And again, I don't see a way around this without an even more drastic rearchitecting of how Python frames work--but again, I think it's worth looking for yourself in hopes that I'm wrong.

There's another problem: every function body compiler to code ends with a return bytecode. If you didn't write one explicitly, you get the equivalent of "return None". That means that, unless you solve that in some way, executing a fragment inline is always going to return from the calling function. And what do you want to do about explicit return? Or break and continue?

Of the two approaches, I think the first one seems cleaner. If you can make a closure cell out of x and then wrap inc_x's code in a normal closure that references it, that still feels like Python. (And having to make "nonlocal x" explicit seems like a good thing, not a limitation.) Fixing up fragments to run in a different frame, and modifying frames at runtime to allow them to be fixed up, seems a lot hackier. And the whole return issue is pretty serious, too.

One last possibility to consider is something between the two: a different kind of object, defined differently, like a proc in Ruby (which is defined with a block rather than a def) might solve some of the problems with either approach. And stealing from Ruby again, procs have their own "mini-frames"; there's a two-level stack where every function stack frame has a proc stack frame, which allows a solution to the return-value problem that wouldn't be available with either closures or fragments. (However, note that the return-value problem is much more serious in Ruby, where everything is supposed to be an expression, with a value; in Python you can just say "fragment calls are statements, so they don't have values" if that's what you want.)

One last note inline:

> A signature object could have a default body that returns the closure.
> 
> And a body (or code) could have a default signature that *takes* a namespace.
> 
> 
> Then a function becomes ...
> 
>     code(sig(...))    <--->   function(...)
> 
> 
> 
> The separate parts could be created with a decorator.
> 
>    @signature
>    def sig_x(x): pass
> 
>    @code
>    def inc_x(): x += 1
> 
>    @code
>    def dec_x(): x -= 1
> 
> 
> In most cases it's best to think of applying code bodies to names spaces.
> 
>    names = sig_x(0)
>    inc_x(names)
>    dec_x(names)
> 
> That is nicer than continuations as each code block is a well defined unit that executes to completion and doesn't require suspending the frame.
> 
> (Yes, it can be done with dictionaries, but that wouldn't give the macro like functionality (see below) this would.  And there may be other benifits to having it at a lower more efficient level.)
> 
> 
> To allow macro like ability a code block needs to be executable in the current scope.  That can be done just by doing...
> 
>    code(locals())      #  Dependable?
> 
> 
> And sugar to do that could be...
> 
>    if x < 10:
>       ^^ inc_x   #just an example syntax.
>    else:
>       ^^ dec_x   # Note the ^^ looks like the M in Macro. ;-)
> 
> 
> Possibly the decorators could be used with lambda directly to get inline functionality.
> 
>    code(lambda : x + 1)

This is a very different thing from what you were doing above. A function that modifies a closure cell's value, like inc_x, can't be written as a lambda (because assignments are statements). And this lambda is completely pointless if you're going to use it in a context where you ignore its return value (like the way you used inc_x above). So, I'm not sure what you're trying to do here, but I think you may have another problem to solve on top of the ones I already mentioned.

> And a bit of sugar to shorten the common uses if needed.
> 
>    spam(x + 1, code(lambda : x + 1))
> 
>    spam(x + 1, ^^: x + 1)
> 
> 
> Cheers,
>   Ron
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Mon Mar 30 02:40:36 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 29 Mar 2015 17:40:36 -0700
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
Message-ID: <9D3E2056-73CD-4FA6-9D20-53C846BCD71F@yahoo.com>

On Mar 29, 2015, at 10:04, Dmitry Kazakov <jsbfox at gmail.com> wrote:
> 
> I propose adding a context manager to contextlib module, which would
> restore the stream position of its argument at exit:
> 
>    with contextlib.closing(file_like_object) as file:
>        file.read(1)
>        old = file.tell()
>        with contextlib.keep_stream_position(file) as f:
>            assert file is f
>            assert file_like_object is f
>            # Do something with the file(-like) object e.g. feed it
>            # to `PIL.Image.open` (will consume the whole stream)
> 
>        assert file.tell() == old
> 
> This CM would allow us to reuse streams efficiently, without requiring
> to call seek() explicitly. I will open a new issue and submit a patch
> in case of favorable responses.

I'm pretty sure an implementation of this is already on PyPI somewhere, although I can't remember exactly where, and it's possible that I just built it myself when needed rather than finding it... 

Anyway, if I'm right, that would be helpful for both bikeshedding the name, and spotting any potential problems that had to be worked around (the need for f.seek(f.tell()) to catch non-seekable files early is the only non-obvious one I can think of, but if someone has been using this for years in a wide range of projects, they may have found one that none of us will immediately think of).


> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From guido at python.org  Mon Mar 30 02:53:43 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Mar 2015 17:53:43 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
Message-ID: <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>

On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Responding to comments off list:
>
> I'm not referring to C-style preprocessor macros, I'm referring to macros
> historically found in functional languages and commonly found in many
> user-targeted languages built in the last few years.
>

Do you have examples and references? IIRC there's something named macros in
Scheme but Scheme, unlike Python, completely unifies code and data, and
there is a standard in-memory representation for code.


> The goal is to create things that look like functions but have access to
> the expression that was passed in.
>
> Some examples where this is useful:
>
>     plot(year, miles / gallon)  # Plot with labels determined by
> input-expressions, e.g. miles/gallon
>
>     assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
> assertRaises function, not before
>
>     run_concurrently(f(x), f(y), f(z))  # Run f three times in three
> threads controlled by run_concurrently
>
> Generally one constructs something that looks like a function but, rather
> than receiving a pre-evaluated input, receives a syntax tree along with the
> associated context.  This allows that function-like-thing to manipulate the
> expression and to control the context in which the evaluation occurs.
>

None of the examples need the syntax tree though. The first wants the
string, the last probably just want a way to turn an argument into a lambda.


> There are lots of arguments against this, mostly focused around potential
> misuse.  I'm looking for history of such arguments and for a general "Yes,
> this is theoretically possible" or "Not a chance in hell" from the
> community.  Both are fine.
>

I don't think this is a mainline need in Python, so it's probably both. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150329/ee412506/attachment.html>

From guido at python.org  Mon Mar 30 03:27:39 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 29 Mar 2015 18:27:39 -0700
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
Message-ID: <CAP7+vJLe2qXYOyCtHJ4kzOLuSU4xcyHd1Mb0-WG2rQpDxvFvqw@mail.gmail.com>

On Sun, Mar 29, 2015 at 10:04 AM, Dmitry Kazakov <jsbfox at gmail.com> wrote:

> I propose adding a context manager to contextlib module, which would
> restore the stream position of its argument at exit:
>
>     with contextlib.closing(file_like_object) as file:
>         file.read(1)
>         old = file.tell()
>         with contextlib.keep_stream_position(file) as f:
>             assert file is f
>             assert file_like_object is f
>             # Do something with the file(-like) object e.g. feed it
>             # to `PIL.Image.open` (will consume the whole stream)
>
>         assert file.tell() == old
>
> This CM would allow us to reuse streams efficiently, without requiring
> to call seek() explicitly. I will open a new issue and submit a patch
> in case of favorable responses.
>

I object to the use of the word "efficiently". You save a line of code and
a try/finally block (though the latter probably rarely matters) but the
cost of seeking around in the file (negligible though it is) is not reduced
by using a CM. It also sounds like this would be a really simple exercise,
hardly worth adding to the stdlib.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150329/c4b36556/attachment.html>

From ron3200 at gmail.com  Mon Mar 30 06:12:03 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 30 Mar 2015 00:12:03 -0400
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com>
Message-ID: <mfaiek$qvt$1@ger.gmane.org>



On 03/29/2015 08:36 PM, Andrew Barnert wrote:
>>> Something related to this that I've wanted to experiment with, but
>>> is hard to do in python to be able to split a function signature and
>>> body, and be able to use them independently. (But in a well defined
>>> way.)

> Almost everything you're asking for is already there.

Yes, I have looked into most of what you mention here.

> A function object contains, among other things, a sequence of closure
> cells, a local and global environment, default parameter values, and a
> code object.
>
> A code object contains, among other things, parameter names, a count of
> locals, and a bytecode string.
>
> You can see the attributes of these objects at runtime, and the inspect
> module docs describe what they mean. You can also construct these
> objects at runtime by using their constructors (you have to use
> types.FunctionType and types.CodeType; the built-in help can show you
> the parameters).
>
> You can also compile source code (or an AST) to a code object with the
> compile function.
>
> You can call a code object with the exec function, which takes a
> namespace (or, optionally, separate local and global namespaces--and, in
> a slightly hacky way, you can also override the builtin namespace).
>
> There are also Signature objects in the inspect module, but they're not
> "live" usable objects, they're nicely-organized-for-human-use
> representations of the signature of a function. So practically you'd use
> a dummy function object or just a dict or something, and create a new
> function from its attributes/members and the new code.
>
> So, except for minor spelling differences, that's exactly what you're
> asking for, and it's already there.

I think it's more than minor spelling differences.  :-)

I've played around with de-constructing functions and using the 
constructors to put them back together again, enough to know it's actually 
quite hard to get everything right with other than the original parts.

The exec function may be a good start to experimenting with this.  I havn't 
used it with code objects enough to be familiar with what limits that has. 
  It may not be that difficult to copy the C source for exec and create a 
new function more specific to this idea.  (as a test)

Even if that's slow, it may be good enough as a start.

> But if I've guessed right about_why_  you want this, it doesn't do what
> you'd want it to, and I don't think there's any way it could.
>
> Bytecode accesses locals (including arguments), constants, and closure
> cells by index from the frame object, not by name from a locals dict
> (although the frame has one of those as well, in case you want to debug
> or introspect, or call locals()). So, when you call a function, Python
> sets up the frame object, matching positional and keyword arguments (and
> default values in the function object) up to parameters and building up
> the sequence of locals. The frame is also essential for returning and
> uncaught exceptions (it has a back pointer to the calling frame).

This isn't a problem if the callable_code object creates a new frame.

It is an issue when running a code block in the current frame.  But I think 
there may be a way to get around that.


> The big thing you can't do directly is to create new closure cells
> programmatically from Python. The compiler has to know which of your
> locals will be used as closure variables by any embedded functions; it
> then stores these specially within your code object, so the MAKE_CLOSURE
> bytecode that creates a function object out of each embedded function
> can create matching closure cells to store in the embedded function
> object. This is the part that you need to add into what Python already
> has, and I'm not sure there's a clean way to do it.


> But you really should learn how all the existing stuff works (the
> inspect docs, the dis module, and the help for the constructors in the
> types module are actually sufficient for this, without having to read
> the C source code, in 3.4 and later) and find out for yourself, because
> if I'm wrong, you may come up with something cool. (Plus, it's fun and
> useful to learn about.)

I'm familiar with most of how python works, and even hacked a bit on 
ceval.c. (for fun)  I haven't played much with the AST side of things, but 
I do know generally how python is put together and works.


> That only gets you through the first half of your message--enough to
> make inc_x work as a local function (well, almost--without a "nonlocal
> x" statement it's going to compile x as a local variable rather than a
> closure variable, and however you execute it, you're just going to get
> UnboundLocalError).
>
> What about the second part, where you execute code in an existing
> frame?
 >
> That's even trickier.

Yes, I definitely agree.  I think one of the tests of a good idea is that 
it makes something that is normally hard (or tricky), simple and easy.  But 
actually doing that may be quite hard.  (or even not possible.)

> A frame already has its complete lists of locals,
> cells, and constants at construction time. If your code objects never
> used any new locals or constants, and never touched any nonlocal
> variables that weren't already touched by the calling function, all you
> need is some kind of "relocation" step that remaps the indices compiled
> into the bytecode into the indices in the calling frame (there's enough
> info in the code objects to do the mapping; for actually creating the
> relocated bytecode from the original you'll want something like the
> byteplay module, which unfortunately doesn't exist for 3.4--although I
> have an incomplete port that might be good enough to play with if you're
> interested).

> You can almost get away with the "no new locals or nonlocal cells" part,
> but "no new constants" is pretty restrictive. For example, if you
> compile inc_x into a fragment that can be executed inline, the number 1
> is going to be constant #0 in its code object. And now, you try to
> "relocate" it to run in a frame with a different code object, and
> (unless that different code object happened to refer to 1 as a constant
> as well) there's nothing to match it up to.
>
> And again, I don't see a way around this without an even more drastic
> rearchitecting of how Python frames work--but again, I think it's worth
> looking for yourself in hopes that I'm wrong.

Think of these things as non_local_blocks.  The difference is they would 
use dynamic scope instead of static scope.  Or to put it another way, they 
would inherit the scope they are executed in.

> There's another problem: every function body compiler to code ends with
> a return bytecode. If you didn't write one explicitly, you get the
> equivalent of "return None". That means that, unless you solve that in
> some way, executing a fragment inline is always going to return from the
> calling function. And what do you want to do about explicit return? Or
> break and continue?

As a non_local_block, a return could be a needed requirement.  It would 
return the value to the current location in the current frame.

Break and continue are harder. Probably give an error the same as what 
would happen if they are used outside a loop.  A break or return would need 
to be local to the loop.  So you can't have a break in a non_local_block 
unless the block also has the loop in it.  That keeps the associated parts 
local to each other.

> Of the two approaches, I think the first one seems cleaner. If you can
> make a closure cell out of x and then wrap inc_x's code in a normal
> closure that references it, that still feels like Python. (And having to
> make "nonlocal x" explicit seems like a good thing, not a limitation.)

Agree... I picture the first approach as a needed step to get to the second 
part.

> Fixing up fragments to run in a different frame, and modifying frames at
> runtime to allow them to be fixed up, seems a lot hackier. And the whole
> return issue is pretty serious, too.
>
> One last possibility to consider is something between the two: a
> different kind of object, defined differently, like a proc in Ruby
> (which is defined with a block rather than a def) might solve some of
> the problems with either approach. And stealing from Ruby again, procs
> have their own "mini-frames"; there's a two-level stack where every
> function stack frame has a proc stack frame, which allows a solution to
> the return-value problem that wouldn't be available with either closures
> or fragments.

This may be closer to how I am thinking it would work.  :-)


> (However, note that the return-value problem is much more
> serious in Ruby, where everything is supposed to be an expression, with
> a value; in Python you can just say "fragment calls are statements, so
> they don't have values" if that's what you want.)

It seems to me they can be either.  Python ignores None when it's returned 
by a function and not assigned to anything.  And if a value is returned, 
than it's returned to the current position in the current frame.  The 
return in this case is a non-local-block return.  So I think it wouldn't be 
an issue.

>
> One last note inline:
>
>>> A signature object could have a default body that returns the
>>> closure.
>>>
>>> And a body (or code) could have a default signature that*takes*  a
>>> namespace.
>>>
>>>
>>> Then a function becomes ...
>>>
>>> code(sig(...))    <--->   function(...)
>>>
>>>
>>>
>>> The separate parts could be created with a decorator.
>>>
>>> @signature def sig_x(x): pass
>>>
>>> @code def inc_x(): x += 1
>>>
>>> @code def dec_x(): x -= 1
>>>
>>>
>>> In most cases it's best to think of applying code bodies to names
>>> spaces.
>>>
>>> names = sig_x(0) inc_x(names) dec_x(names)
>>>
>>> That is nicer than continuations as each code block is a well
>>> defined unit that executes to completion and doesn't require
>>> suspending the frame.
>>>
>>> (Yes, it can be done with dictionaries, but that wouldn't give the
>>> macro like functionality (see below) this would.  And there may be
>>> other benifits to having it at a lower more efficient level.)
>>>
>>>
>>> To allow macro like ability a code block needs to be executable in
>>> the current scope.  That can be done just by doing...
>>>
>>> code(locals())      #  Dependable?
>>>
>>>
>>> And sugar to do that could be...
>>>
>>> if x < 10: ^^ inc_x   #just an example syntax. else: ^^ dec_x   #
>>> Note the ^^ looks like the M in Macro.;-)
>>>
>>>
>>> Possibly the decorators could be used with lambda directly to get
>>> inline functionality.
>>>
>>> code(lambda : x + 1)

> This is a very different thing from what you were doing above. A
> function that modifies a closure cell's value, like inc_x, can't be
> written as a lambda (because assignments are statements). And this
> lambda is completely pointless if you're going to use it in a context
> where you ignore its return value (like the way you used inc_x above).
> So, I'm not sure what you're trying to do here, but I think you may have
> another problem to solve on top of the ones I already mentioned.

It was an incomplete example.  It should have been...

     add_1_to_x = code(lambda: x + 1)

and then later you could use it in the same way as above.

     x = ^^ add_1_to_x

This is just an example to show how the first option above connects to the 
examples below.  with "^^: x + 1" being equivalent to "code(lambda: x + 1)".

Which would also be equivalent to ...

    @code
    def add_1_to_x():
        return x + 1

    x = ^^ add_1_to_x



>>> And a bit of sugar to shorten the common uses if needed.
>>>
>>> spam(x + 1, code(lambda : x + 1))
>>>
>>> spam(x + 1, ^^: x + 1)



From jheiv at jheiv.com  Mon Mar 30 07:21:30 2015
From: jheiv at jheiv.com (James Edwards)
Date: Mon, 30 Mar 2015 01:21:30 -0400
Subject: [Python-ideas] "Loose" descriptors
In-Reply-To: <20150329023405.GA7521@k3>
References: <CANbgQ_GwxgRaX9hJ0fd630zOxCKPD0pV4CevrOJ1NF_qa3mZvA@mail.gmail.com>
 <20150329023405.GA7521@k3>
Message-ID: <CANbgQ_F0=T--H82ew88+ywUo9jhzFc3PdPJYE++C4aXVR19Tjw@mail.gmail.com>

On Sat, Mar 28, 2015 at 10:34 PM, David Wilson <dw+python-ideas at hmmz.org> wrote:
> FWIW you could probably experiment with this using PyPy's object spaces
> feature without modifying the interpreter:
>
>     http://pypy.readthedocs.org/en/latest/objspace-proxies.html
>

That's David, I'll look into these.  "Proxy" type objects were exactly
the motivation for my suggestion.  I'm working with very deep nested
dictionaries -- not the most efficient data structure, but they're a
very direct representation of what I'm trying to do.

I wanted to create an ItemProxy object, for example:

    class ItemProxy:
        def __init__(self, obj, key):
            self.obj = obj
            self.key = key

        def get(self):      return self.obj[self.key]
        def set(self, val): self.obj[self.key] = val


    d = {
        '0': {'a':'foo', 'b':'bar'},
        '1': {'c':'car', 'd':'baz'},
    }

    p = ItemProxy(d['0'], 'a')

    print(p.get())
    p.set('abc')
    print(p.get())

But instead of writing print(p.get()), and p.set('abc'), I wanted to
just write print(p), and p='abc'.

w.r.t. using eval(), since posting I've dug up this Mar 2011 email to
this list from Raymond Hettinger: "descriptors outside of classes"

    https://mail.python.org/pipermail/python-ideas/2011-March/009657.html

Here he describes a method for simulating the object.__getattribute__
indirection that descriptors provide explicitly by using eval and
passing a locals dictionary.


On Sun, Mar 29, 2015 at 7:23 PM, Nathaniel Smith <njs at pobox.com> wrote:
> It got implemented. In 3.5, you can do:
> # mymodule.py:
> class MyModuleType(types.ModuleType):
>     ...
> sys.modules[__name__].__class__ = MyModuleType
> and now accessing mymodule.someattr will print a message.

I saw this approach also, (I believe, I can't test this in 3.5
currently and fails on 3.4) using slightly different process but with
the same end result (and possible on 2.7, at least):
http://www.dr-josiah.com/2013/12/properties-on-python-modules.html

And while this certainly works, it is restricted to modules, and I'd
imagine breaks traditional static checking.  In order to not break
static checking, you could use the approach as described in the link,
but the process described there is certainly a workaround and not a
great solution.


On Sat, Mar 28, 2015 at 10:17 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> (1) Can modules use a dict subclass for __dict__?
> (2) Can you replace the __dict__ on the fly like this?
> (3) If not, how do you bootstrap the module to have a magic dict instead
>     of an ordinary dict?

If this were possible, it seems like it would provide the flexibility
necessary to implement "global" descriptors, or at least module-level
descriptors, but as far as I can tell, it is not.

Trying to directly replace __dict__ results in AttributeError:
readonly attribute

Even if you try to replace the module reference in sys.modules with a
class that implements a custom __dict__, it doesn't appear to be not
used.  For example:

    # mymod.py
    import sys

    class CustomDict(dict):
        def __getattribute__(self, *args, **kwargs):
            print(["getattribute", args, kwargs])
            return dict.__getattribute__(self, *args, **kwargs)

        def __setattr__(self, *args, **kwargs):
            print(["setattr", args, kwargs])
            return dict.__setattr__(*args, **kwargs)

        def __missing__(self, key):
            print("Missing:", key)
            return None

    class Module: pass

    bar = "bar"     # Some module attribute, that will get carried into module

    module = Module()
    module.__dict__ = CustomDict(globals())

    sys.modules[module.__name__] = module

When inspected, the dict type is "correct":

    print(type(mymod.__dict__))  # <class 'mymod.CustomDict'>

But when module attributes are accessed/set, the expected print
functions aren't shown:

    print(mymod.bar)    # bar (and nothing else)
    mymod.bar = "baz"   # (nothing)
    print(mymod.bar)    # baz (and nothing else)

    # Attempt to access undefined module attribute, hoping to trigger
__missing__
    print(mymod.zzz)    # AttributeError: 'Module' object has no attribute 'zzz'

If it were possible to do this on a per-module basis, I think it would
provide much added flexibility, while restricting the lookup penalty
to modules that chose to implement it.

From abarnert at yahoo.com  Mon Mar 30 09:03:47 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Mar 2015 00:03:47 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <mfaiek$qvt$1@ger.gmane.org>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <mfaiek$qvt$1@ger.gmane.org>
Message-ID: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com>

On Mar 29, 2015, at 21:12, Ron Adam <ron3200 at gmail.com> wrote:
> 
> On 03/29/2015 08:36 PM, Andrew Barnert wrote:
>>>> Something related to this that I've wanted to experiment with, but
>>>> is hard to do in python to be able to split a function signature and
>>>> body, and be able to use them independently. (But in a well defined
>>>> way.)
> 
>> Almost everything you're asking for is already there.
> 
> Yes, I have looked into most of what you mention here.
> 
>> A function object contains, among other things, a sequence of closure
>> cells, a local and global environment, default parameter values, and a
>> code object.
>> 
>> A code object contains, among other things, parameter names, a count of
>> locals, and a bytecode string.
>> 
>> You can see the attributes of these objects at runtime, and the inspect
>> module docs describe what they mean. You can also construct these
>> objects at runtime by using their constructors (you have to use
>> types.FunctionType and types.CodeType; the built-in help can show you
>> the parameters).
>> 
>> You can also compile source code (or an AST) to a code object with the
>> compile function.
>> 
>> You can call a code object with the exec function, which takes a
>> namespace (or, optionally, separate local and global namespaces--and, in
>> a slightly hacky way, you can also override the builtin namespace).
>> 
>> There are also Signature objects in the inspect module, but they're not
>> "live" usable objects, they're nicely-organized-for-human-use
>> representations of the signature of a function. So practically you'd use
>> a dummy function object or just a dict or something, and create a new
>> function from its attributes/members and the new code.
>> 
>> So, except for minor spelling differences, that's exactly what you're
>> asking for, and it's already there.
> 
> I think it's more than minor spelling differences.  :-)

The objects you're asking for already exist, but in some cases with slightly different names. There isn't neat calling syntax for things like cloning a function object with a different code object but the same other attributes is a bit painful, but it's just a few two-line wrapper functions that you only have to write once (I think you can even subclass CodeType and FunctionType to add the syntax; if not, you can wrap and delegate). So, what you're asking for really is already part of Python, except for minor spelling differences. The problem is that what you're asking for doesn't give you what you want, because other things (like "the environment") don't work the way you're assuming they do, or because of consequences you probably haven't thought of (like the return issue).

> I've played around with de-constructing functions and using the constructors to put them back together again, enough to know it's actually quite hard to get everything right with other than the original parts.

It really isn't, once you learn how they work. Getting the values to put into them (assembling bytecode, creating the lnotab, etc.) is a different story, but if your intention is to do that just by compiling function (or fragment) source, it's all done for you.

> The exec function may be a good start to experimenting with this.  I havn't used it with code objects enough to be familiar with what limits that has.  It may not be that difficult to copy the C source for exec and create a new function more specific to this idea.  (as a test)
> 
> Even if that's slow, it may be good enough as a start.

Why would it be slow?

Also, what do you want your function to do differently from exec? What you described is wanting to run a code object in a specified environment; that's what exec does.

I think you're thinking of Python environments as if they were scheme environments, but they aren't. In particular, Python closures don't work in terms of accessing variables from the environment by name; they work in terms of accessing cells from the function object.

>> But if I've guessed right about_why_  you want this, it doesn't do what
>> you'd want it to, and I don't think there's any way it could.
>> 
>> Bytecode accesses locals (including arguments), constants, and closure
>> cells by index from the frame object, not by name from a locals dict
>> (although the frame has one of those as well, in case you want to debug
>> or introspect, or call locals()). So, when you call a function, Python
>> sets up the frame object, matching positional and keyword arguments (and
>> default values in the function object) up to parameters and building up
>> the sequence of locals. The frame is also essential for returning and
>> uncaught exceptions (it has a back pointer to the calling frame).
> 
> This isn't a problem if the callable_code object creates a new frame.

Yes, it is, as I explained before. There's the minor problem that your code object needs to use nonlocal if it wants to reassign to the calling frame's x variable, and the major problem that there is no way to give it a closure that gives it access to that frame's x variable unless, when that calling frame's code was compiled, there was already an embedded function that needed to close over x.

I suppose you could hack up the compiler to generate cells for every local variable instead of just those that are actually needed. Then you could sort of do what you want--instead of exec, you construct a function with the appropriate cells in the closure (by matching up the callee code's freevars with the calling code's cellvars).

> It is an issue when running a code block in the current frame.  But I think there may be a way to get around that.
> 
> 
>> The big thing you can't do directly is to create new closure cells
>> programmatically from Python. The compiler has to know which of your
>> locals will be used as closure variables by any embedded functions; it
>> then stores these specially within your code object, so the MAKE_CLOSURE
>> bytecode that creates a function object out of each embedded function
>> can create matching closure cells to store in the embedded function
>> object. This is the part that you need to add into what Python already
>> has, and I'm not sure there's a clean way to do it.
> 
> 
>> But you really should learn how all the existing stuff works (the
>> inspect docs, the dis module, and the help for the constructors in the
>> types module are actually sufficient for this, without having to read
>> the C source code, in 3.4 and later) and find out for yourself, because
>> if I'm wrong, you may come up with something cool. (Plus, it's fun and
>> useful to learn about.)
> 
> I'm familiar with most of how python works, and even hacked a bit on ceval.c. (for fun)  I haven't played much with the AST side of things, but I do know generally how python is put together and works.
> 
> 
>> That only gets you through the first half of your message--enough to
>> make inc_x work as a local function (well, almost--without a "nonlocal
>> x" statement it's going to compile x as a local variable rather than a
>> closure variable, and however you execute it, you're just going to get
>> UnboundLocalError).
>> 
>> What about the second part, where you execute code in an existing
>> frame?
> >
>> That's even trickier.
> 
> Yes, I definitely agree.  I think one of the tests of a good idea is that it makes something that is normally hard (or tricky), simple and easy.  But actually doing that may be quite hard.  (or even not possible.)
> 
>> A frame already has its complete lists of locals,
>> cells, and constants at construction time. If your code objects never
>> used any new locals or constants, and never touched any nonlocal
>> variables that weren't already touched by the calling function, all you
>> need is some kind of "relocation" step that remaps the indices compiled
>> into the bytecode into the indices in the calling frame (there's enough
>> info in the code objects to do the mapping; for actually creating the
>> relocated bytecode from the original you'll want something like the
>> byteplay module, which unfortunately doesn't exist for 3.4--although I
>> have an incomplete port that might be good enough to play with if you're
>> interested).
> 
>> You can almost get away with the "no new locals or nonlocal cells" part,
>> but "no new constants" is pretty restrictive. For example, if you
>> compile inc_x into a fragment that can be executed inline, the number 1
>> is going to be constant #0 in its code object. And now, you try to
>> "relocate" it to run in a frame with a different code object, and
>> (unless that different code object happened to refer to 1 as a constant
>> as well) there's nothing to match it up to.
>> 
>> And again, I don't see a way around this without an even more drastic
>> rearchitecting of how Python frames work--but again, I think it's worth
>> looking for yourself in hopes that I'm wrong.
> 
> Think of these things as non_local_blocks.  The difference is they would use dynamic scope instead of static scope.  Or to put it another way, they would inherit the scope they are executed in.

That's a bit of a weird name, since it's a block that's executed locally on the current frame, as opposed to the normal non-local way, but... OK.

It makes sense conceptually, but practically it doesn't fit in with Python's execution model.

First, the scope is partly defined at compilation time--that's where the list of constants comes from, and the lists of local and free names. If the frame's code is still the calling function, none of these things are available in the frame. If, on the other hand, it's the non_local_block (what I was falling the "fragment"), then the calling scope's variables aren't available. (And just copying the caller's code's stuff into a new frame only gives you copies of the caller's variables, which doesn't let you do the main thing you wanted to do.) Unless you somehow make them both available (e.g., the Ruby-style two-level call stack I mentioned), I don't see a way around that.

Second, remember that Python bytecode accesses locals by index; unless you do something like the relocation I described above, you have no way to access the calling scope's variables, and that's doable, but not at all trivial, and doesn't seem very Pythonic.

Also, even if you do that, you've still got a problem.  Python's LEGB rule that decides which scope to find a name in is handled mostly at compile time. The compiler decides whether to emit LOAD_FAST or one of the other LOAD_*, and likewise for STORE_*, based on the static lexical scope the name is defined in. That's going to make it very hard to hack in dynamic scope on top of Python bytecode--the "relocation" has to actually involve not just renumbering locals, but reproducing all the work the compiler does to decide what's local/global/etc. and replacing instructions as appropriate.

As an alternative, maybe you don't want code objects at all, but rather AST objects. Then they can be compiled to bytecode in a given scope (with the builtin compile function) and then executed there (with exec). This solves most of the new problems added by the dynamic scoping/non_local_block/fragment idea while still allowing most of the benefits. The downside, of course, is that you're compiling stuff all over the place--but that's how dynamic code works in most Lisps, so...

>> There's another problem: every function body compiler to code ends with
>> a return bytecode. If you didn't write one explicitly, you get the
>> equivalent of "return None". That means that, unless you solve that in
>> some way, executing a fragment inline is always going to return from the
>> calling function. And what do you want to do about explicit return? Or
>> break and continue?
> 
> As a non_local_block, a return could be a needed requirement.  It would return the value to the current location in the current frame.

Do you mean that a call to a non_local_block is an expression, and the value of the expression is the value returned by the block? If so, that makes sense, but then the RETURN_VALUE handler in the interpreter has to be sensitive to the context it's in and do two different things, or the compiler has to be sensitive to what it's compiling--which seems impossible, given that you want to def a regular function and then decorate it after the fact--and issue a different opcode for return (and for the implicit return None). 

> Break and continue are harder. Probably give an error the same as what would happen if they are used outside a loop.  A break or return would need to be local to the loop.  So you can't have a break in a non_local_block unless the block also has the loop in it.  That keeps the associated parts local to each other.

That definitely makes things simpler--and, I think, better. It will piss off people who expect (from Lisp or Ruby) to be able to do flow control across the boundaries of a non_local_block, but too bad for them. :)

>> Of the two approaches, I think the first one seems cleaner. If you can
>> make a closure cell out of x and then wrap inc_x's code in a normal
>> closure that references it, that still feels like Python. (And having to
>> make "nonlocal x" explicit seems like a good thing, not a limitation.)
> 
> Agree... I picture the first approach as a needed step to get to the second part.

I don't think it really is; most of what you need to solve for the first becomes irrelevant for the second. For example, figuring out a way to dynamically generate closure cells to share variables across frame's is useless when you switch to running with the parent's variables as locals in the same frame.

>> Fixing up fragments to run in a different frame, and modifying frames at
>> runtime to allow them to be fixed up, seems a lot hackier. And the whole
>> return issue is pretty serious, too.
>> 
>> One last possibility to consider is something between the two: a
>> different kind of object, defined differently, like a proc in Ruby
>> (which is defined with a block rather than a def) might solve some of
>> the problems with either approach. And stealing from Ruby again, procs
>> have their own "mini-frames"; there's a two-level stack where every
>> function stack frame has a proc stack frame, which allows a solution to
>> the return-value problem that wouldn't be available with either closures
>> or fragments.
> 
> This may be closer to how I am thinking it would work.  :-)

It sounds like it might be. I think it's a really clumsy solution, but obviously it can work or Ruby wouldn't work. :)

Notice that Ruby also avoids a lot of the problems below by having completely different syntax and semantics for defining procs vs. functions, and not allowing you to convert them into each other. But if you don't want that, then you have to solve all the problems they got around this way.

>> (However, note that the return-value problem is much more
>> serious in Ruby, where everything is supposed to be an expression, with
>> a value; in Python you can just say "fragment calls are statements, so
>> they don't have values" if that's what you want.)
> 
> It seems to me they can be either.  Python ignores None when it's returned by a function and not assigned to anything.  And if a value is returned, than it's returned to the current position in the current frame.  

No, that's not how things work. Python doesn't do anything special with None. 

A value is _always_ returned, whether None or otherwise. That value always becomes the value of the calling expression. And Python certainly doesn't care whether you assign it to something--e.g., you can use it inside a larger expression or return it or yield it without assigning it to anything. Of course if the outermost expression is part of an expression statement, the value that it evaluated to is ignored, but that has nothing to do with the value being None, or coming from a function call, or anything else; an expression statement just means "evaluate this expression, then throw away the results".

Also, you're missing the bigger point: look at how RETURN_VALUE actually works. If you're running inside the caller's scope, it's going to return from the caller's scope unless you do something (which you need to figure out) to make that not true. 

> The return in this case is a non-local-block return.  So I think it wouldn't be an issue.

What does "a non-local-block return" mean at the implementation level? A different opcode from RETURN_VALUE? A RETURN_VALUE executed from within a non_local_block object's code? A RETURN_VALUE executed from a frame that has a non_local_block running? Whatever the answer, how does the compiler or interpreter (as appropriate) know to do that?

>> One last note inline:
>> 
>>>> A signature object could have a default body that returns the
>>>> closure.
>>>> 
>>>> And a body (or code) could have a default signature that*takes*  a
>>>> namespace.
>>>> 
>>>> 
>>>> Then a function becomes ...
>>>> 
>>>> code(sig(...))    <--->   function(...)
>>>> 
>>>> 
>>>> 
>>>> The separate parts could be created with a decorator.
>>>> 
>>>> @signature def sig_x(x): pass
>>>> 
>>>> @code def inc_x(): x += 1
>>>> 
>>>> @code def dec_x(): x -= 1
>>>> 
>>>> 
>>>> In most cases it's best to think of applying code bodies to names
>>>> spaces.
>>>> 
>>>> names = sig_x(0) inc_x(names) dec_x(names)
>>>> 
>>>> That is nicer than continuations as each code block is a well
>>>> defined unit that executes to completion and doesn't require
>>>> suspending the frame.
>>>> 
>>>> (Yes, it can be done with dictionaries, but that wouldn't give the
>>>> macro like functionality (see below) this would.  And there may be
>>>> other benifits to having it at a lower more efficient level.)
>>>> 
>>>> 
>>>> To allow macro like ability a code block needs to be executable in
>>>> the current scope.  That can be done just by doing...
>>>> 
>>>> code(locals())      #  Dependable?
>>>> 
>>>> 
>>>> And sugar to do that could be...
>>>> 
>>>> if x < 10: ^^ inc_x   #just an example syntax. else: ^^ dec_x   #
>>>> Note the ^^ looks like the M in Macro.;-)
>>>> 
>>>> 
>>>> Possibly the decorators could be used with lambda directly to get
>>>> inline functionality.
>>>> 
>>>> code(lambda : x + 1)
> 
>> This is a very different thing from what you were doing above. A
>> function that modifies a closure cell's value, like inc_x, can't be
>> written as a lambda (because assignments are statements). And this
>> lambda is completely pointless if you're going to use it in a context
>> where you ignore its return value (like the way you used inc_x above).
>> So, I'm not sure what you're trying to do here, but I think you may have
>> another problem to solve on top of the ones I already mentioned.
> 
> It was an incomplete example.  It should have been...
> 
>    add_1_to_x = code(lambda: x + 1)
> 
> and then later you could use it in the same way as above.
> 
>    x = ^^ add_1_to_x

OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant.

In fact, maybe it would be simpler to just do what Lisp does: explicitly define individual _variables_ as dynamically scoped, effectively the same way we can already define variables as global or nonlocal, instead of compiling a function and then trying to turn some of its variables into dynamic variables after the fact.

And the good news is, I'm 99% sure someone already did this and wrote a blog post about it. I don't know where, and it may be a few years and versions out of date, but it would be nice if you could look at what he did, see that you're 90% of the way to what you want, and just have to solve the last 10%.

Plus, you can experiment with this without hacking up anything, with a bit of clumsiness. It's pretty easy to create a class whose instances dynamically scope their attributes with an explicit stack. (If it isn't obvious how, let me know and I'll write it for you.) Then you just instantiate that class (globally, if you want), and have both the caller and the callee use an attribute of that instance instead of a normal variable whenever you want a dynamically-scoped variable, and you're done. You can write nice examples that actually work in Python today to show how this would be useful, and then compare to how much better it would look with real dynamic variable support.

> This is just an example to show how the first option above connects to the examples below.  with "^^: x + 1" being equivalent to "code(lambda: x + 1)".
> 
> Which would also be equivalent to ...
> 
>   @code
>   def add_1_to_x():
>       return x + 1
> 
>   x = ^^ add_1_to_x
> 
> 
> 
>>>> And a bit of sugar to shorten the common uses if needed.
>>>> 
>>>> spam(x + 1, code(lambda : x + 1))
>>>> 
>>>> spam(x + 1, ^^: x + 1)
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From jsbfox at gmail.com  Mon Mar 30 10:35:17 2015
From: jsbfox at gmail.com (Dmitry Kazakov)
Date: Mon, 30 Mar 2015 09:35:17 +0100
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAP7+vJLe2qXYOyCtHJ4kzOLuSU4xcyHd1Mb0-WG2rQpDxvFvqw@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAP7+vJLe2qXYOyCtHJ4kzOLuSU4xcyHd1Mb0-WG2rQpDxvFvqw@mail.gmail.com>
Message-ID: <CAMPw9HTZsD3r0sm1KiJW4Wggoeky3BN+i9gzSJ4_km-__dMLag@mail.gmail.com>

If the final resolution is "we don't need this in the library", that's fine
- I agree that the proposal is trivial and doesn't gain much. OTOH the
contextlib.closing can also be said to be a simple exercise, because its
implementation is 22 lines long and it's equivalent to

@contextmanager
def closing(thing):
    try:
        yield thing
    finally:
        thing.close()

(from the docs)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/352cb47c/attachment.html>

From abarnert at yahoo.com  Mon Mar 30 11:03:37 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Mar 2015 02:03:37 -0700
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAMPw9HTZsD3r0sm1KiJW4Wggoeky3BN+i9gzSJ4_km-__dMLag@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAP7+vJLe2qXYOyCtHJ4kzOLuSU4xcyHd1Mb0-WG2rQpDxvFvqw@mail.gmail.com>
 <CAMPw9HTZsD3r0sm1KiJW4Wggoeky3BN+i9gzSJ4_km-__dMLag@mail.gmail.com>
Message-ID: <CBB611ED-469E-494C-BCED-0CBB4C78479C@yahoo.com>

On Mar 30, 2015, at 01:35, Dmitry Kazakov <jsbfox at gmail.com> wrote:
> 
> If the final resolution is "we don't need this in the library", that's fine - I agree that the proposal is trivial and doesn't gain much. OTOH the contextlib.closing can also be said to be a simple exercise, because its implementation is 22 lines long and it's equivalent to
> 
> @contextmanager
> def closing(thing):
>     try:
>         yield thing
>     finally:
>         thing.close()
> 
> (from the docs)
I suspect that if closing were proposed today (or if you were designing Python from scratch today) it never would have made it into contextlib. At the time, nobody had any experience with context managers. And there used to be tons of types that it was useful for; now, most of those types are context managers and therefore don't need closing. Python didn't come with pip, so having things in the stdlib was a lot more important. And it was one of the motivating examples in the PEP, which made it a look like a no-brainer addition.

That doesn't mean there might not be a good argument for this idea. For example, if it turns out to be something that _looks_ trivial to code but is very easy to get subtly wrong, so a lot of people are likely to do it themselves and end up with buggy code, then it sounds like a good candidate. (Well, it would to me, but then I'm not who you have to convince...) But if you agree that it really is trivial, I don't see why it needs to be in the stdlib.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/0ed1c080/attachment.html>

From steve at pearwood.info  Mon Mar 30 13:11:11 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Mar 2015 22:11:11 +1100
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <mfaiek$qvt$1@ger.gmane.org>
 <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com>
Message-ID: <20150330111110.GF15925@ando.pearwood.info>

On Mon, Mar 30, 2015 at 12:03:47AM -0700, Andrew Barnert wrote:
> On Mar 29, 2015, at 21:12, Ron Adam <ron3200 at gmail.com> wrote:
[snip stuff about dynamic scoping]

Raymond Hettinger's ChainMap (now also in the std lib) may possibly be 
used to emulate different scoping rules, including dynamic.

http://code.activestate.com/recipes/577434-nested-contexts-a-chain-of-mapping-objects/

http://code.activestate.com/recipes/305268-chained-map-lookups/

If you can build a function object with __globals__ set to a chained 
map, you may be able to accomplish most of what you are discussing. 

By memory, __globals__ needs to be a dict, so you have to use a dict 
subclass, something like this:

class Context(ChainMap, dict): pass


-- 
Steve

From jsbueno at python.org.br  Mon Mar 30 14:52:28 2015
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Mon, 30 Mar 2015 09:52:28 -0300
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>
Message-ID: <CAH0mxTQXXkhjug9mD2+hDpA3etRFqj1Q07ozUrorcBnwVuuhCw@mail.gmail.com>

On 29 March 2015 at 19:01, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
> On 30 March 2015 at 04:04, Dmitry Kazakov <jsbfox at gmail.com> wrote:
>>
>> I propose adding a context manager to contextlib module, which would
>> restore the stream position of its argument at exit:
>>
>>     with contextlib.closing(file_like_object) as file:
>>         file.read(1)
>>         old = file.tell()
>>         with contextlib.keep_stream_position(file) as f:
>>             assert file is f
>>             assert file_like_object is f
>>             # Do something with the file(-like) object e.g. feed it
>>             # to `PIL.Image.open` (will consume the whole stream)
>>
>>         assert file.tell() == old
>>
>> This CM would allow us to reuse streams efficiently, without requiring
>> to call seek() explicitly. I will open a new issue and submit a patch
>> in case of favorable responses.
>
>
> I like the idea. Might need some bikeshedding on the name (maybe
> restore_position?).
>
> To test for seeking, I would suggest having a seek to current position
> during construction. Not perfect (e.g. there may be streams that can only
> seek forwards, not backwards) but I think it's the best non-destructive
> option.
>

If that is being done, anyway, I propose an optional parameter that
would allow the context to internally use itertools.tee if the stream
can't seek itself.

Maybe with even three states:
"strict":  raises an exception if the stream is not [back]seekable
"default": creates a tee cache if the stream is not back-seekable
"cache": always cache.

(And maybe a maxbuffer parameter to go along? )



> Tim Delaney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Mon Mar 30 15:52:16 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Mar 2015 06:52:16 -0700
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAH0mxTQXXkhjug9mD2+hDpA3etRFqj1Q07ozUrorcBnwVuuhCw@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>
 <CAH0mxTQXXkhjug9mD2+hDpA3etRFqj1Q07ozUrorcBnwVuuhCw@mail.gmail.com>
Message-ID: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com>

On Mar 30, 2015, at 05:52, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
> 
>> On 29 March 2015 at 19:01, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
>>> On 30 March 2015 at 04:04, Dmitry Kazakov <jsbfox at gmail.com> wrote:
>>> 
>>> I propose adding a context manager to contextlib module, which would
>>> restore the stream position of its argument at exit:
>>> 
>>>    with contextlib.closing(file_like_object) as file:
>>>        file.read(1)
>>>        old = file.tell()
>>>        with contextlib.keep_stream_position(file) as f:
>>>            assert file is f
>>>            assert file_like_object is f
>>>            # Do something with the file(-like) object e.g. feed it
>>>            # to `PIL.Image.open` (will consume the whole stream)
>>> 
>>>        assert file.tell() == old
>>> 
>>> This CM would allow us to reuse streams efficiently, without requiring
>>> to call seek() explicitly. I will open a new issue and submit a patch
>>> in case of favorable responses.
>> 
>> 
>> I like the idea. Might need some bikeshedding on the name (maybe
>> restore_position?).
>> 
>> To test for seeking, I would suggest having a seek to current position
>> during construction. Not perfect (e.g. there may be streams that can only
>> seek forwards, not backwards) but I think it's the best non-destructive
>> option.
> 
> If that is being done, anyway, I propose an optional parameter that
> would allow the context to internally use itertools.tee if the stream
> can't seek itself.

That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading.

> Maybe with even three states:
> "strict":  raises an exception if the stream is not [back]seekable
> "default": creates a tee cache if the stream is not back-seekable
> "cache": always cache.
> 
> (And maybe a maxbuffer parameter to go along? )
> 
> 
> 
>> Tim Delaney
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rosuav at gmail.com  Mon Mar 30 15:59:10 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 31 Mar 2015 00:59:10 +1100
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>
 <CAH0mxTQXXkhjug9mD2+hDpA3etRFqj1Q07ozUrorcBnwVuuhCw@mail.gmail.com>
 <1345887E-789E-4F74-B825-01E54A460135@yahoo.com>
Message-ID: <CAPTjJmoFvJ0Eb20f9vb=0NsVt6n1vTSpLAGCjc32xuhcX170GQ@mail.gmail.com>

On Tue, Mar 31, 2015 at 12:52 AM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
>> If that is being done, anyway, I propose an optional parameter that
>> would allow the context to internally use itertools.tee if the stream
>> can't seek itself.
>
> That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading.
>

That shouldn't be an issue, though, as those will just use seek/tell.
I'm more wondering about the ones that _do_ need the tee'ing; how,
after the context manager completes, do you have the file object
return something from the other half of the tee? I'm not sure this
mode will work. If the stream can't use seek(), this context manager
should simply let the exception bubble IMO.

ChrisA

From abarnert at yahoo.com  Mon Mar 30 17:36:40 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Mar 2015 08:36:40 -0700
Subject: [Python-ideas] Add a context manager to keep stream position
	unchanged
In-Reply-To: <CAPTjJmoFvJ0Eb20f9vb=0NsVt6n1vTSpLAGCjc32xuhcX170GQ@mail.gmail.com>
References: <CAMPw9HTPZGzLJgbU1F4nm-v0vtw3AvYA-P1nStxuiw85oy=B4w@mail.gmail.com>
 <CAN8CLgnjAuhZZ=r-DLvjOcZfu0jeOuL15mfjdnisbhTPHLBu1g@mail.gmail.com>
 <CAH0mxTQXXkhjug9mD2+hDpA3etRFqj1Q07ozUrorcBnwVuuhCw@mail.gmail.com>
 <1345887E-789E-4F74-B825-01E54A460135@yahoo.com>
 <CAPTjJmoFvJ0Eb20f9vb=0NsVt6n1vTSpLAGCjc32xuhcX170GQ@mail.gmail.com>
Message-ID: <A6D25C28-E1C8-4ECC-965C-3A301C7EABD1@yahoo.com>

On Mar 30, 2015, at 06:59, Chris Angelico <rosuav at gmail.com> wrote:
> 
> On Tue, Mar 31, 2015 at 12:52 AM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>>> If that is being done, anyway, I propose an optional parameter that
>>> would allow the context to internally use itertools.tee if the stream
>>> can't seek itself.
>> 
>> That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading.
> 
> That shouldn't be an issue, though, as those will just use seek/tell.

The whole point of this context manager is to seek at enter and tell at exit. If it's useful to people using files as iterators, it's at least as useful to people using them as files. More so, in fact, because most code that asks for a file-like iterator isn't going to want to assume it's actually a file, while code that asks for something that has file methods doesn't have that problem.

> I'm more wondering about the ones that _do_ need the tee'ing; how,
> after the context manager completes, do you have the file object
> return something from the other half of the tee?

You're right; tee doesn't do anything to the iterable you give it, it creates two new iterators that share the iterable and a cache. So the caller has to use one of those new iterators, which would be clumsy. Something like:

    with restoring_position(f, tee_if_needed=True) as f, flocal:
        # do stuff with flocal as an iterator of lines
    # do stuff with f, which is an iterator of the same lines

Of course this only works if f only lives within this scope, or is a closure or global variable (if it was a normal parameter, as soon as you return to the caller, he's got the original file object seeked to the wrong place). On top of the issue with f now being an iterator rather than whatever kind of file-like object you started with, the potential for confusion and misuse is so high that I don't think I'd do this even internally to my own project, much less suggest it for the stdlib... But at least the implementation is still pretty simple:

    @contextmanager
    def restoring_position(t, tee_if_needed=False):
        try:
            pos = f.tell()
            f.seek(pos)
        except (AttributeError, IForgetTheExceptionForNotSeekable):
            if not tee_jf_needed: raise
            f, flocal = tee(f)
            frestore = None
        else:
            flocal = frestore = f
        try:
            yield f, flocal
        finally:
            if frestore is not None:
                frestore.seek(pos)

A better solution might be to build an actual file tee--a BufferedReader or TextIOWrapper that wraps another object of the same type, doesn't discard used data from the buffer, and can therefore seek back to the start of its buffer without  seeking the wrapped file.

But even then, I don't think I'd use it this way. I'd construct the wrapper explicitly, and give it a method that returns a pos-restoring context manager.

> I'm not sure this
> mode will work. If the stream can't use seek(), this context manager
> should simply let the exception bubble IMO.

Agreed. If you really want to, just try the context manager, and fall back to tee if it fails. A couple lines of code, and a lot more explicit and readable, and probably not that commonly needed...


From ron3200 at gmail.com  Mon Mar 30 20:55:47 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 30 Mar 2015 14:55:47 -0400
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <mfaiek$qvt$1@ger.gmane.org>
 <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com>
Message-ID: <mfc67j$e4b$1@ger.gmane.org>



On 03/30/2015 03:03 AM, Andrew Barnert wrote:


>>> Break and continue are harder. Probably give an error the same as
>>> what would happen if they are used outside a loop. A break or return
>>> would need to be local to the loop. So you can't have a break in a
>>> non_local_block unless the block also has the loop in it. That keeps
>>> the associated parts local to each other.
>
> That definitely makes things simpler--and, I think, better. It will piss
> off people who expect (from Lisp or Ruby) to be able to do flow control
> across the boundaries of a non_local_block, but too bad for them.

Or in my own language I wrote to test ideas like this.  In it, the keywords 
are objects so you can return a statement/keyword from a function and have 
it execute at that location.  So for example a function can increment a 
counter and return a "nil" keyword, (a no-op like pythons pass keyword), 
and once a limit is reached, return a "break" keyword.  Which breaks the 
loop.  The memory model is by static hash-map lookup.  Each defined 
function object gets a reference to the parents hash-map.  So name 
resolution walks the hash-map tree until it finds the name.  There are a 
number of ways to make that more efficient, but for now it keeps the 
language simple.

Python on the other hand does many more checks at compile time, and pull's 
the name references into the code object.  That allows for smaller and 
faster byte code execution.  This seems to be where the main issues are.

But as you noted, many of the pieces are already there. So it shouldn't be 
that difficult to come up with a working test implementation.  For now, 
experimenting with exec may be the best way to test the idea.  And possibly 
come back with some real code to discuss and see where that goes.  I think 
until some real code is written we will go in circles pointing out things 
wrong with the idea.  So lets wait a bit for some real code and examples.


When people think of macro's they may be thinking of several different 
concepts.

One is to redefine a statement or expression in a way that makes it easier 
to use. In that case the expression is transposed to another form at 
compile time.  Lisp macro's work in this way.

Another is to in-line a block of code defined in a single location to 
numerous other locations.  Generally this is done with pre-processors.

I think in-lining python functions has been discussed before and that idea 
would overlap this one.


My interest in functions that can be taken apart and reused with 
signature/scope objects is a bit different.  The idea of mutating the name 
space by applying code to it rather than calling code by applying values to 
it.  (A normal function call.)  This of course is what objects do, they 
have a state and methods are used to alter the state.  But there are other 
things I think may be of interest in this idea that may relate back to 
other areas of python.

(NOTE: Just read Stevens message about Raymond's chainmap.  So I'm going to 
see it is useful.)


And the reason for bringing this idea up here was I think it could be used 
to implement the lite macro behaviour that was suggested with a bit of 
added syntax.

On the other hand it appears to me, that Python is going in the direction 
of making it easier to compile to C code.  More dynamic features may not be 
helpful in the long run.


>> >It was an incomplete example.  It should have been...
>> >
>> >    add_1_to_x = code(lambda: x + 1)
>> >
>> >and then later you could use it in the same way as above.
>> >
>> >    x = ^^ add_1_to_x
> OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant.
>
> In fact, maybe it would be simpler to just do what Lisp does: explicitly
> define individual_variables_ as dynamically scoped, effectively the same
> way we can already define variables as global or nonlocal, instead of
> compiling a function and then trying to turn some of its variables into
> dynamic variables after the fact.

It's something to try, but if a code block needs boiler plate to work, or 
the function it's put in needs it, it really isn't go to be very nice.

> And the good news is, I'm 99% sure someone already did this and wrote a
> blog post about it. I don't know where, and it may be a few years and
> versions out of date, but it would be nice if you could look at what he
> did, see that you're 90% of the way to what you want, and just have to
> solve the last 10%.
>
> Plus, you can experiment with this without hacking up anything, with a
> bit of clumsiness. It's pretty easy to create a class whose instances
> dynamically scope their attributes with an explicit stack. (If it isn't
> obvious how, let me know and I'll write it for you.) Then you just
> instantiate that class (globally, if you want), and have both the caller
> and the callee use an attribute of that instance instead of a normal
> variable whenever you want a dynamically-scoped variable, and you're done.
> You can write nice examples that actually work in Python today to show how
> this would be useful, and then compare to how much better it would look
> with real dynamic variable support.

I'm going to play with the idea a bit over the next few days.  :-)

Cheers,
   Ron



>>> This is just an example to show how the first option above connects
>>> tothe examples below. with "^^: x + 1" being equivalent to "code(lambda: x + 1)".
>> >
>> >Which would also be equivalent to ...
>> >
>> >   @code
>> >   def add_1_to_x():
>> >       return x + 1
>> >
>> >   x = ^^ add_1_to_x
>> >
>> >
>> >
>>>>> >>>>And a bit of sugar to shorten the common uses if needed.
>>>>> >>>>
>>>>> >>>>spam(x + 1, code(lambda : x + 1))
>>>>> >>>>
>>>>> >>>>spam(x + 1, ^^: x + 1)


From abarnert at yahoo.com  Mon Mar 30 22:54:54 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Mar 2015 13:54:54 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <mfc67j$e4b$1@ger.gmane.org>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <mfaiek$qvt$1@ger.gmane.org>
 <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <mfc67j$e4b$1@ger.gmane.org>
Message-ID: <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com>

On Mar 30, 2015, at 11:55, Ron Adam <ron3200 at gmail.com> wrote:
> 
> On 03/30/2015 03:03 AM, Andrew Barnert wrote:

[snip]

> And the reason for bringing this idea up here was I think it could be used to implement the lite macro behaviour that was suggested with a bit of added syntax.
> 
> On the other hand it appears to me, that Python is going in the direction of making it easier to compile to C code.  More dynamic features may not be helpful in the long run.

I don't think this is true. There definitely isn't any momentum in that direction--shedskin and its two competitors are dead, while PyPy and numba are alive and kicking ass. And I don't think it's desirable, or that the core developers think it's desirable.

If you think Guido's and Jukka's static type annotations are a step in the direction of static compilation, read the two PEPs; that's very explicitly not a goal, at least in the near term. (And the impression I get from the list discussions is that Guido is skeptical that it will be in the long term, but he's willing to keep an open mind.)

Anyway, usually, macros are used to generate a few functions that are called many times, not to generate a whole lot of functions that are only called once, so they don't cause any problems for a decent tracing JIT.

>>> >It was an incomplete example.  It should have been...
>>> >
>>> >    add_1_to_x = code(lambda: x + 1)
>>> >
>>> >and then later you could use it in the same way as above.
>>> >
>>> >    x = ^^ add_1_to_x
>> OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant.
>> 
>> In fact, maybe it would be simpler to just do what Lisp does: explicitly
>> define individual_variables_ as dynamically scoped, effectively the same
>> way we can already define variables as global or nonlocal, instead of
>> compiling a function and then trying to turn some of its variables into
>> dynamic variables after the fact.
> 
> It's something to try, but if a code block needs boiler plate to work, or the function it's put in needs it, it really isn't go to be very nice.

I think I didn't explain this very well.

Take a step back. Instead of having functions whose variables are all lexically scoped, and blocks whose variables are all dynamically scoped, what if you had just one kind of function, but two kinds of variables? Exactly like defvar in Lisp (although hopefully with a better name...). 

That's obviously more flexible. And it should be much simpler to implement. (And it's dead simple to fake in standard CPython 3.4, without hacking anything--as Steven pointed out, ChainMap makes it even easier).

Once you have dynamic variables, it's not that hard to add a new defblock statement that acts just like def except its variables are dynamic by default, instead of lexical.

That skips over all the problems with your first version and immediately gets you your second version, and it avoids most of the problems there as well. The only thing you're missing in the end is the ability to convert functions to blocks and vice-versa at runtime. That's the part that causes all the problems, and I don't think is actually necessary for what you want (but if I'm wrong, I guess never mind :).

From ron3200 at gmail.com  Tue Mar 31 00:36:21 2015
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 30 Mar 2015 18:36:21 -0400
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <20150328172624.GG25453@ando.pearwood.info>
 <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com>
 <20150329015144.GJ25453@ando.pearwood.info> <mf9c9t$b87$1@ger.gmane.org>
 <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <mfaiek$qvt$1@ger.gmane.org>
 <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <mfc67j$e4b$1@ger.gmane.org>
 <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com>
Message-ID: <mfcj56$8ul$1@ger.gmane.org>



On 03/30/2015 04:54 PM, Andrew Barnert wrote:
> That skips over all the problems with your first version and immediately
> gets you your second version, and it avoids most of the problems there
> as well. The only thing you're missing in the end is the ability to
> convert functions to blocks and vice-versa at runtime. That's the part
> that causes all the problems, and I don't think is actually necessary
> for what you want (but if I'm wrong, I guess never mind:).

I don't think its necessary either and converting function blocks to an 
insertable code block isn't what I had in mind.  If you can run a function 
block in specified frame, then it isn't needed.  Just grab the current 
frame, execute the function block with it, as a function.

Of course the programmer would be responsible for making sure the names in 
the code block are available in the name space used with it.  Name errors 
should propagate normally.

      result = call_with_frame(function_code_object, sys.getframe(0))

But I think it will still runs into the name issues you mentioned earlier.

And as you also mentioned it's quite possible someone has already done a 
call_with_frame function.  I just haven't found it yet.

Cheers,
    Ron


From tjreedy at udel.edu  Tue Mar 31 02:41:26 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 30 Mar 2015 20:41:26 -0400
Subject: [Python-ideas] PEP 484: Generating test inputs from type hints
Message-ID: <mfcqgd$mbb$1@ger.gmane.org>

PEP 484 proposes "a standard syntax for type annotations", principally 
for static analysis (and other static uses), with runtime checking or 
optimization mentioned as a possibility.

Another (real) runtime use is generating unit test inputs.  Today on 
python-list, David MacIver announced Hypothesis, a testing library that 
does just that. (The thread title is "Hypothesis 1.0: A production 
quality property-based testing library for Python".)  Here is an example 
from http://hypothesis.readthedocs.org/en/latest/quickstart.html

from hypothesis import given
@given([int])
def test_reversing_twice_gives_same_list(xs):
     assert xs == list(reversed(reversed(xs)))

When a test runner such as pytest calls the wrapper, it calls the real 
function with a variety of int lists (including  [], but details not 
important here). @given() also works with unittest TestCase methods.

When I mentioned PEP 484 and the alternative syntax List(int), he 
replied "Having something like that as standard would be great for 
Hypothesis and I intend to support it once it becomes available."

I presume 'support' could and would include a version of given() that 
accessed .__annotations__.  Of course, if every 'test_xyz' were to be 
decorated with the same decorator, then the decorator could be omitted 
and the fuzzing moved into the test runner.

-- 
Terry Jan Reedy


From guido at python.org  Tue Mar 31 04:18:24 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Mar 2015 19:18:24 -0700
Subject: [Python-ideas] PEP 484: Generating test inputs from type hints
In-Reply-To: <mfcqgd$mbb$1@ger.gmane.org>
References: <mfcqgd$mbb$1@ger.gmane.org>
Message-ID: <CAP7+vJL66fyknR6mywWtqxg574ACx2kOrUJAzpiBkSh+vm1exg@mail.gmail.com>

On Mon, Mar 30, 2015 at 5:41 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> PEP 484 proposes "a standard syntax for type annotations", principally for
> static analysis (and other static uses), with runtime checking or
> optimization mentioned as a possibility.
>
> Another (real) runtime use is generating unit test inputs.  Today on
> python-list, David MacIver announced Hypothesis, a testing library that
> does just that. (The thread title is "Hypothesis 1.0: A production quality
> property-based testing library for Python".)  Here is an example from
> http://hypothesis.readthedocs.org/en/latest/quickstart.html
>
> from hypothesis import given
> @given([int])
> def test_reversing_twice_gives_same_list(xs):
>     assert xs == list(reversed(reversed(xs)))
>
> When a test runner such as pytest calls the wrapper, it calls the real
> function with a variety of int lists (including  [], but details not
> important here). @given() also works with unittest TestCase methods.
>
> When I mentioned PEP 484 and the alternative syntax List(int), he replied
> "Having something like that as standard would be great for Hypothesis and I
> intend to support it once it becomes available."
>
> I presume 'support' could and would include a version of given() that
> accessed .__annotations__.  Of course, if every 'test_xyz' were to be
> decorated with the same decorator, then the decorator could be omitted and
> the fuzzing moved into the test runner.


This sounds cool. Note that you shouldn't access __annotations__ directly
-- it may contain string literals. The typing module exports a functio
get_type_hints() that takes a function object and returns a dict like
__annotations__ but with the string literals evaluated. It also honors the
@no_type_check decorator. For details see here:
https://github.com/ambv/typehinting/blob/master/prototyping/typing.py#L1143

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/509e8bac/attachment.html>

From mrocklin at gmail.com  Tue Mar 31 05:20:47 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Mon, 30 Mar 2015 20:20:47 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
Message-ID: <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>

Lisps like Scheme do indeed have an easier time with these due to the whole
code-is-data thing, it's quite doable in languages with real syntax
though.  R and Julia would be good examples of syntactic languages with
full macros.  The Julia implementation might be a good model for what
Python could do.   Their docs
<http://julia.readthedocs.org/en/latest/manual/metaprogramming/> are also a
nice read if anyone isn't familiar with the topic.

Macropy represents unevaluated expressions with the objects from the ast
module.  This seems like a sane choice.

To be a little pedantic I'll give a brief example loosely showing what a
macro is, then I'll talk about assert statements as a use case where macros
might help with a pain point in normal Python programming.

*Brief Educational Blurb*

We write code as text

defmacro f(x):
    ...

f(a + b) * sin(c)

We then parse parts of that text into syntax trees.

[image: Inline image 1]
Usually we translate these trees into byte-code and evaluate bottom-up,
starting with a + b, then applying f, etc... Macros stop this process.
They capture the subtrees beneath them before execution.  Whenever we see a
macro (f), we don't evaluate its subtree (a + b).  Instead we transform the
subtree into an in-memory representation (perhaps ast.BinOp(a, ast.Add(), b))
and hand that to f to do with as it will.  Lets see an example with
assertions.

*Use case with Assertions*

When testing we often want to write statements like the following

assert x == y
assert x in y
etc...

When these statements fail we want to emit statements that are well
informed of the full expression, e.g.

    5 != 6
    5 was not found in {1, 2, 3}

In Python we can't do this; assert only gets True or False and doesn't
understand what generated that value .  We've come up with a couple of
workarounds.  The first is the venerable unittest.TestCase methods that
take the two sides of the comparison explicitly e.g. assertEquals(a, b),
assertContains(a, b).  This was sufficiently uncomfortable that projects
like py.test arose and gained adoption.  Py.test goes through the trouble
of parsing the python test_.py files in order to generate nicer error
messages.

Having macros around would allow users to write this kind of functionality
directly in Python rather than resorting to full text parsing and code
transformation.  Macros provide an escape out of pure bottom-up evaluation.


On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum <guido at python.org> wrote:

> On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin <mrocklin at gmail.com>
> wrote:
>
>> Responding to comments off list:
>>
>> I'm not referring to C-style preprocessor macros, I'm referring to macros
>> historically found in functional languages and commonly found in many
>> user-targeted languages built in the last few years.
>>
>
> Do you have examples and references? IIRC there's something named macros
> in Scheme but Scheme, unlike Python, completely unifies code and data, and
> there is a standard in-memory representation for code.
>
>
>> The goal is to create things that look like functions but have access to
>> the expression that was passed in.
>>
>> Some examples where this is useful:
>>
>>     plot(year, miles / gallon)  # Plot with labels determined by
>> input-expressions, e.g. miles/gallon
>>
>>     assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
>> assertRaises function, not before
>>
>>     run_concurrently(f(x), f(y), f(z))  # Run f three times in three
>> threads controlled by run_concurrently
>>
>> Generally one constructs something that looks like a function but, rather
>> than receiving a pre-evaluated input, receives a syntax tree along with the
>> associated context.  This allows that function-like-thing to manipulate the
>> expression and to control the context in which the evaluation occurs.
>>
>
> None of the examples need the syntax tree though. The first wants the
> string, the last probably just want a way to turn an argument into a lambda.
>
>
>> There are lots of arguments against this, mostly focused around potential
>> misuse.  I'm looking for history of such arguments and for a general "Yes,
>> this is theoretically possible" or "Not a chance in hell" from the
>> community.  Both are fine.
>>
>
> I don't think this is a mainline need in Python, so it's probably both. :-)
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/57d05b51/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: macro.png
Type: image/png
Size: 17961 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/57d05b51/attachment-0001.png>

From mrocklin at gmail.com  Tue Mar 31 05:27:21 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Mon, 30 Mar 2015 20:27:21 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
Message-ID: <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>

Also, just to credentialize myself, I am not a huge Lisp lover.  I don't
want macros to do crazy logic programming or whatever.  I write numeric
code in the scientific Python ecosystem.  I want macros to build better
interfaces for downstream users.  This seems to be the modern use case in
user-focused languages rather than lisp-magic-hell.

On Mon, Mar 30, 2015 at 8:20 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Lisps like Scheme do indeed have an easier time with these due to the
> whole code-is-data thing, it's quite doable in languages with real syntax
> though.  R and Julia would be good examples of syntactic languages with
> full macros.  The Julia implementation might be a good model for what
> Python could do.   Their docs
> <http://julia.readthedocs.org/en/latest/manual/metaprogramming/> are also
> a nice read if anyone isn't familiar with the topic.
>
> Macropy represents unevaluated expressions with the objects from the ast
> module.  This seems like a sane choice.
>
> To be a little pedantic I'll give a brief example loosely showing what a
> macro is, then I'll talk about assert statements as a use case where macros
> might help with a pain point in normal Python programming.
>
> *Brief Educational Blurb*
>
> We write code as text
>
> defmacro f(x):
>     ...
>
> f(a + b) * sin(c)
>
> We then parse parts of that text into syntax trees.
>
> [image: Inline image 1]
> Usually we translate these trees into byte-code and evaluate bottom-up,
> starting with a + b, then applying f, etc... Macros stop this process.
> They capture the subtrees beneath them before execution.  Whenever we see a
> macro (f), we don't evaluate its subtree (a + b).  Instead we transform
> the subtree into an in-memory representation (perhaps ast.BinOp(a,
> ast.Add(), b)) and hand that to f to do with as it will.  Lets see an
> example with assertions.
>
> *Use case with Assertions*
>
> When testing we often want to write statements like the following
>
> assert x == y
> assert x in y
> etc...
>
> When these statements fail we want to emit statements that are well
> informed of the full expression, e.g.
>
>     5 != 6
>     5 was not found in {1, 2, 3}
>
> In Python we can't do this; assert only gets True or False and doesn't
> understand what generated that value .  We've come up with a couple of
> workarounds.  The first is the venerable unittest.TestCase methods that
> take the two sides of the comparison explicitly e.g. assertEquals(a, b),
> assertContains(a, b).  This was sufficiently uncomfortable that projects
> like py.test arose and gained adoption.  Py.test goes through the trouble
> of parsing the python test_.py files in order to generate nicer error
> messages.
>
> Having macros around would allow users to write this kind of functionality
> directly in Python rather than resorting to full text parsing and code
> transformation.  Macros provide an escape out of pure bottom-up evaluation.
>
>
> On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin <mrocklin at gmail.com>
>> wrote:
>>
>>> Responding to comments off list:
>>>
>>> I'm not referring to C-style preprocessor macros, I'm referring to
>>> macros historically found in functional languages and commonly found in
>>> many user-targeted languages built in the last few years.
>>>
>>
>> Do you have examples and references? IIRC there's something named macros
>> in Scheme but Scheme, unlike Python, completely unifies code and data, and
>> there is a standard in-memory representation for code.
>>
>>
>>> The goal is to create things that look like functions but have access to
>>> the expression that was passed in.
>>>
>>> Some examples where this is useful:
>>>
>>>     plot(year, miles / gallon)  # Plot with labels determined by
>>> input-expressions, e.g. miles/gallon
>>>
>>>     assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
>>> assertRaises function, not before
>>>
>>>     run_concurrently(f(x), f(y), f(z))  # Run f three times in three
>>> threads controlled by run_concurrently
>>>
>>> Generally one constructs something that looks like a function but,
>>> rather than receiving a pre-evaluated input, receives a syntax tree along
>>> with the associated context.  This allows that function-like-thing to
>>> manipulate the expression and to control the context in which the
>>> evaluation occurs.
>>>
>>
>> None of the examples need the syntax tree though. The first wants the
>> string, the last probably just want a way to turn an argument into a lambda.
>>
>>
>>> There are lots of arguments against this, mostly focused around
>>> potential misuse.  I'm looking for history of such arguments and for a
>>> general "Yes, this is theoretically possible" or "Not a chance in hell"
>>> from the community.  Both are fine.
>>>
>>
>> I don't think this is a mainline need in Python, so it's probably both.
>> :-)
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/fd2c3a60/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: macro.png
Type: image/png
Size: 17961 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/fd2c3a60/attachment-0001.png>

From mrocklin at gmail.com  Tue Mar 31 06:26:05 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Mon, 30 Mar 2015 21:26:05 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
Message-ID: <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>

>
> Is something stopping you from exploring this? Do you have specific ideas
> on how to improve on macropy?
>

Macropy is great but it requires an import-hook.  Many scientific users
work interactively.


> It sounds almost as if you would like to implement this but you want some
> kind of promise ahead of time that your work will be incorporated into the
> language. But that's just not how it works. When you want to explore a big
> idea like this, at some point you have to be willing to take the risk of
> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to
> be incorporated into Python -- in fact he was surprised at the amount of
> uptake it got.
>

The hard problem isn't building macros, it's deciding whether or not macros
are good for Python.  I'm trying to start a discussion.  If this isn't the
right place for that then I apologize.


> You've received quite a bit of feedback (and, may I say, push back :-)
> from a small number of python-ideas veterans -- you can take this or leave
> it, but at this point I think you've gotten about as much mileage out of
> the list as can be expected.
>

My apologies.  I didn't realize that I was misusing this list.  I also
didn't realize that I was receiving push-back, the comments here seemed
friendly and encouraging.

Last year at SciPy the message I heard was "If you want to convince the
core team then come to python-ideas armed with motivating use cases."  Here
I am :)

Anyway, if there isn't any interest then I'll leave off.  Thank you all for
your time,
-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/1397e7e1/attachment.html>

From mrocklin at gmail.com  Tue Mar 31 06:30:19 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Mon, 30 Mar 2015 21:30:19 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
Message-ID: <CAJ8oX-FYmAOVECGHWciR0JT5NO-Jje2ReF1bZY3vcJd-mLmc0w@mail.gmail.com>

Whoops, my apologies.  Apparently I don't get e-mails sent only to
python-ideas and not also to me.  There was a lot of conversation to which
I was ignorant.

On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Is something stopping you from exploring this? Do you have specific ideas
>> on how to improve on macropy?
>>
>
> Macropy is great but it requires an import-hook.  Many scientific users
> work interactively.
>
>
>> It sounds almost as if you would like to implement this but you want some
>> kind of promise ahead of time that your work will be incorporated into the
>> language. But that's just not how it works. When you want to explore a big
>> idea like this, at some point you have to be willing to take the risk of
>> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to
>> be incorporated into Python -- in fact he was surprised at the amount of
>> uptake it got.
>>
>
> The hard problem isn't building macros, it's deciding whether or not
> macros are good for Python.  I'm trying to start a discussion.  If this
> isn't the right place for that then I apologize.
>
>
>> You've received quite a bit of feedback (and, may I say, push back :-)
>> from a small number of python-ideas veterans -- you can take this or leave
>> it, but at this point I think you've gotten about as much mileage out of
>> the list as can be expected.
>>
>
> My apologies.  I didn't realize that I was misusing this list.  I also
> didn't realize that I was receiving push-back, the comments here seemed
> friendly and encouraging.
>
> Last year at SciPy the message I heard was "If you want to convince the
> core team then come to python-ideas armed with motivating use cases."  Here
> I am :)
>
> Anyway, if there isn't any interest then I'll leave off.  Thank you all
> for your time,
> -Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/ad4a39bc/attachment.html>

From mrocklin at gmail.com  Tue Mar 31 07:09:58 2015
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Mon, 30 Mar 2015 22:09:58 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
Message-ID: <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>

The !(x + y) solution would release some pressure.  People have considered
using lambda: to delay execution in Pandas queries.  The result is a bit
odd:

https://github.com/pydata/pandas/issues/9229#issuecomment-69691738

On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Is something stopping you from exploring this? Do you have specific ideas
>> on how to improve on macropy?
>>
>
> Macropy is great but it requires an import-hook.  Many scientific users
> work interactively.
>
>
>> It sounds almost as if you would like to implement this but you want some
>> kind of promise ahead of time that your work will be incorporated into the
>> language. But that's just not how it works. When you want to explore a big
>> idea like this, at some point you have to be willing to take the risk of
>> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to
>> be incorporated into Python -- in fact he was surprised at the amount of
>> uptake it got.
>>
>
> The hard problem isn't building macros, it's deciding whether or not
> macros are good for Python.  I'm trying to start a discussion.  If this
> isn't the right place for that then I apologize.
>
>
>> You've received quite a bit of feedback (and, may I say, push back :-)
>> from a small number of python-ideas veterans -- you can take this or leave
>> it, but at this point I think you've gotten about as much mileage out of
>> the list as can be expected.
>>
>
> My apologies.  I didn't realize that I was misusing this list.  I also
> didn't realize that I was receiving push-back, the comments here seemed
> friendly and encouraging.
>
> Last year at SciPy the message I heard was "If you want to convince the
> core team then come to python-ideas armed with motivating use cases."  Here
> I am :)
>
> Anyway, if there isn't any interest then I'll leave off.  Thank you all
> for your time,
> -Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/05ad0a8c/attachment.html>

From shoyer at gmail.com  Tue Mar 31 08:21:50 2015
From: shoyer at gmail.com (Stephan Hoyer)
Date: Mon, 30 Mar 2015 23:21:50 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
 <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>
Message-ID: <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>

Macros would be an extremely useful feature for pandas, the main data
analysis library for Python (for which I'm a core developer).

Why? Well, right now, R has better syntax than Python for writing data
analysis code. The difference comes down to two macros that R developers
have written within the past few years.

Here's an example borrowed from the documentation for the dplyr R package
[1]:

flights %>%
  group_by(year, month, day) %>%
  select(arr_delay, dep_delay) %>%
  summarise(
    arr = mean(arr_delay),
    dep = mean(dep_delay)
  ) %>%
  filter(arr > 30 | dep > 30)

Here "flights" is a dataframe, similar to a table in spreadsheet. It is
also the only global variables in the analysis -- variables like "year" and
"arr_delay" are actually columns in the dataframe. R evaluates variables
lazily, in the context of the provided frame. In Python, functions like
groupby_by would need to be macros.

The other macro is the "pipe" or chaining operator %>%. This operator is
used to avoid the need many temporary or highly nested expressions. The
result is quite readable, but again, it needs to be a macro, because
group_by and filter are simply functions that take a dataframe as their
first argument. The fact that chaining works with plain functions means
that it works even on libraries that weren't designed for it. We could do
function chaining in Python by abusing an exist binary operator like >> or
|, but all the objects on which it works would need to be custom types.

What does this example look using pandas? Well, it's not as nice, and
there's not much we can do about it because of the limitations of Python
syntax:

(flights
 .group_by('year', 'month', 'day')
 .select('arr_delay', 'dep_delay')
 .summarize(
    arr = lambda df: mean(df.arr_delay)),
    dep = lambda df: mean(df.dep_delay)))
 .filter(lambda df: (df.arr > 30) | (df.dep > 30)))

(Astute readers will note that I've taken a few liberties with pandas
syntax to make more similar to dplyr.)

Instead of evaluating expressions in the delayed context of a dataframes,
we use strings or functions. With all the lambdas there's a lot more noise
than the R example, and it's harder to keep track of what's on. In
principle we could simplify the lambda expressions to not use any arguments
(Matthew linked to the GitHub comment where I showed what that would look
like [2]), but the code remains awkwardly verbose.

For chaining, instead of using functions and the pipe operator, we use
methods. This works fine as long as users are only using pandas, but it
means that unlike R, the Python dataframe is a closed ecosystem. Python
developers (rightly) frown upon monkey-patching, so there's no way for
external libraries to add their own functions (e.g., for custom plotting or
file formats) on an equal footing to the methods built-in to pandas.

I hope these use cases are illustrative. I don't have strong opinions on
the technical merits of particular proposals. The "light lambda" syntax
described by Andrew Barnert would at least solve the delayed evaluation
use-case nicely, though the colon character is not ideal because it would
rule out using light lambdas inside indexing brackets.

Best,
Stephan

[1]
http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining
[2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/13367c1c/attachment-0001.html>

From songofacandy at gmail.com  Tue Mar 31 08:35:49 2015
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 31 Mar 2015 15:35:49 +0900
Subject: [Python-ideas] Adding rule about trailing comma.
Message-ID: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>

What about adding rule about trailing comma?

Yes:

foo(aaa,
    bbb)

or

foo(aaa,
    bbb,
)

No:

foo(aaa,
    bbb,)  # except one-value tuple.

or

foo(aaa,
    bbb
)



-- 
INADA Naoki  <songofacandy at gmail.com>

From david at drmaciver.com  Tue Mar 31 09:10:59 2015
From: david at drmaciver.com (David MacIver)
Date: Tue, 31 Mar 2015 09:10:59 +0200
Subject: [Python-ideas] PEP 484: Generating test inputs from type hints
Message-ID: <CADZYRLd37CYewza8kL8HG1A3_1SQK773ihmNGnW03vef8qZu=Q@mail.gmail.com>

(Sorry for the manual thread continuation, I wasn't previously on the
mailing list)

I presume 'support' could and would include a version of given() that
> accessed .__annotations__.


This is on my list of planned features independently of PEP 484 FWIW. It's
on the long list of nice to have but minor things that didn't make it into
1.0.

One of the problems is that I am for the moment somewhat committed to
supporting Python 2.7 as well as Python 3 (I really wish I wasn't), or I
would have probably built Hypothesis on using annotations from the
beginning.

To a degree though it's fortunate that I didn't given PEP 484. It doesn't
look like I would be able to build the full range of features Hypothesis
needs for @given using only type annotations, and 484 suggests the goal is
to move to those being the only valid ones.

The reason for this is that Hypothesis deliberately lets you mix strategies
in with the specifiers used for annotation. So for example you can do
something like:

Palindromes = strategy(str).map(lambda x: x + str(reversed(x)))

This is now a SearchStrategy object that still returns strings, but only
ones that are palindromes.

You can now do something like

@given([Palindromes])
def test_this_is_a_list_of_palindromes(xs):
   # I'm not actually sure what interesting things you can say about a list
of palindromes...

So the point is that you can use a SearchStrategy deep inside the "type"
signatures that Hypothesis uses, despite SearchStrategy not actually being
at all type-like.

(It actually used to be the case that an earlier version of SearchStrategy
allowed you to test whether a value could have come from that strategy,
which would have made them a bit more "type-like", but this proved to be a
bad idea and the work that let me remove it was one of the best decisions I
made in the design of Hypothesis)

So basically I would fully intend for Hypothesis to become a consumer of
types and annotations, but I don't think they can be used as a primary
building block of it.

(Note: I don't consider this a problem with PEP 484, which looks like a
good proposal. Hypothesis's requirements are just weird and specialized and
don't quite match type checking)


> Of course, if every 'test_xyz' were to be
> decorated with the same decorator, then the decorator could be omitted
> and the fuzzing moved into the test runner.


I'd be moderately against this being the primary mode of using Hypothesis,
even without the above caveats. One of the things I like about Hypothesis
right now is that because it just gives you Python functions it can be very
unopinionated about what you're using to run your tests (even if in
practice basically everyone is using unittest or pytest).  However it
should be relatively easy to support alongside once I've done some much
needed refactoring of the core example exploration code.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/845b3baf/attachment.html>

From abarnert at yahoo.com  Tue Mar 31 09:19:33 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 31 Mar 2015 00:19:33 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
 <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>
 <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>
Message-ID: <13487A22-3B2D-45F5-B950-FBC9FBB53FD4@yahoo.com>

On Mar 30, 2015, at 23:21, Stephan Hoyer <shoyer at gmail.com> wrote:
> 
> Macros would be an extremely useful feature for pandas, the main data analysis library for Python (for which I'm a core developer).
> 
> Why? Well, right now, R has better syntax than Python for writing data analysis code. The difference comes down to two macros that R developers have written within the past few years.
> 
> Here's an example borrowed from the documentation for the dplyr R package [1]:
> 
> flights %>%
>   group_by(year, month, day) %>%
>   select(arr_delay, dep_delay) %>%
>   summarise(
>     arr = mean(arr_delay),
>     dep = mean(dep_delay)
>   ) %>%
>   filter(arr > 30 | dep > 30)
> 
> Here "flights" is a dataframe, similar to a table in spreadsheet. It is also the only global variables in the analysis -- variables like "year" and "arr_delay" are actually columns in the dataframe. R evaluates variables lazily, in the context of the provided frame. In Python, functions like groupby_by would need to be macros.
> 
> The other macro is the "pipe" or chaining operator %>%. This operator is used to avoid the need many temporary or highly nested expressions. The result is quite readable, but again, it needs to be a macro, because group_by and filter are simply functions that take a dataframe as their first argument. The fact that chaining works with plain functions means that it works even on libraries that weren't designed for it. We could do function chaining in Python by abusing an exist binary operator like >> or |, but all the objects on which it works would need to be custom types.
> 
> What does this example look using pandas? Well, it's not as nice, and there's not much we can do about it because of the limitations of Python syntax:
> 
> (flights
>  .group_by('year', 'month', 'day')
>  .select('arr_delay', 'dep_delay')
>  .summarize(
>     arr = lambda df: mean(df.arr_delay)),
>     dep = lambda df: mean(df.dep_delay)))
>  .filter(lambda df: (df.arr > 30) | (df.dep > 30)))
> 
> (Astute readers will note that I've taken a few liberties with pandas syntax to make more similar to dplyr.)
> 
> Instead of evaluating expressions in the delayed context of a dataframes, we use strings or functions. With all the lambdas there's a lot more noise than the R example, and it's harder to keep track of what's on. In principle we could simplify the lambda expressions to not use any arguments (Matthew linked to the GitHub comment where I showed what that would look like [2]), but the code remains awkwardly verbose.
> 
> For chaining, instead of using functions and the pipe operator, we use methods. This works fine as long as users are only using pandas, but it means that unlike R, the Python dataframe is a closed ecosystem. Python developers (rightly) frown upon monkey-patching, so there's no way for external libraries to add their own functions (e.g., for custom plotting or file formats) on an equal footing to the methods built-in to pandas.

One way around this is to provide a documented, clean method for hooking your types--e.g., a register classmethod that then makes the function appear as a method in all instances. 

Functionally this is the same as monkeypatching, but it looks a lot more inviting to the user. (And it also allows you to rewrite things under the covers in ways that would break direct monkeypatching, if you ever want to.) There are more examples of opening up modules this way than classes, but it's the same idea.

> I hope these use cases are illustrative. I don't have strong opinions on the technical merits of particular proposals. The "light lambda" syntax described by Andrew Barnert would at least solve the delayed evaluation use-case nicely, though the colon character is not ideal because it would rule out using light lambdas inside indexing brackets.

The bare colon was just one of multiple suggestions that came up last time the idea was discussed (last February or so), and in previous discussions (going back to the long-abandoned PEP 312). In many cases, it looks very nice, but in others it's either ugly or (at least to a human) ambiguous without parens (which obviously defeat the whole point). I don't think anyone noticed the indexing issue, but someone (Nick Coghlan, I think) pointed out essentially the same issue in dict displays (e.g., for making a dynamic jump table).

If you seriously want to revive that discussion--which might be worth doing, if your use cases are sufficiently different from the ones that were discussed (Tkinter and RPC server callbacks were the primary motivating case)--somewhere I have notes on all the syntaxes people suggested that I could dump on you.

For the strings, another possibility is a namespace object that can effectively defer attribute lookup and have it done by the real table when you get it, as done by various ORMs, appscript, etc. Instead of this:

> (flights
>  .group_by('year', 'month', 'day')

... You write:

> (flights
>  .group_by(c.year, c.month, c.day)

That "c" is just an instance of a simple type that wraps its __getattr__ argument up so you can access it later--when get an argument to group_by that's one of those wrappers, you look it up on the table.

It's the same number of characters, and looks a little more magical, but arguably it's more readable, at least once people are used to your framework.

For example, here's a query expression to find broken tracks (with missing filenames) in iTunes via appscript that was used in a real application (until iTunes Match made this no longer work...):

    playlist.tracks[its.location == k.missing]

Both its and k are special objects you import from appscript; its.__getattr__ returns a wrapper that's used to look up "location" in the members of whatever class playlist.tracks instances turn out to be at runtime, and k.missing returns a wrapper that's used to look up "missing" in the global keywords of whatever app playlist turns out to be part of at runtime.

You can take this even further: if mean weren't a function that takes a column and returns its mean, but instead a function that takes a column name and returns a function that takes a table and returns the mean of the column with that name, then you could just write this:

>  .summarize(
>     arr = mean(c.arr_delay)

But of course that isn't what mean is. But it can be what, say, f.mean is, if f is another one of those attribute-delaying objects: f.mean(c.arr_delay) returns a wrapper that summarize can use to call the function named "mean" on the column named "arr_delay".

So, the whole thing reduces to:

> (flights
>  .group_by(c.year, c.month, c.day)
>  .select(c.arr_delay, c.dep_delay)
>  .summarize(
>     arr = f.mean(c.arr_delay)),
>     dep = f.mean(c.dep_delay)))
>   .filter(c.arr > 30 | c.dep > 30)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/8cb98e76/attachment-0001.html>

From liik.joonas at gmail.com  Tue Mar 31 09:22:11 2015
From: liik.joonas at gmail.com (Joonas Liik)
Date: Tue, 31 Mar 2015 10:22:11 +0300
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
 <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>
 <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>
Message-ID: <CAB1GNpQ=EDQHDr74809ZEtXGOB3g8ErqySgg7dxnuwX9HpqPsg@mail.gmail.com>

isn't this:
    lambda df: mean(df.arr_delay)
the same as..
    functools.partial(mean, df.arr_delay)

I kind of like the idea of a pipe operator (tho %>% looks just terrible and
| is already taken)
But consider: if we could get functools.compose that too would be
alleviated.
    (something like
https://mathieularose.com/function-composition-in-python/  )


On 31 March 2015 at 09:21, Stephan Hoyer <shoyer at gmail.com> wrote:

> Macros would be an extremely useful feature for pandas, the main data
> analysis library for Python (for which I'm a core developer).
>
> Why? Well, right now, R has better syntax than Python for writing data
> analysis code. The difference comes down to two macros that R developers
> have written within the past few years.
>
> Here's an example borrowed from the documentation for the dplyr R package
> [1]:
>
> flights %>%
>   group_by(year, month, day) %>%
>   select(arr_delay, dep_delay) %>%
>   summarise(
>     arr = mean(arr_delay),
>     dep = mean(dep_delay)
>   ) %>%
>   filter(arr > 30 | dep > 30)
>
> Here "flights" is a dataframe, similar to a table in spreadsheet. It is
> also the only global variables in the analysis -- variables like "year" and
> "arr_delay" are actually columns in the dataframe. R evaluates variables
> lazily, in the context of the provided frame. In Python, functions like
> groupby_by would need to be macros.
>
> The other macro is the "pipe" or chaining operator %>%. This operator is
> used to avoid the need many temporary or highly nested expressions. The
> result is quite readable, but again, it needs to be a macro, because
> group_by and filter are simply functions that take a dataframe as their
> first argument. The fact that chaining works with plain functions means
> that it works even on libraries that weren't designed for it. We could do
> function chaining in Python by abusing an exist binary operator like >> or
> |, but all the objects on which it works would need to be custom types.
>
> What does this example look using pandas? Well, it's not as nice, and
> there's not much we can do about it because of the limitations of Python
> syntax:
>
> (flights
>  .group_by('year', 'month', 'day')
>  .select('arr_delay', 'dep_delay')
>  .summarize(
>     arr = lambda df: mean(df.arr_delay)),
>     dep = lambda df: mean(df.dep_delay)))
>  .filter(lambda df: (df.arr > 30) | (df.dep > 30)))
>
> (Astute readers will note that I've taken a few liberties with pandas
> syntax to make more similar to dplyr.)
>
> Instead of evaluating expressions in the delayed context of a dataframes,
> we use strings or functions. With all the lambdas there's a lot more noise
> than the R example, and it's harder to keep track of what's on. In
> principle we could simplify the lambda expressions to not use any arguments
> (Matthew linked to the GitHub comment where I showed what that would look
> like [2]), but the code remains awkwardly verbose.
>
> For chaining, instead of using functions and the pipe operator, we use
> methods. This works fine as long as users are only using pandas, but it
> means that unlike R, the Python dataframe is a closed ecosystem. Python
> developers (rightly) frown upon monkey-patching, so there's no way for
> external libraries to add their own functions (e.g., for custom plotting or
> file formats) on an equal footing to the methods built-in to pandas.
>
> I hope these use cases are illustrative. I don't have strong opinions on
> the technical merits of particular proposals. The "light lambda" syntax
> described by Andrew Barnert would at least solve the delayed evaluation
> use-case nicely, though the colon character is not ideal because it would
> rule out using light lambdas inside indexing brackets.
>
> Best,
> Stephan
>
> [1]
> http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining
> [2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/6a85d197/attachment.html>

From abarnert at yahoo.com  Tue Mar 31 09:30:50 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 31 Mar 2015 00:30:50 -0700
Subject: [Python-ideas] Adding rule about trailing comma.
In-Reply-To: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
References: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
Message-ID: <E4AAA09E-B795-4329-82D2-6FD8498DBBA2@yahoo.com>

On Mar 30, 2015, at 23:35, INADA Naoki <songofacandy at gmail.com> wrote:
> 
> What about adding rule about trailing comma?

I don't understand the proposed rule from your example. Are you suggesting that a trailing comma be a syntax error unless it's following by a newline? That would explain both "yesses" and the first "no", but then what's wrong with the second "no"? You can always add newlines in the middle of any parenthesized expression; are you suggesting that you shouldn't be allowed to add one at the very end, unless it's immediately following a comma? If so, why?

Also, you seem to realize you're adding a gratuitous inconsistency with one-element tuples. Haven't they been through enough teasing without having to make them even more different?

Most of all, whatever your actual rule is: Why? What's the intended benefit here?

> 
> Yes:
> 
> foo(aaa,
>    bbb)
> 
> or
> 
> foo(aaa,
>    bbb,
> )
> 
> No:
> 
> foo(aaa,
>    bbb,)  # except one-value tuple.
> 
> or
> 
> foo(aaa,
>    bbb
> )
> 
> 
> 
> -- 
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From rosuav at gmail.com  Tue Mar 31 09:36:58 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 31 Mar 2015 18:36:58 +1100
Subject: [Python-ideas] Adding rule about trailing comma.
In-Reply-To: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
References: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
Message-ID: <CAPTjJmp+0CjZEmH9FtxV8OTtLjj5XVaFRd+qrNBBeXF+9jpaUQ@mail.gmail.com>

On Tue, Mar 31, 2015 at 5:35 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> What about adding rule about trailing comma?

My crystal ball tells me that you're possibly talking about PEP 8
rather than about actual language changes. Is that correct?

There's absolutely no need to put restrictions like this into the
language itself. If the trailing comma is permitted, it should be
permitted whether there's a newline after it or not.

In your own style guide, you're most welcome to advocate what you
will, without needing any approval from the Python core devs. If you
want to say, for instance, "we follow PEP 8, with the following
additional requirements", and then list your trailing comma rule and
any others you want to require, people will understand. Actually, most
people will do something like this, anyway.

ChrisA

From abarnert at yahoo.com  Tue Mar 31 09:57:26 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 31 Mar 2015 00:57:26 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAB1GNpQ=EDQHDr74809ZEtXGOB3g8ErqySgg7dxnuwX9HpqPsg@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
 <CAJ8oX-Gu7C7sTRHK0XQ7Hs-vGa5if_xFHHbfW07MsTE3bA-zNg@mail.gmail.com>
 <CAEQ_TvdWQhktGyVONKgLu985iF8zkg8tAGkzHmJUh0OH_wN87A@mail.gmail.com>
 <CAB1GNpQ=EDQHDr74809ZEtXGOB3g8ErqySgg7dxnuwX9HpqPsg@mail.gmail.com>
Message-ID: <DD47624A-F44B-4F9F-9783-DC3B12DE7F78@yahoo.com>

On Mar 31, 2015, at 00:22, Joonas Liik <liik.joonas at gmail.com> wrote:
> 
> isn't this:
>     lambda df: mean(df.arr_delay)
> the same as..
>     functools.partial(mean, df.arr_delay)

No, because in the first one df is a parameter (which gets the value of whatever DataFrame this is run on) while in the second it's a free variable (which just raises NameError, if you're lucky).

You could do this with operator.attrgetter and composition, of course:

    compose(mean, attrgetter('arr_delay'))

But I don't think that's more readable. Even if composition were an infix operator:

    (mean . attrgetter('arr_delay'))

> I kind of like the idea of a pipe operator (tho %>% looks just terrible and | is already taken)
> But consider: if we could get functools.compose that too would be alleviated.
>     (something like  https://mathieularose.com/function-composition-in-python/  )

Why do you need to "get functools.compose"? If you just want the trivial version from that blog post, it's two lines that any novice can write himself. If you want one of the more complex versions that he dismisses, then there might be a better argument, but the post you linked argues against that, not for it. And if you don't trust yourself to write the two lines yourself, you can always pip install funcy or toolz or functional3 or more-functools.

As far as using compose again for piping... Well, it's backward from what you want, and it also requires you to stack up enough parens to choke a Lisp guru, not to mention all the repetitions of compose itself (that's why Haskell has infix compose and apply operators with the precedence and associativity they have, so you can avoid all the parens).

>> On 31 March 2015 at 09:21, Stephan Hoyer <shoyer at gmail.com> wrote:
>> Macros would be an extremely useful feature for pandas, the main data analysis library for Python (for which I'm a core developer).
>> 
>> Why? Well, right now, R has better syntax than Python for writing data analysis code. The difference comes down to two macros that R developers have written within the past few years.
>> 
>> Here's an example borrowed from the documentation for the dplyr R package [1]:
>> 
>> flights %>%
>>   group_by(year, month, day) %>%
>>   select(arr_delay, dep_delay) %>%
>>   summarise(
>>     arr = mean(arr_delay),
>>     dep = mean(dep_delay)
>>   ) %>%
>>   filter(arr > 30 | dep > 30)
>> 
>> Here "flights" is a dataframe, similar to a table in spreadsheet. It is also the only global variables in the analysis -- variables like "year" and "arr_delay" are actually columns in the dataframe. R evaluates variables lazily, in the context of the provided frame. In Python, functions like groupby_by would need to be macros.
>> 
>> The other macro is the "pipe" or chaining operator %>%. This operator is used to avoid the need many temporary or highly nested expressions. The result is quite readable, but again, it needs to be a macro, because group_by and filter are simply functions that take a dataframe as their first argument. The fact that chaining works with plain functions means that it works even on libraries that weren't designed for it. We could do function chaining in Python by abusing an exist binary operator like >> or |, but all the objects on which it works would need to be custom types.
>> 
>> What does this example look using pandas? Well, it's not as nice, and there's not much we can do about it because of the limitations of Python syntax:
>> 
>> (flights
>>  .group_by('year', 'month', 'day')
>>  .select('arr_delay', 'dep_delay')
>>  .summarize(
>>     arr = lambda df: mean(df.arr_delay)),
>>     dep = lambda df: mean(df.dep_delay)))
>>  .filter(lambda df: (df.arr > 30) | (df.dep > 30)))
>> 
>> (Astute readers will note that I've taken a few liberties with pandas syntax to make more similar to dplyr.)
>> 
>> Instead of evaluating expressions in the delayed context of a dataframes, we use strings or functions. With all the lambdas there's a lot more noise than the R example, and it's harder to keep track of what's on. In principle we could simplify the lambda expressions to not use any arguments (Matthew linked to the GitHub comment where I showed what that would look like [2]), but the code remains awkwardly verbose.
>> 
>> For chaining, instead of using functions and the pipe operator, we use methods. This works fine as long as users are only using pandas, but it means that unlike R, the Python dataframe is a closed ecosystem. Python developers (rightly) frown upon monkey-patching, so there's no way for external libraries to add their own functions (e.g., for custom plotting or file formats) on an equal footing to the methods built-in to pandas.
>> 
>> I hope these use cases are illustrative. I don't have strong opinions on the technical merits of particular proposals. The "light lambda" syntax described by Andrew Barnert would at least solve the delayed evaluation use-case nicely, though the colon character is not ideal because it would rule out using light lambdas inside indexing brackets.
>> 
>> Best,
>> Stephan
>> 
>> [1] http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining
>> [2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738
>> 
>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/c18c9435/attachment-0001.html>

From songofacandy at gmail.com  Tue Mar 31 10:28:31 2015
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 31 Mar 2015 17:28:31 +0900
Subject: [Python-ideas] Adding rule about trailing comma.
In-Reply-To: <E4AAA09E-B795-4329-82D2-6FD8498DBBA2@yahoo.com>
References: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
 <E4AAA09E-B795-4329-82D2-6FD8498DBBA2@yahoo.com>
Message-ID: <CAEfz+TxqFZgCpbatvTdHromqDLq_P_0eQHoGz0Rnh6pO3Qh+Ng@mail.gmail.com>

On Tue, Mar 31, 2015 at 4:30 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Mar 30, 2015, at 23:35, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> What about adding rule about trailing comma?
>
> I don't understand the proposed rule from your example. Are you suggesting that a trailing comma be a syntax error unless it's following by a newline?

I'm sorry, I meant about adding rule to PEP 8. Not Python syntax.

> That would explain both "yesses" and the first "no", but then what's wrong with the second "no"? You can always add newlines in the middle of any parenthesized expression; are you suggesting that you shouldn't be allowed to add one at the very end, unless it's immediately following a comma? If so, why?
>
> Also, you seem to realize you're adding a gratuitous inconsistency with one-element tuples. Haven't they been through enough teasing without having to make them even more different?
>
> Most of all, whatever your actual rule is: Why? What's the intended benefit here?

I recently write Go program more than Python.
I feel good about line continuation rule in Go: Some characters
including comma means line continuation.

Benefit of this style is:

1) More consistent
2) No garbage diff just for just adding comma

>
>>
>> Yes:
>>
>> foo(aaa,
>>    bbb)
>>
>> or
>>
>> foo(aaa,
>>    bbb,
>> )
>>
>> No:
>>
>> foo(aaa,
>>    bbb,)  # except one-value tuple.
>>
>> or
>>
>> foo(aaa,
>>    bbb
>> )
>>
>>
>>
>> --
>> INADA Naoki  <songofacandy at gmail.com>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
INADA Naoki  <songofacandy at gmail.com>

From rosuav at gmail.com  Tue Mar 31 10:34:19 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 31 Mar 2015 19:34:19 +1100
Subject: [Python-ideas] Adding rule about trailing comma.
In-Reply-To: <CAEfz+TxqFZgCpbatvTdHromqDLq_P_0eQHoGz0Rnh6pO3Qh+Ng@mail.gmail.com>
References: <CAEfz+TyChY3TgXUh+=qw5DHQnqw7mRJQ22qqSDK1b=VeE6dVVg@mail.gmail.com>
 <E4AAA09E-B795-4329-82D2-6FD8498DBBA2@yahoo.com>
 <CAEfz+TxqFZgCpbatvTdHromqDLq_P_0eQHoGz0Rnh6pO3Qh+Ng@mail.gmail.com>
Message-ID: <CAPTjJmpY5bAGU7JDZWVr-wr_8Dnw6PwBEY9_KCwhUZSu9VLnLA@mail.gmail.com>

On Tue, Mar 31, 2015 at 7:28 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> Benefit of this style is:
>
> 1) More consistent
> 2) No garbage diff just for just adding comma

So what you have there is a good reason for adopting this style. Is
there strong reason for mentioning it in PEP 8? Remember, every
recommendation in a style guide has a cost - too many rules and it
just becomes onerous, not to mention the increased likelihood of the
rule giving the wrong advice in some corner case (which leads
intelligent people to disregard the rule, and unintelligent checkers
to spew warnings which then have to be ignored - this is one of the
fundamental difficulties of scripting PEP 8 or any other style guide);
ideally, trust people to be intelligent, and don't enforce more than
you have to.

ChrisA

From guido at python.org  Tue Mar 31 05:47:17 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Mar 2015 20:47:17 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
Message-ID: <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>

Matthew,

Is something stopping you from exploring this? Do you have specific ideas
on how to improve on macropy?

It sounds almost as if you would like to implement this but you want some
kind of promise ahead of time that your work will be incorporated into the
language. But that's just not how it works. When you want to explore a big
idea like this, at some point you have to be willing to take the risk of
writing code without a guaranteed pay off. Haoyi didn't ask for macropy to
be incorporated into Python -- in fact he was surprised at the amount of
uptake it got.

You've received quite a bit of feedback (and, may I say, push back :-) from
a small number of python-ideas veterans -- you can take this or leave it,
but at this point I think you've gotten about as much mileage out of the
list as can be expected.

Good luck!

--Guido

On Mon, Mar 30, 2015 at 8:27 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Also, just to credentialize myself, I am not a huge Lisp lover.  I don't
> want macros to do crazy logic programming or whatever.  I write numeric
> code in the scientific Python ecosystem.  I want macros to build better
> interfaces for downstream users.  This seems to be the modern use case in
> user-focused languages rather than lisp-magic-hell.
>
> On Mon, Mar 30, 2015 at 8:20 PM, Matthew Rocklin <mrocklin at gmail.com>
> wrote:
>
>> Lisps like Scheme do indeed have an easier time with these due to the
>> whole code-is-data thing, it's quite doable in languages with real syntax
>> though.  R and Julia would be good examples of syntactic languages with
>> full macros.  The Julia implementation might be a good model for what
>> Python could do.   Their docs
>> <http://julia.readthedocs.org/en/latest/manual/metaprogramming/> are
>> also a nice read if anyone isn't familiar with the topic.
>>
>> Macropy represents unevaluated expressions with the objects from the ast
>> module.  This seems like a sane choice.
>>
>> To be a little pedantic I'll give a brief example loosely showing what a
>> macro is, then I'll talk about assert statements as a use case where macros
>> might help with a pain point in normal Python programming.
>>
>> *Brief Educational Blurb*
>>
>> We write code as text
>>
>> defmacro f(x):
>>     ...
>>
>> f(a + b) * sin(c)
>>
>> We then parse parts of that text into syntax trees.
>>
>> [image: Inline image 1]
>> Usually we translate these trees into byte-code and evaluate bottom-up,
>> starting with a + b, then applying f, etc... Macros stop this process.
>> They capture the subtrees beneath them before execution.  Whenever we see a
>> macro (f), we don't evaluate its subtree (a + b).  Instead we transform
>> the subtree into an in-memory representation (perhaps ast.BinOp(a,
>> ast.Add(), b)) and hand that to f to do with as it will.  Lets see an
>> example with assertions.
>>
>> *Use case with Assertions*
>>
>> When testing we often want to write statements like the following
>>
>> assert x == y
>> assert x in y
>> etc...
>>
>> When these statements fail we want to emit statements that are well
>> informed of the full expression, e.g.
>>
>>     5 != 6
>>     5 was not found in {1, 2, 3}
>>
>> In Python we can't do this; assert only gets True or False and doesn't
>> understand what generated that value .  We've come up with a couple of
>> workarounds.  The first is the venerable unittest.TestCase methods that
>> take the two sides of the comparison explicitly e.g. assertEquals(a, b),
>> assertContains(a, b).  This was sufficiently uncomfortable that projects
>> like py.test arose and gained adoption.  Py.test goes through the trouble
>> of parsing the python test_.py files in order to generate nicer error
>> messages.
>>
>> Having macros around would allow users to write this kind of
>> functionality directly in Python rather than resorting to full text parsing
>> and code transformation.  Macros provide an escape out of pure bottom-up
>> evaluation.
>>
>> On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin <mrocklin at gmail.com>
>>> wrote:
>>>
>>>> Responding to comments off list:
>>>>
>>>> I'm not referring to C-style preprocessor macros, I'm referring to
>>>> macros historically found in functional languages and commonly found in
>>>> many user-targeted languages built in the last few years.
>>>>
>>>
>>> Do you have examples and references? IIRC there's something named macros
>>> in Scheme but Scheme, unlike Python, completely unifies code and data, and
>>> there is a standard in-memory representation for code.
>>>
>>>
>>>> The goal is to create things that look like functions but have access
>>>> to the expression that was passed in.
>>>>
>>>> Some examples where this is useful:
>>>>
>>>>     plot(year, miles / gallon)  # Plot with labels determined by
>>>> input-expressions, e.g. miles/gallon
>>>>
>>>>     assertRaises(ZeroDivisionError, 1/0)  # Evaluate the rhs 1/0 within
>>>> assertRaises function, not before
>>>>
>>>>     run_concurrently(f(x), f(y), f(z))  # Run f three times in three
>>>> threads controlled by run_concurrently
>>>>
>>>> Generally one constructs something that looks like a function but,
>>>> rather than receiving a pre-evaluated input, receives a syntax tree along
>>>> with the associated context.  This allows that function-like-thing to
>>>> manipulate the expression and to control the context in which the
>>>> evaluation occurs.
>>>>
>>>
>>> None of the examples need the syntax tree though. The first wants the
>>> string, the last probably just want a way to turn an argument into a lambda.
>>>
>>>
>>>> There are lots of arguments against this, mostly focused around
>>>> potential misuse.  I'm looking for history of such arguments and for a
>>>> general "Yes, this is theoretically possible" or "Not a chance in hell"
>>>> from the community.  Both are fine.
>>>>
>>>
>>> I don't think this is a mainline need in Python, so it's probably both.
>>> :-)
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/3e2e218b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: macro.png
Type: image/png
Size: 17961 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150330/3e2e218b/attachment-0001.png>

From guido at python.org  Tue Mar 31 16:52:38 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 31 Mar 2015 07:52:38 -0700
Subject: [Python-ideas] History on proposals for Macros?
In-Reply-To: <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
References: <CAJ8oX-Fo4vGYyZTQvZthv74sXYfm=YOizMf2e0dY=sFUU9TwRQ@mail.gmail.com>
 <CALxg4FX3o0SP4a2P-9eEijLvDx+qsgJGTitABFFx4+Q0Yf8zkA@mail.gmail.com>
 <CAJ8oX-GwpFML6+p5ivLk3BmD43UeK7VitvKc3PZBLYmr1dwz8w@mail.gmail.com>
 <CAP7+vJLe8xcAZCfL65JDZv=LLXY3Cec9xvKe4T3jzN_0g9eTsA@mail.gmail.com>
 <CAJ8oX-GDb=RT+cqn1Crb8gJ2NQNJH1rOOeXQmvv98Sxrte3FZA@mail.gmail.com>
 <CAJ8oX-G5pzy06TqWy8qSnwnvt+5abi9ZXAXcDNfh8OBbHFy_xg@mail.gmail.com>
 <CAP7+vJJ=stZfLL68fYDHXb3F-N-EUU0ErNPf8UBBDPm0UFg-yw@mail.gmail.com>
 <CAJ8oX-G5JcboyOJtDT2f2M6w4deP1kNH4H+3Zuq6bV=L+OYY=A@mail.gmail.com>
Message-ID: <CAP7+vJKACghN_2d9KhSExoe3iZSpwLwjgip7kqdjWx6fgmjKqg@mail.gmail.com>

On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Is something stopping you from exploring this? Do you have specific ideas
>> on how to improve on macropy?
>>
>
> Macropy is great but it requires an import-hook.  Many scientific users
> work interactively.
>
>
>> It sounds almost as if you would like to implement this but you want some
>> kind of promise ahead of time that your work will be incorporated into the
>> language. But that's just not how it works. When you want to explore a big
>> idea like this, at some point you have to be willing to take the risk of
>> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to
>> be incorporated into Python -- in fact he was surprised at the amount of
>> uptake it got.
>>
>
> The hard problem isn't building macros, it's deciding whether or not
> macros are good for Python.  I'm trying to start a discussion.  If this
> isn't the right place for that then I apologize.
>

This is the right place, and we're now at the point where it's your job to
either show a concrete design spec that can actually be implemented, and
have its tires kicked, or just go off and build something. In the latter
case you'll probably learn about some practical issues that nobody might
have thought of yet.


> You've received quite a bit of feedback (and, may I say, push back :-)
>> from a small number of python-ideas veterans -- you can take this or leave
>> it, but at this point I think you've gotten about as much mileage out of
>> the list as can be expected.
>>
>
> My apologies.  I didn't realize that I was misusing this list.  I also
> didn't realize that I was receiving push-back, the comments here seemed
> friendly and encouraging.
>

You weren't misusing the list. Maybe (based on your next message) you
weren't reading it though. :-)


> Last year at SciPy the message I heard was "If you want to convince the
> core team then come to python-ideas armed with motivating use cases."  Here
> I am :)
>
> Anyway, if there isn't any interest then I'll leave off.  Thank you all
> for your time,
>

I think you misunderstand. There's interest but there are also real
concerns. I really do think that an implementation (if, as you say, that
isn't the hard part) would be very helpful to judge whether it is a
desirable feature. (Maybe you discover you can do it in a way that can be
distributed via PYPI -- "pip install macros".)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150331/815d6598/attachment.html>

From guettliml at thomas-guettler.de  Tue Mar 31 21:17:39 2015
From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=)
Date: Tue, 31 Mar 2015 21:17:39 +0200
Subject: [Python-ideas] infinite dates: psycopg2 would support them
Message-ID: <551AF2D3.7010102@thomas-guettler.de>

Hi,

any progress on infinite datetimes?

I just got the commitment from psycopg2 (lib to connect to PostgreSQL) developers, 
that they will support them, if python would:

  https://github.com/psycopg/psycopg2/issues/283#issuecomment-88016783

I can't help you with code in this context.

I have never started a crowd funding campaign. Do you think it would
be successful?


Regards,
  Thomas G?ttler

-- 
http://www.thomas-guettler.de/