From tjreedy at udel.edu  Sun Jul  1 00:01:08 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 30 Jun 2012 18:01:08 -0400
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <CAMgkT__4EUv-Qwi8mu3bg3iwunszOcJ-_ZY4GsV81_UJriTB1A@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<CAEjd=rqy2+M34yPSnJLY834D6=Dk3050+GjPpvoE850dfjk2xQ@mail.gmail.com>
	<CAMgkT__4EUv-Qwi8mu3bg3iwunszOcJ-_ZY4GsV81_UJriTB1A@mail.gmail.com>
Message-ID: <jsnsvd$bto$1@dough.gmane.org>

On 6/30/2012 11:54 AM, Christopher Reay wrote:
> The only hope for a large archive like this one is to wait long enough
> to make sure you dont re hash the really regular ideas.
>
> ... ponders... Do I have time to read the archives?

If you have time to post and discuss an idea, you have time to search 
gmane.comp.python.ideas at search.gmane.org and perhaps find out why not 
to bother posting.

? Do people mind adminiing the repetitive ideas?

yes.

-- 
Terry Jan Reedy





From tjreedy at udel.edu  Sun Jul  1 00:57:58 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 30 Jun 2012 18:57:58 -0400
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <CAEBZo3M2gcM5MmXHVXr+Y4ObeXL=_jScZBOivJq88ZY7mAMPxg@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
	<CAEBZo3M2gcM5MmXHVXr+Y4ObeXL=_jScZBOivJq88ZY7mAMPxg@mail.gmail.com>
Message-ID: <4FEF8476.1050609@udel.edu>

On 6/29/2012 5:36 PM, Mike Graham wrote:
> On Fri, Jun 29, 2012 at 4:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:
>> so far there were no negative votes
>
> As far as I know, Raymond Hettinger is the itertools maintainer and he
> has repeatedly objected to this idea in the past (e.g.
> http://bugs.python.org/issue6021 ). Hopefully we can get his input
> again.

See my other post on this thread. I think people should really spend a 
few minutes researching before posting repeated ideas. In this case, 
read the itertools doc to see if such a function exists. It does, in the 
recipes, as 'grouper'. Has anyone ever before proposed that the recipe 
be added as a function? Search the tracker for 'itertools grouper'. 
Besides #6021, there are also

http://bugs.python.org/issue1643 'Add group() to itertools'

Raymond: "Sorry, I'm not interested in adding this to the module. 
Discussions to-
date on the subject seem to show more interest in playing with grouper
variants than in actual use cases.  While the recipe given in the docs
is somewhat opaque, it runs at C-speed (zero trips around the eval-
loop) and it is encapsulated in a re-usable function. Writing this in C
does nothing to improve the situation.  Also, when people like to play
with variants, there is no general agreement on useful requirements
(like fill-in behavior or raising an exception on uneven length
inputs). Trying to write option to meet all needs (n=2, step=1) makes
the code more difficult to learn and use -- see several variants in
Alex's Python Cookbook.  Another issue is that we have to be very
selective about adding tools to the module.  Each addition makes the
overall toolset harder to use -- it is better to have a good set of
basic building blocks."

http://bugs.python.org/issue13095
"Support for splitting lists/tuples into chunks"

Raymond: "These have been rejected before.  There is always a trade-off 
in adding tools such as this -- it can take more time to learn and 
remember them than to write a trivial piece of code to do it yourself. 
Another issue is that people tend to disagree on how to handle an odd 
sized left-over group -- different use cases require different handling.

We're trying to keep the core toolset reasonably small so that python 
remains simple and learnable.  That raises the threshold for adding new 
tools."

There are now 18 itertools, up from the original. Grouper, or any 
generic function, may not be the best for what one wants with a list. I 
proposed in my other post that we *do* need a new doc section or how-to 
on this topic. (I am working on an outline.)

-- 
Terry Jan Reedy





From steve at pearwood.info  Sun Jul  1 02:24:47 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 01 Jul 2012 10:24:47 +1000
Subject: [Python-ideas] the optional "as" statement inside
	"if"	statements
In-Reply-To: <CAGaVwhSHx_pmhEc=ncc6yFPqo_Up3sAMzZ9DJvTF-KgFof1CVw@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CAGaVwhSHx_pmhEc=ncc6yFPqo_Up3sAMzZ9DJvTF-KgFof1CVw@mail.gmail.com>
Message-ID: <4FEF98CF.5050804@pearwood.info>

Calvin Spealman wrote:
> On Sat, Jun 30, 2012 at 9:59 AM,  <fiatjaf at yahoo.com.br> wrote:
>> the idea is to make an variable assignment at the same time that the
>> existence of that variable -- which is being returned by a function -- is
>> made.
>>
>> suppose we are returning a variable from the method 'get' from the 'request'
>> object and them making some stuff with it, but that stuff we will only do if
>> it exists, if not, we'll just pass, instead of writing:
>>
>> variable = self.request.get('variable')
>> if variable:
>>    print variable
>>
>> we could write
>>
>> if self.request.get('variable') as variable:
>>    print variable
>>
>> seems stupid (or not?), but with lots of variables to process, this
>> pre-assignment could be very unpleasant -- especially if, as the in the
>> example case, very little use will be made of the tested variable.
>>
>> also, the "as" expression already exists and is very pythonic.
> 
> This is probably the best solution to the problem 


What problem?

I'm not convinced that what fiatjaf at yahoo.com.br describes is an actual 
problem that needs solving. So what if you have to bind a reference to a name 
before using it in an if statement? That's not a problem. "Solving" it doesn't 
make programming any easier, or give you more power. There is absolutely no 
difference in programming power or expressiveness between:

if func(a, b) as x:
     process(x)

and

x = func(a, b)
if x:
     process(x)


(In fact, I would argue that the second is *slightly* more readable, as the 
name binding comes first, not last.)

At best, this proposal merely adds a trivial bit of syntactic sugar to the 
language. Contrast that with "import as", which truly does add expressiveness 
to the language. Working around the lack of "import as" if Python didn't have 
it would not be so easy. The obvious solution is also the wrong solution:

# import long_name as ln
import long_name
ln = long_name
del long_name


That's wrong, because it has the side-effect of replacing, then deleting, any 
existing binding to long_name. "import as" does not do that.

Good suggestions for syntactic sugar should add power or expressiveness to the 
language, not merely save a line of code or a few keystrokes. As given, this 
idea becomes one more special case for people to learn, with no corresponding 
benefit.

-1 on the idea as given.


By the way, to the Original Poster, please configure your mail client to 
display a name that we can refer to you by, or sign your posts. It doesn't 
have to be your real name, just something that you would like to be known as. 
It is annoying to have to refer to you by your email address.


-- 
Steven


From mwm at mired.org  Sun Jul  1 02:48:52 2012
From: mwm at mired.org (Mike Meyer)
Date: Sat, 30 Jun 2012 20:48:52 -0400
Subject: [Python-ideas] the optional "as" statement inside "if"
 statements
In-Reply-To: <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
Message-ID: <20120630204852.4b898a0a@bhuda.mired.org>

On Sun, 1 Jul 2012 01:06:39 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> This proposal has been considered and rejected many times. It's not
> general enough - it *only* works for those cases where the value to be
> retained *and* the interesting condition are the same.
> 
> Consider the simple case of a value that may be either None (not
> interesting) or a number (interesting). Since the interesting values
> include "0", which evaluates as False along with None, this limited
> form of embedded assignment syntax would not help.

What Nick failed to point out is that the existing uses of "as" all
bind *specific classes of objects*, not general ones.

> Embedded assignment in C isn't that limited., but nobody has yet
> volunteered to take the radical step of proposing "(X as Y)" as a
> general embedded assignment syntax. I suggest anyone consider such an
> idea do a *lot* of research in the python-ideas archives first, though
> (as the idea has seen plenty of discussion). It is not as obviously
> flawed as the if-and-while statement only variant, but it would still
> involve being rather persuasive to make such a significant change to
> the language.

Hasn't the BDFL rejected a general embedded assignment in any case?

As a final note, reading checking the archives for your ideas will
also give you an idea of what kinds of things are and aren't accepted
(i.e. - what's "Pythonic") as well as process - including the kinds of
questions you'll be asked here - that an idea goes through before
being accepted.

      <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From steve at pearwood.info  Sun Jul  1 02:59:23 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 01 Jul 2012 10:59:23 +1000
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsn7um$2j1$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org>
Message-ID: <4FEFA0EB.4030407@pearwood.info>

Serhiy Storchaka wrote:
> As shown in issue #15016 [1], there is a use cases when it is useful to 
> determine that string can be encoded in ASCII or Latin1. In working with 
> Tk or Windows console applications can be useful to determine that 
> string can be encoded in UCS2. C API provides interface for this, but at 
> Python level it is not available.
> 
> I propose to add to strings class new methods: isascii(), islatin1() and 
> isbmp() (in addition to such methods as isalpha() or isdigit()). The 
> implementation will be trivial.
> 
> Pro: The current trick with trying to encode has O(n) complexity and has 
> overhead of exception raising/catching.

Are you suggesting that isascii and friends would be *better* than O(n)? How 
can that work -- wouldn't it have to scan the string and look at each character?

Why just ASCII, Latin1 and BMP (whatever that is, googling has not come up 
with anything relevant)? It seems to me that adding these three tests will 
open the doors to a steady stream of requests for new methods
is<insert encoding name here>.

I suggest that a better API would be a method that takes the name of an 
encoding (perhaps defaulting to 'ascii') and returns True|False:

string.encodable(encoding='ascii') -> True|False

Return True if string can be encoded using the named encoding, otherwise False.


One last pedantic issue: strings aren't ASCII or Latin1, etc., but Unicode. 
There is enough confusion between Unicode text strings and bytes without 
adding methods whose names blur the distinction slightly.




-- 
Steven



From greg.ewing at canterbury.ac.nz  Sun Jul  1 01:22:26 2012
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 01 Jul 2012 11:22:26 +1200
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsnbel$pua$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org>
	<CADiSq7e_hGXhjVuHjAUMV4WsdOMVQMMH5duhtja5pGG73tBV2A@mail.gmail.com>
	<20120630184316.5de2ce5e@pitrou.net> <jsnbel$pua$1@dough.gmane.org>
Message-ID: <4FEF8A32.50607@canterbury.ac.nz>

Serhiy Storchaka wrote:
> Several enchantments have already been rejected for this reason.
           ^^^^^^^^^^^^

Yeah, programming does seem to be a black art sometimes...

--
Greg


From tjreedy at udel.edu  Sun Jul  1 04:05:21 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 30 Jun 2012 22:05:21 -0400
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <4FEFA0EB.4030407@pearwood.info>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
Message-ID: <jsob9b$qo7$1@dough.gmane.org>

On 6/30/2012 8:59 PM, Steven D'Aprano wrote:

> Why just ASCII, Latin1 and BMP (whatever that is, googling has not come
> up with anything relevant)?

BMP = Unicode Basic MultilingualPlane, the first 2**16 codepoints
http://unicode.org/roadmaps/bmp/

I presume the proposed isbmp would exclude surrogates in 16 bit 
implementations, but that was not clearly defined.

> It seems to me that adding these three tests

The temptation for these three tests is the the info is already 
available (at least for 2 of them) as an internal 
implementation-specific C-level attribute in 3.3(+). No O(n) scan 
needed. Someone could make a CPython3.3-specific module available on PyPI.

> will open the doors to a steady stream of requests for new methods
> is<insert encoding name here>.

> I suggest that a better API would be a method that takes the name of an
> encoding (perhaps defaulting to 'ascii') and returns True|False:
>
> string.encodable(encoding='ascii') -> True|False
>
> Return True if string can be encoded using the named encoding, otherwise
> False.

But then one might as well try the encoding and check for exception. The 
point of the proposal is to avoid things like

try:
   body = text.encode('ascii')
   header = 'ascii'  #abbreviating here
except UnicodeEncodeError:
   try:
     body = text.encode('latin1')
     header = 'latin1'
   except UnicodeEncodeError:
     body = text.encode('utf-8')
     header = 'utf-8'

> One last pedantic issue: strings aren't ASCII or Latin1, etc., but
> Unicode. There is enough confusion between Unicode text strings and
> bytes without adding methods whose names blur the distinction slightly.

yes!

-- 
Terry Jan Reedy





From steve at pearwood.info  Sun Jul  1 05:21:32 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 01 Jul 2012 13:21:32 +1000
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsob9b$qo7$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org>
Message-ID: <4FEFC23C.3040805@pearwood.info>

Terry Reedy wrote:
> On 6/30/2012 8:59 PM, Steven D'Aprano wrote:

>> I suggest that a better API would be a method that takes the name of an
>> encoding (perhaps defaulting to 'ascii') and returns True|False:
>>
>> string.encodable(encoding='ascii') -> True|False
>>
>> Return True if string can be encoded using the named encoding, otherwise
>> False.
> 
> But then one might as well try the encoding and check for exception. The 
> point of the proposal is to avoid things like
> 
> try:
>   body = text.encode('ascii')
>   header = 'ascii'  #abbreviating here
> except UnicodeEncodeError:
>   try:
>     body = text.encode('latin1')
>     header = 'latin1'
>   except UnicodeEncodeError:
>     body = text.encode('utf-8')
>     header = 'utf-8'

Right. And re-written with the hypothetical encodable method, you have the 
usual advantage of LBYL that it is slightly more concise:

body = header = None
for encoding in ('ascii', 'latin1', 'utf-8'):
     if text.encodable(encoding):
         body = text.encode(encoding)
         header = encoding


instead of:

body = header = None
for encoding in ('ascii', 'latin1', 'utf-8'):
     try:
         body = text.encode(encoding)
         header = encoding
     except UnicodeEncodeError:
         pass


As for as expressibility goes, it is not much of an advantage. But:

- if there are optimizations that apply to some encodings but not others,
   the encodable method can take advantage of them without it being a
   promise of the language;

- it only adds a single string method (and presumably a single bytes
   method, decodable) rather than a plethora of methods;


So, I don't care much either way for a LBYL test, but if there is a good use 
case for such a test, better for it to be a single method taking the encoding 
name rather than a multitude of tests, or exposing an implementation-specific 
value that the coder then has to interpret themselves.

-1 on isascii, islatin1, isbmp
-1 on exposing max_code_point
+0.5 on encodable




-- 
Steven


From storchaka at gmail.com  Sun Jul  1 07:31:05 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 01 Jul 2012 08:31:05 +0300
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <4FEF8A32.50607@canterbury.ac.nz>
References: <jsn7um$2j1$1@dough.gmane.org>
	<CADiSq7e_hGXhjVuHjAUMV4WsdOMVQMMH5duhtja5pGG73tBV2A@mail.gmail.com>
	<20120630184316.5de2ce5e@pitrou.net> <jsnbel$pua$1@dough.gmane.org>
	<4FEF8A32.50607@canterbury.ac.nz>
Message-ID: <json9m$r67$1@dough.gmane.org>

On 01.07.12 02:22, Greg Ewing wrote:
> Serhiy Storchaka wrote:
>> Several enchantments have already been rejected for this reason.
>            ^^^^^^^^^^^^
>
> Yeah, programming does seem to be a black art sometimes...

Sorry, I mean enhancements. But yes, there is no good programming 
without magic.



From stefan_ml at behnel.de  Sun Jul  1 07:44:56 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Jul 2012 07:44:56 +0200
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <4FEF6B10.9040409@udel.edu>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
Message-ID: <jsoo4p$via$1@dough.gmane.org>

Terry Reedy, 30.06.2012 23:09:
> For the doc, I think it would be helpful here and in most module
> subchapters if there were a subchapter table of contents at the top (under
> 9.1 in this case). Even though just 2 lines here (currently, but see
> below), it would let people know that there *is* a recipes section. After
> the appropriate tables, mention that there are example uses in the recipe
> section. Possibly add similar tables in the recipe section.

+1, the recipes are interesting enough to be highly visible, both as usage
examples and to solve actual problems.


> Another addition could be a new subsection on grouping (chunking) that
> would discuss post-processing of grouper (as discussed above), as well as
> other recipes, including ones specific to strings and sequences. It would
> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
> Chunking Sequences and Iterables". The synonyms will help external
> searching. A toc would let people who have found this doc know to look for
> this at the bottom.

If it really is such an important use case for so many people, I agree that
it's worth special casing it in the docs. It's not a trivial algorithmic
step from a sequential iterable to a grouped iterable.

Stefan



From tjreedy at udel.edu  Sun Jul  1 07:48:14 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 01 Jul 2012 01:48:14 -0400
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <4FEFC23C.3040805@pearwood.info>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
Message-ID: <jsooav$11k$1@dough.gmane.org>

On 6/30/2012 11:21 PM, Steven D'Aprano wrote:
> Terry Reedy wrote:
>> On 6/30/2012 8:59 PM, Steven D'Aprano wrote:
>
>>> I suggest that a better API would be a method that takes the name of an
>>> encoding (perhaps defaulting to 'ascii') and returns True|False:
>>>
>>> string.encodable(encoding='ascii') -> True|False
>>>
>>> Return True if string can be encoded using the named encoding, otherwise
>>> False.
>>
>> But then one might as well try the encoding and check for exception.
>> The point of the proposal is to avoid things like
>>
>> try:
>>   body = text.encode('ascii')
>>   header = 'ascii'  #abbreviating here
>> except UnicodeEncodeError:
>>   try:
>>     body = text.encode('latin1')
>>     header = 'latin1'
>>   except UnicodeEncodeError:
>>     body = text.encode('utf-8')
>>     header = 'utf-8'
>
> Right. And re-written with the hypothetical encodable method, you have
> the usual advantage of LBYL that it is slightly more concise:
>
> body = header = None
> for encoding in ('ascii', 'latin1', 'utf-8'):
>      if text.encodable(encoding):
>          body = text.encode(encoding)
>          header = encoding

But you are doing about half the work twice.


> instead of:
>
> body = header = None
> for encoding in ('ascii', 'latin1', 'utf-8'):
>      try:
>          body = text.encode(encoding)
>          header = encoding
>      except UnicodeEncodeError:
>          pass

> As for as expressibility goes, it is not much of an advantage. But:
>
> - if there are optimizations that apply to some encodings but not others,
>    the encodable method can take advantage of them without it being a
>    promise of the language;

It would be an optimization limited to a couple of encodings with 
CPython. Using it for cross-version code would be something like the 
trap of depending on the CPython optimization of repeated string 
concatenation.

> - it only adds a single string method (and presumably a single bytes
>    method, decodable) rather than a plethora of methods;

Decodable would always require a scan of the bytes. Might as well just 
decode and look for UnicodeDecodeError.

> So, I don't care much either way for a LBYL test, but if there is a good
> use case for such a test,

My claim is that there is only a good use case if it is O(1), which 
would only be a few cases on CPython.

> better for it to be a single method taking the
> encoding name rather than a multitude of tests, or exposing an
> implementation-specific value that the coder then has to interpret
> themselves.
>
> -1 on isascii, islatin1, isbmp

I do not see much of any use for isbmp. Maybe I missed something in the 
original post.

> -1 on exposing max_code_point

Jython and IronPython are stuck with the underlying platform 
implementations, which I believe are like the current semi-utf-16 narrow 
builds. So it would have to be a CPython-only attribute for now. (PyPy 
might consider adopting the new Unicode implementation someday too.)

> +0.5 on encodable

encodable would indirectly expose max_code_point since it would only be 
really useful and likely used when max_code_point was available and 
applicable. In other words, s.encodable('latin1') is equivalent to 
s.max_code_point == 255.

if isbmp *is* useful, I don't think it can be duplicated with 
.encodable. Python seems not to have a ucs-2 codec.

-- 
Terry Jan Reedy





From ncoghlan at gmail.com  Sun Jul  1 08:27:25 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Jul 2012 16:27:25 +1000
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsooav$11k$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
	<jsooav$11k$1@dough.gmane.org>
Message-ID: <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com>

On Sun, Jul 1, 2012 at 3:48 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> encodable would indirectly expose max_code_point since it would only be
> really useful and likely used when max_code_point was available and
> applicable. In other words, s.encodable('latin1') is equivalent to
> s.max_code_point == 255.
>
> if isbmp *is* useful, I don't think it can be duplicated with .encodable.
> Python seems not to have a ucs-2 codec.

Rewinding back to the reasons the question is being asked, the reason
this information is useful at the Python level is the same reason it
is useful at the C level: it matters for finding the most efficient
means of
representing the text as bytes (which can then have further
implications for the kind of quoting used, etc). The interesting
breakpoints can actually be expressed in terms of the number of bits
in the highest code point:
7 - encode as ASCII (or latin-1 or utf-8)
8 - encode as latin-1
8+ - encode as utf-8

Specifically, it's a payload microoptimisation for the latin-1 case -
the latin-1 string will be shorter than the corresponding utf-8 string
(how much shorter depends on the number of non-ASCII characters). I
believe it also makes an additional difference in the email case by
changing the kind of quoting that is used to something with lower
overhead that can't handle utf-8.

The "try it and see" approach suffers a potentially high speed penalty
if the non-latin-1 characters appear late in the string:

    try:
        # Likely no need to try ASCII, since there's no efficiency
gain over latin-1
        payload = message.encode("latin-1")
    except UnicodeEncodeError:
        payload = message.encode("utf-8")

Using max() and ord() to check in advance doesn't help, since that
*locks in* the O(n) penalty.

The reason I think a max_code_point() method is a good potential
solution is that it can be advertised as O(n) worst case, but
potentially O(1) if the implementation caches the answer internally.
Another alternative would be a __max__ and __min__ protocol that
allowed efficient answers for the max() and min() builtins. The latter
would have the advantage of allowing other containers (like range
objects) to provide efficient implementations.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From raymond.hettinger at gmail.com  Sun Jul  1 09:07:56 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 1 Jul 2012 00:07:56 -0700
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <jsoo4p$via$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
Message-ID: <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>


On Jun 30, 2012, at 10:44 PM, Stefan Behnel wrote:

>> 
>> Another addition could be a new subsection on grouping (chunking) that
>> would discuss post-processing of grouper (as discussed above), as well as
>> other recipes, including ones specific to strings and sequences. It would
>> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
>> Chunking Sequences and Iterables". The synonyms will help external
>> searching. A toc would let people who have found this doc know to look for
>> this at the bottom.
> 
> If it really is such an important use case for so many people, I agree that
> it's worth special casing it in the docs. It's not a trivial algorithmic
> step from a sequential iterable to a grouped iterable.

I'm not too keen on adding a section like this to the itertools docs.

Instead, I would be open adding "further reading" section with external links 
to interesting iterator writeups in blogs, cookbooks, stack overflow answers, wikis, etc.

If one of you wants to craft an elegant blog post on "Grouping, Blocking, or
Chunking Sequences and Iterables", I would be happy to link to it.


Raymond




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120701/0532fc7f/attachment.html>

From solipsis at pitrou.net  Sun Jul  1 18:22:33 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Jul 2012 18:22:33 +0200
Subject: [Python-ideas] Happy leap second
References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com>
Message-ID: <20120701182233.589552b5@pitrou.net>

On Sat, 30 Jun 2012 10:23:45 -0400
Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:

> Even though many have hoped that the authorities would stop fiddling
> with our clocks, today a leap second will be inserted in UTC.
> Systems using Olson/IANA timezone database have a way to deal with
> this without adjusting their clocks, but few systems are configured
> that way:
> 
> $ TZ=right/UTC date -d @1341100824
> Sat Jun 30 23:59:60 UTC 2012

If I believe the Internet, many server systems got broken by the leap
second (*).
Even one of our buildbots seems affected:
- before:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/419
- after:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/420
(weird errors with timeouts, and a crash)


(*) https://linuxfr.org/users/nono/journaux/leap-second

Regards

Antoine.




From guido at python.org  Sun Jul  1 21:11:25 2012
From: guido at python.org (Guido van Rossum)
Date: Sun, 1 Jul 2012 21:11:25 +0200
Subject: [Python-ideas] Happy leap second
In-Reply-To: <20120701182233.589552b5@pitrou.net>
References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com>
	<20120701182233.589552b5@pitrou.net>
Message-ID: <CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com>

So would "supporting" the leap second in Python have made any difference?

On Sun, Jul 1, 2012 at 6:22 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 30 Jun 2012 10:23:45 -0400
> Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>
>> Even though many have hoped that the authorities would stop fiddling
>> with our clocks, today a leap second will be inserted in UTC.
>> Systems using Olson/IANA timezone database have a way to deal with
>> this without adjusting their clocks, but few systems are configured
>> that way:
>>
>> $ TZ=right/UTC date -d @1341100824
>> Sat Jun 30 23:59:60 UTC 2012
>
> If I believe the Internet, many server systems got broken by the leap
> second (*).
> Even one of our buildbots seems affected:
> - before:
> http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/419
> - after:
> http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/420
> (weird errors with timeouts, and a crash)
>
>
> (*) https://linuxfr.org/users/nono/journaux/leap-second
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
--Guido van Rossum (python.org/~guido)


From solipsis at pitrou.net  Sun Jul  1 21:43:35 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Jul 2012 21:43:35 +0200
Subject: [Python-ideas] Happy leap second
References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com>
	<20120701182233.589552b5@pitrou.net>
	<CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com>
Message-ID: <20120701214335.59220b05@pitrou.net>

On Sun, 1 Jul 2012 21:11:25 +0200
Guido van Rossum <guido at python.org> wrote:
> So would "supporting" the leap second in Python have made any difference?

No. These look like system (kernel?) bugs.

Regards

Antoine.




From ncoghlan at gmail.com  Mon Jul  2 00:14:03 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 2 Jul 2012 08:14:03 +1000
Subject: [Python-ideas] Happy leap second
In-Reply-To: <20120701214335.59220b05@pitrou.net>
References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com>
	<20120701182233.589552b5@pitrou.net>
	<CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com>
	<20120701214335.59220b05@pitrou.net>
Message-ID: <CADiSq7eME-+uy==QLGQofZ5mno=b9j_pEf8_xOPv7z9P-EWJmg@mail.gmail.com>

It was a kernel race condition (and subsequent live lock) after ntpd
notified the clock subsystem of the coming leap second (
http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second)


The joys of fine grained locking :)

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)
On Jul 2, 2012 5:45 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:

> On Sun, 1 Jul 2012 21:11:25 +0200
> Guido van Rossum <guido at python.org> wrote:
> > So would "supporting" the leap second in Python have made any difference?
>
> No. These look like system (kernel?) bugs.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120702/1c9f243a/attachment.html>

From steve at pearwood.info  Mon Jul  2 10:38:19 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Jul 2012 18:38:19 +1000
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsooav$11k$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
	<jsooav$11k$1@dough.gmane.org>
Message-ID: <20120702083819.GB27130@ando>

On Sun, Jul 01, 2012 at 01:48:14AM -0400, Terry Reedy wrote:

> >As for as expressibility goes, it is not much of an advantage. But:
> >
> >- if there are optimizations that apply to some encodings but not others,
> >   the encodable method can take advantage of them without it being a
> >   promise of the language;
> 
> It would be an optimization limited to a couple of encodings with 
> CPython. Using it for cross-version code would be something like the 
> trap of depending on the CPython optimization of repeated string 
> concatenation.

I'd hardly call it a trap. It's not like string concatenation which is 
expected to be O(N**2) on CPython but occasionally falls back to 
O(N**2). It would be O(N) expected on all platforms, but occasionally 
does better.

Perhaps an anti-trap -- sometimes it does better than expected, rather 
than worse.


> >- it only adds a single string method (and presumably a single bytes
> >   method, decodable) rather than a plethora of methods;
> 
> Decodable would always require a scan of the bytes. Might as well just 
> decode and look for UnicodeDecodeError.

*shrug* Perhaps so. bytes.decodable() would only be a LBYL convenience 
method.


> >So, I don't care much either way for a LBYL test, but if there is a good
> >use case for such a test,
> 
> My claim is that there is only a good use case if it is O(1), which 
> would only be a few cases on CPython.

*shrug* Again, I'm not exactly championing this proposal. I can see that 
an encodable method would be useful, but not that much more useful than 
trying to encode and catching the exception. A naive O(N) version of 
encodable() is trivial to implement.



-- 
Steven


From steve at pearwood.info  Mon Jul  2 10:52:24 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Jul 2012 18:52:24 +1000
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
	<jsooav$11k$1@dough.gmane.org>
	<CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com>
Message-ID: <20120702085224.GC27130@ando>

On Sun, Jul 01, 2012 at 04:27:25PM +1000, Nick Coghlan wrote:

> Rewinding back to the reasons the question is being asked, the reason
> this information is useful at the Python level is the same reason it
> is useful at the C level: it matters for finding the most efficient
> means of
> representing the text as bytes (which can then have further
> implications for the kind of quoting used, etc). The interesting
> breakpoints can actually be expressed in terms of the number of bits
> in the highest code point:
> 7 - encode as ASCII (or latin-1 or utf-8)
> 8 - encode as latin-1
> 8+ - encode as utf-8

I'm of two minds here. 

On the one hand, I question the wisdom of encouraging the use of 
anything but UTF-8. It's unfortunate enough that there are still cases 
where people have to use older encodings, without encouraging people to 
use Latin1 or ASCII in order to save a handful of bytes in a 20K email.

On the other hand, there are use-cases for non-UTF-8 encodings, and 
people will want to check whether or not a string is encodable in 
various encodings. Why make that harder/slower/less convenient than it 
need be?


> Specifically, it's a payload microoptimisation for the latin-1 case -
> the latin-1 string will be shorter than the corresponding utf-8 string

Just to be clear here, you're referring to byte strings, yes?


> (how much shorter depends on the number of non-ASCII characters). I
> believe it also makes an additional difference in the email case by
> changing the kind of quoting that is used to something with lower
> overhead that can't handle utf-8.
> 
> The "try it and see" approach suffers a potentially high speed penalty
> if the non-latin-1 characters appear late in the string:
> 
>     try:
>         # Likely no need to try ASCII, since there's no efficiency
> gain over latin-1
>         payload = message.encode("latin-1")
>     except UnicodeEncodeError:
>         payload = message.encode("utf-8")
> 
> Using max() and ord() to check in advance doesn't help, since that
> *locks in* the O(n) penalty.
> 
> The reason I think a max_code_point() method is a good potential
> solution is that it can be advertised as O(n) worst case, but
> potentially O(1) if the implementation caches the answer internally.

The downside is that the caller is then responsible for interpreting 
that value (i.e. mapping a max code point to an encoding).

The other downside is that doesn't do anything to help those who are 
stuck with legacy encodings. Although maybe that doesn't matter, since 
they will just do the "try it and see" approach.


> Another alternative would be a __max__ and __min__ protocol that
> allowed efficient answers for the max() and min() builtins. The latter
> would have the advantage of allowing other containers (like range
> objects) to provide efficient implementations.

+1 on that, although I think that should be a separate issue.


-- 
Steven



From ironfroggy at gmail.com  Mon Jul  2 12:45:43 2012
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Mon, 2 Jul 2012 06:45:43 -0400
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jsn7um$2j1$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org>
Message-ID: <CAGaVwhRAtaOmdaArg-yES+CiDAdOPgh+eHrV9XK9hwwNXag9yQ@mail.gmail.com>

On Sat, Jun 30, 2012 at 12:03 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> As shown in issue #15016 [1], there is a use cases when it is useful to
> determine that string can be encoded in ASCII or Latin1. In working with Tk
> or Windows console applications can be useful to determine that string can
> be encoded in UCS2. C API provides interface for this, but at Python level
> it is not available.
>
> I propose to add to strings class new methods: isascii(), islatin1() and
> isbmp() (in addition to such methods as isalpha() or isdigit()). The
> implementation will be trivial.
>
> Pro: The current trick with trying to encode has O(n) complexity and has
> overhead of exception raising/catching.
>
> Contra: In most cases after determining characters range we still need to
> encode a string with the appropriate encoding. New methods will complicate
> already overloaded strings class.
>
> Objections?

-1

It doesn't make sense to special case them, instead of a simpler canencode()
method added. It could save memory, but I don't see it saving time.

> [1] http://bugs.python.org/issue15016
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From storchaka at gmail.com  Mon Jul  2 13:32:50 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 02 Jul 2012 14:32:50 +0300
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <20120702085224.GC27130@ando>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
	<jsooav$11k$1@dough.gmane.org>
	<CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com>
	<20120702085224.GC27130@ando>
Message-ID: <jss0rs$6a9$1@dough.gmane.org>

On 02.07.12 11:52, Steven D'Aprano wrote:
>> Another alternative would be a __max__ and __min__ protocol that
>> allowed efficient answers for the max() and min() builtins. The latter
>> would have the advantage of allowing other containers (like range
>> objects) to provide efficient implementations.
>
> +1 on that, although I think that should be a separate issue.

This is issue #15226.



From tjreedy at udel.edu  Mon Jul  2 18:35:58 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 02 Jul 2012 12:35:58 -0400
Subject: [Python-ideas] isascii()/islatin1()/isbmp()
In-Reply-To: <jss0rs$6a9$1@dough.gmane.org>
References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info>
	<jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info>
	<jsooav$11k$1@dough.gmane.org>
	<CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com>
	<20120702085224.GC27130@ando> <jss0rs$6a9$1@dough.gmane.org>
Message-ID: <jssilh$3md$1@dough.gmane.org>

On 7/2/2012 7:32 AM, Serhiy Storchaka wrote:
> On 02.07.12 11:52, Steven D'Aprano wrote:
>>> Another alternative would be a __max__ and __min__ protocol that
>>> allowed efficient answers for the max() and min() builtins. The latter
>>> would have the advantage of allowing other containers (like range
>>> objects) to provide efficient implementations.
>>
>> +1 on that, although I think that should be a separate issue.
>
> This is issue #15226.

http://bugs.python.org/issue15226 was about exposing the max codepoint, 
which the OP thought was readily available in C, but is not. (It is at 
creation, but it is then replaced by 1 of 4 values.)

-- 
Terry Jan Reedy





From raymond.hettinger at gmail.com  Tue Jul  3 06:23:29 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 2 Jul 2012 21:23:29 -0700
Subject: [Python-ideas] [Python-Dev]  itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jspe70$4gq$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
	<2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
	<jspe70$4gq$1@dough.gmane.org>
Message-ID: <877C3C8D-1DE3-4DB0-812A-B352E8608364@gmail.com>


On Jul 1, 2012, at 5:01 AM, Stefan Behnel wrote:

> To address the main problem of users not finding what they need, what about
> simply extending the docstring of the grouper()


Here's  a small change to the docstring:  http://hg.python.org/cpython/rev/d32f21d87363

FWIW, if you're interested in load balancing applications, George Sakkis's itertools 
recipe for roundrobin() may be of interest.

Another interesting iterator technique that is not well known is the two-argument 
form of iter() which is a marvel for transforming callables into iterators:

    for block in iter(partial(f.read, 1024), ''):
           ...

    for diceroll in iter(partial(randrange(1, 7), 4):
           ...


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120702/8550f12c/attachment.html>

From techtonik at gmail.com  Wed Jul  4 11:57:20 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 4 Jul 2012 12:57:20 +0300
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <jsl3cb$knn$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
Message-ID: <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>

On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> On 26.06.2012 10:03, anatoly techtonik wrote:
>>
>> Now that Python 3 is all about iterators (which is a user killer
>> feature for Python according to StackOverflow -
>> http://stackoverflow.com/questions/tagged/python) would it be nice to
>> introduce more first class functions to work with them? One function
>> to be exact to split string into chunks.
>>
>>      itertools.chunks(iterable, size, fill=None)
>>
>> Which is the 33th most voted Python question on SO -
>>
>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>>
>> P.S. CC'ing to python-dev@ to notify about the thread in python-ideas.
>>
>
> Anatoly, so far there were no negative votes -- would you care to go
> another step and propose a patch?

Was about to say "no problem", but in fact - there is. Sorry from
whining from my side and thanks for nudging. The only thought that a
simple task of copy/pasting relevant code from
http://docs.python.org/library/itertools.html?highlight=itertools#recipes
will require a few hours waiting of download (still not everybody has
a high-speed internet) makes me switch to other less time consuming
tasks before getting around to it. These tasks become more important
in a few hours, and basically I've passed through this many times
before. It then becomes quite hard to switch back.

I absolutely don't mind someone else being credited for the idea,
because ideas usually worthless without implementation. It will be
interesting to design how the process could work in a separate thread.
For now the best thing I can do (I don't risk even to mention anything
with 3.3) is to copy/paste code from the docs here:

from itertools import izip_longest
def chunks(iterable, size, fill=None):
    """Split an iterable into blocks of fixed-length"""
    # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * size
    return izip_longest(fillvalue=fill, *args)

BTW, this doesn't work as expected (at least for strings). Expected is:
  chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx'
got:
  chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x')

Needs more round tuits definitely.


From anacrolix at gmail.com  Wed Jul  4 18:19:10 2012
From: anacrolix at gmail.com (Matt Joiner)
Date: Thu, 5 Jul 2012 00:19:10 +0800
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <20120630204852.4b898a0a@bhuda.mired.org>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
Message-ID: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>

There's no point doing this in Python since scoping is function-level
and not lexical. Just move the assignment to the line preceding the if
statement.

On Sun, Jul 1, 2012 at 8:48 AM, Mike Meyer <mwm at mired.org> wrote:
> On Sun, 1 Jul 2012 01:06:39 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> This proposal has been considered and rejected many times. It's not
>> general enough - it *only* works for those cases where the value to be
>> retained *and* the interesting condition are the same.
>>
>> Consider the simple case of a value that may be either None (not
>> interesting) or a number (interesting). Since the interesting values
>> include "0", which evaluates as False along with None, this limited
>> form of embedded assignment syntax would not help.
>
> What Nick failed to point out is that the existing uses of "as" all
> bind *specific classes of objects*, not general ones.
>
>> Embedded assignment in C isn't that limited., but nobody has yet
>> volunteered to take the radical step of proposing "(X as Y)" as a
>> general embedded assignment syntax. I suggest anyone consider such an
>> idea do a *lot* of research in the python-ideas archives first, though
>> (as the idea has seen plenty of discussion). It is not as obviously
>> flawed as the if-and-while statement only variant, but it would still
>> involve being rather persuasive to make such a significant change to
>> the language.
>
> Hasn't the BDFL rejected a general embedded assignment in any case?
>
> As a final note, reading checking the archives for your ideas will
> also give you an idea of what kinds of things are and aren't accepted
> (i.e. - what's "Pythonic") as well as process - including the kinds of
> questions you'll be asked here - that an idea goes through before
> being accepted.
>
>       <mike
> --
> Mike Meyer <mwm at mired.org>              http://www.mired.org/
> Independent Software developer/SCM consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From jeanpierreda at gmail.com  Wed Jul  4 18:35:57 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Wed, 4 Jul 2012 12:35:57 -0400
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
	<CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
Message-ID: <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com>

On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote:
> There's no point doing this in Python since scoping is function-level
> and not lexical. Just move the assignment to the line preceding the if
> statement.

I'm not sure what distinction you want to draw here between "lexical
scoping" and "function-level scoping". Is your issue that scopes
cannot be inferred during the lexing phase? Or is it something else?
Generally, AIUI, because lexical scoping is only really interesting in
contrast to static scoping, any variable whose scope can be determined
statically based on where it's defined is considered "lexically
scoped".

-- Devin


From masklinn at masklinn.net  Wed Jul  4 19:25:01 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 4 Jul 2012 19:25:01 +0200
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
	<CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
Message-ID: <2CDC2D81-0A20-4B93-B8BA-7BAD8C5D5C21@masklinn.net>

On 2012-07-04, at 18:19 , Matt Joiner wrote:

> There's no point doing this in Python since scoping is function-level
> and not lexical. Just move the assignment to the line preceding the if
> statement.

Technically, scoping is *both* lexical and function-level (as opposed
to block-based), although writing to a lexical scope is slightly more
complex than in other languages with mutable lexical scopes.


From masklinn at masklinn.net  Wed Jul  4 19:30:12 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 4 Jul 2012 19:30:12 +0200
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
	<CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
	<CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com>
Message-ID: <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net>

On 2012-07-04, at 18:35 , Devin Jeanpierre wrote:

> On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote:
>> There's no point doing this in Python since scoping is function-level
>> and not lexical. Just move the assignment to the line preceding the if
>> statement.
> 
> I'm not sure what distinction you want to draw here between "lexical
> scoping" and "function-level scoping". Is your issue that scopes
> cannot be inferred during the lexing phase? Or is it something else?

Guessing he misused "lexical scoping" for block-based scoping (as in
e.g. C) or explicit scoping constructs (e.g. `let`-type constructs).

> Generally, AIUI, because lexical scoping is only really interesting in
> contrast to static scoping

And I see you miswrite as well, "lexical scoping" and "static scoping"
are the same thing, and opposed to "dynamic scoping" (where name
resolution traverses the call stack).


From techtonik at gmail.com  Thu Jul  5 15:36:24 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 16:36:24 +0300
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <4FEF6B10.9040409@udel.edu>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
Message-ID: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>

Before anything else I must apologize for significant lags in my
replies. I can not read all of them to hold in my head, so I reply one
by one as it goes trying not to miss a single point out there. It
would be much easier to do this in unified interface for threaded
discussions, but for now there is no capabilities for that neither in
Mailman nor in GMail. And when it turns out that the amount of text is
too big, and I spend a lot of time trying to squeeze it down and then
it becomes pointless to send at all.

Now back on the topic:

On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/29/2012 4:32 PM, Georg Brandl wrote:
>>
>> On 26.06.2012 10:03, anatoly techtonik wrote:
>>>
>>> Now that Python 3 is all about iterators (which is a user killer
>>> feature for Python according to StackOverflow -
>>> http://stackoverflow.com/questions/tagged/python) would it be nice to
>>> introduce more first class functions to work with them? One function
>>> to be exact to split string into chunks.
>
> Nothing special about strings.

It seemed so, but it just appeared that grouper recipe didn't work for me.

>>>      itertools.chunks(iterable, size, fill=None)
>
> This is a renaming of itertools.grouper in 9.1.2. Itertools Recipes. You
> should have mentioned this. I think of 'blocks' rather than 'chunks', but I
> notice several SO questions with 'chunk(s)' in the title.

I guess `block` gives too low signal/noize ration in search results.
That's why it probably also called chunks in other languages, where
`block` stand for something else (I speak of Ruby blocks).

>>> Which is the 33th most voted Python question on SO -
>>>
>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>
> I am curious how you get that number. I do note that there are about 15
> other Python SO questions that seem to be variations on the theme. There
> might be more if 'blocks' and 'groups' were searched for.

It's easy:
1. Go http://stackoverflow.com/
2. Search [python]
3. Click `votes` tab
4. Choose `30 per page` at the bottom
5. Jump to the second page, there it is 4th from the top:
http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30

As for duplicates - feel free to mark them as such. SO allows
everybody to do this (unlike Roundup).

>> Anatoly, so far there were no negative votes -- would you care to go
>> another step and propose a patch?
>
> That is because Raymond H. is not reading either list right now ;-)
> Hence the Cc:. Also because I did not yet respond to a vague, very
> incomplete idea.
>
> From Raymond's first message on http://bugs.python.org/issue6021 , add
> grouper:
>
> "This has been rejected before.

I quite often see such arguments and I can't stand to repeat that
these are not arguments. It is good to know, but when people use that
as a reason to close tickets - that's just disgusting. To the
Raymond's honor he cares to explain.

> * It is not a fundamental itertool primitive.  The recipes section in
> the docs shows a clean, fast implementation derived from zip_longest().

What is the definition of 'fundamental primitive'?
To me the fact that top answer for chunking strings on SO has 2+ times
more votes than itertools versions is a clear 5 sigma indicator that
something is wrong with this Standard model without chunks boson.

> * There is some debate on a correct API for odd lengths.  Some people
> want an exception, some want fill-in values, some want truncation, and
> some want a partially filled-in tuple.  The alone is reason enough not
> to set one behavior in stone.

use case 3.1: odd lengths exception (CHOOSE ONE)
1. I see that no itertools function throws exceptions, check manually:
    len(iterable) / float(size) ==  len(iterable) // float(size)
2. Explicitly
  -  itertools.chunks(iterable, size, fill=None)
  +  itertools.chunks(iterable, size, fill=None, exception=False)

use case 3.2. fill in value. it is here (SOLVED)

use case 3.3: truncation
no itertools support truncation, do manually
   chunks(iter, size)[:len(iter)//size)

use case 4: partially filled-in tuple
  What should be there?
   >>> chunks('ABCDEFG', 3, 'x')
   >>> |


More replies and workarounds to some of the raised points are below.

> * There is an issue with having too many itertools.  The module taken as
> a whole becomes more difficult to use as new tools are added."

There can be only two reasons to that:
* chosen basis is bad (many functions that are rarely used or easily emulated)
* basis is good, but insufficient, because iterators universe is more
complicated
  than we think

> This is not to say that the question should not be re-considered. Given the
> StackOverflow experience in addition to that of the tracker and python-list
> (and maybe python-ideas), a special exception might be made in relation to
> points 1 and 3.

--[offtopic about Python enhancements / proposals feedback]--
Yes, without SO I probably wouldn't trigger this at all. Because
tracker doesn't help with raising importance - there are no votes, no
feature proposals, no "stars". And what I "like" the most is that very
"nice" resolution status - "committed/rejected" - which doesn't say
anything at all. Python list? I try not to disrupt the frequency
there. Python ideas? Too low participation level for gathering
signals. There are many people that read, support, but don't want to
reply (don't want to stand out or just lazy). There are many outside
who don't want to be subscribed at all. There are 2000+ people
spending time on Python conferences all over the world each year we
see only a couple of reactions for every Python idea here. Quite often
there are mistakes and omissions that would be nice to correct and you
can't. So StackOverflow really helps here, but it is a Q&A tool, which
is still much better than ML that are solely for chatting,
brainstorming and all the crazy reading / writing stuff. They don't
help to develop ideas collaboratively. Quite often I am just lost in
amount of text to handle.
--[/offtopic]--

> It regard to point 2: many 'proposals', including Anatoly's, neglect this
> detail. But the function has to do *something* when seqlen % grouplen != 0.
> So an 'idea' is not really a concrete programmable proposal until
> 'something' is specified.
>
> Exception -- not possible for an itertool until the end of the iteration
> (see below). To raise immediately for sequences, one could wrap grouper.
>
> def exactgrouper(sequence, k):  # untested
>   if len(sequence) % k:
>     raise ValueError('Sequence length {} must be a multiple of group length
> {}'.format(len(sequence), k)
>   else:
>     return itertools.grouper(sequence, k)

Right. Iterator is not a sequence, because it doesn't know the length
of its sequence. The method should not belong to itertools at all
then.

Python 3 is definitely become more complicated. I'd prefer to keep
separated from iterator stuff, but it seems more harder with every
iteration.

> Of course, sequences can also be directly sequentially sliced (but should
> the result be an iterable or sequence of blocks?). But we do not have a
> seqtools module and I do not think there should be another method added to
> the seq protocol.

I'd expect strings chunked into strings and lists into lists. Don't
want to know anything about protocols.

> Fill -- grouper always does this, with a default of None.
>
> Truncate, Remainder -- grouper (zip_longest) cannot directly do this and no
> recipes are given in the itertools docs. (More could be, see below.)
>
> Discussions on python-list gives various implementations either for
> sequences or iterables. For the latter, one approach is "it =
> iter(iterable)" followed by repeated islice of the first n items. Another is
> to use a sentinal for the 'fill' to detect a final incomplete block (tuple
> for grouper).
>
> def grouper_x(n, iterable):  # untested
>   sentinal = object()
>   for g in grouper(n, iterable, sentinal):
>     if g[-1] != sentinal:
>       yield g
>     else:
>       # pass to truncate
>       # yield g[:g.index(sentinal) for remainer
>       # raise ValueError for delayed exception

We need a simple function to split a sequence into chunks(). Now we
face with the problem to apply that technique to a sequence of
infinite length when a last element of infinite sequence is
encountered. You might be thinking now that this is a reduction to
absurdity. But I'd say it is an exit from the trap. Mathematically
this problem can't be solved. I am not ignoring your solution - I
think it's quite feasible, but isn't it an overcomplication?

I mean 160 people out of 149 who upvoted the question are pretty happy
with an answer that just outputs the last chunk as-is:
http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python

   chunks('ABCDEFG', 3) --> 'ABC' 'DEF' 'G'

And it is quite nice solution to me, because you're free to do
anything you'd like if you expect you data to be odd:

   for chunk in chunks('ABCDEFG', size):
     if len(chunk) < size:
       raise Tail

You can make a helper iterator out of it too.

> ---
> The above discussion of point 2 touches on point 4, which Raymond neglected
> in the particular message above but which has come up before: What are the
> allowed input and output types? An idea is not a programmable proposal until
> the domain, range, and mapping are specified.

Domain? Mapping? I am not ignoring existing knowledge and experience.
I just don't want to complicate and don't see appropriate `import
usecase` in current context, so I won't try to guess what this means.

in string -> out list of strings
in list -> out list of lists

> Possible inputs are a specific sequence (string, for instance), any
> sequence, any iterable. Possible outputs are a sequence or iterator of
> sequence or iterator. The various python-list and stackoverflow posts
> questions asks for various combinations. zip_longest and hence grouper takes
> any iterable and returns an iterator of tuples. (An iterator of maps might
> be more useful as a building block.) This is not what one usually wants with
> string input, for instance, nor with range input. To illustrate:

Allright. Got it. Sequences have a length and can be sliced with
[i:j], iterator can't be sliced (and hence no chunks can be made). So
this function doesn't belong to itertools - it is a missing string or
sequence method. We can't have a chunk with an iterator, because
iterator over a string decomposes it into a group of pieces with no
reverse function. We can have a group and then join the group into
something. But this requires the knowledge of appropriate join()
function for the iterator, and probably not efficient. As there are no
such function (must be that Mapping you referenced above) - the
recomposition into chunks is impossible.

> import itertools as it
>
> def grouper(n, iterable, fillvalue=None):
>     "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
>     args = [iter(iterable)] * n
>     return it.zip_longest(*args, fillvalue=fillvalue)
>
> print(*(grouper(3, 'ABCDEFG', 'x')))  # probably not wanted
> print(*(''.join(g) for g in grouper(3, 'ABCDEFG', 'x')))
> #
> ('A', 'B', 'C') ('D', 'E', 'F') ('G', 'x', 'x')
> ABC DEF Gxx
>
> --
> What to do? One could easily write 20 different functions. So more thought
> is needed before adding anything. -1 on the idea as is.

I've learned a new English type of argument - "straw man" (I used to
call this "hijacking"). This -1 doesn't belong to original idea. It
belongs to proposal of  itertools.chunks() with a long list of above
points and completely different user stories (i.e. not "split string
into chunks"). I hope you still +1 with 160 people on SO that think
Python needs an easy way to chunk sequences.

> For the doc, I think it would be helpful here and in most module subchapters
> if there were a subchapter table of contents at the top (under 9.1 in this
> case). Even though just 2 lines here (currently, but see below), it would
> let people know that there *is* a recipes section. After the appropriate
> tables, mention that there are example uses in the recipe section. Possibly
> add similar tables in the recipe section.

Unfortunately, it appeared that grouper() is not chunks(). It doesn't
delivers list of list of chars given string as an input instead of
list of chunks.

> Another addition could be a new subsection on grouping (chunking) that would
> discuss post-processing of grouper (as discussed above), as well as other
> recipes, including ones specific to strings and sequences. It would
> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
> Chunking Sequences and Iterables". The synonyms will help external
> searching. A toc would let people who have found this doc know to look for
> this at the bottom.

This makes matters pretty ugly. In ideal language there should be less
docs, not more.


From techtonik at gmail.com  Thu Jul  5 15:47:53 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 16:47:53 +0300
Subject: [Python-ideas] [Python-Dev]  itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jspe70$4gq$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
	<2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
	<jspe70$4gq$1@dough.gmane.org>
Message-ID: <CAPkN8xJrKgAUaLw3siTpzAjUB=eg1S08oWu-Z=7ThHqgU0eM+w@mail.gmail.com>

On Sun, Jul 1, 2012 at 3:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> To address the main problem of users not finding what they need, what about
> simply extending the docstring of the grouper() function with a sentence
> like this:
>
> "This functionality is also called 'chunking' or 'blocking' and can be used
> for load distribution and sharding."
>
> That would make it easy for users to find what they are looking for when
> they search the page for "chunk". I find that a much more common and less
> ambiguous name than "grouping", which reminds me more of "group by".

In appeared that "chunking" and "grouping" are different kind of
tasks. You can chunk a sequence (sting) by slicing it into smaller
sequences, but you can not chunk in iterable - you can only group it.

There is an data loss about the structure that occurs when a sequence
(string) becomes an iterator:
 chunks ABCDE -> AB CD E
 group ABCDE -> A B C D E -> (A B) (C D) (D E)


From techtonik at gmail.com  Thu Jul  5 16:33:19 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 17:33:19 +0300
Subject: [Python-ideas] [Python-Dev]  itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jt225u$iih$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
	<jt225u$iih$1@dough.gmane.org>
Message-ID: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>

On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/4/2012 5:57 AM, anatoly techtonik wrote:
>>
>> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:
>
>
>>> Anatoly, so far there were no negative votes -- would you care to go
>>> another step and propose a patch?
>>
>>
>> Was about to say "no problem",
>
>
> Did you read that there *are* strong negative votes? And that this idea has
> been rejected before? I summarized the objections in my two responses and
> pointed to the tracker issues. One of the objections is that there are 4
> different things one might want if the sequence length is not an even
> multiple of the chunk size. Your original 'idea' did not specify.

I actually meant that there is a problem to propose a patch in the
sense of getting checkout, working on a diff, sending it by attaching
to bug tracker as developer guide says.

>> For now the best thing I can do (I don't risk even to mention anything
>> with 3.3) is to copy/paste code from the docs here:
>>
>> from itertools import izip_longest
>> def chunks(iterable, size, fill=None):
>>      """Split an iterable into blocks of fixed-length"""
>>      # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
>>      args = [iter(iterable)] * size
>>      return izip_longest(fillvalue=fill, *args)
>
>
> Python ideas is about Python 3 ideas. Please post Python 3 code.
>
> This is actually a one liner
>
>     return zip_longest(*[iter(iterable)]*size, fillvalue=file)
>
> We don't generally add such to the stdlib.

Can you figure out from the code what this stuff does?
It doesn't give chunks of strings.

>> BTW, this doesn't work as expected (at least for strings). Expected is:
>>    chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx'
>> got:
>>    chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x')
>
>
> One of the problems with idea of 'add a chunker' is that there are at least
> a dozen variants that different people want.

That's not the problem. People always want something extra. The
problem that we don't have a real wish distribution. If 1000 people
want chunks and 1 wants groups with exception - we still account these
as equal variants.

Therefore my idea is deliberately limited to "string to chunks" user
story, and SO implementation proposal.

> I discussed the problem of
> return types issue in my responses. I showed how to get the 'expected'
> response above using grouper, but also suggested that it is the wrong basis
> for splitting strings. Repeated slicing make more sense for concrete
> sequence types.
>
> def seqchunk_odd(s, size):
>     # include odd size left over
>     for i in range(0, len(s), size):
>         yield s[i:i+size]
>
> print(list(seqchunk_odd('ABCDEFG', 3)))
> #
> ['ABC', 'DEF', 'G']

Right. That's the top answer on SO that people think should be in
stdlib. Great we are talking about the same thing actually.

> def seqchunk_even(s, size):
>     # only include even chunks
>     for i in range(0, size*(len(s)//size), size):
>         yield s[i:i+size]
>
> print(list(seqchunk_even('ABCDEFG', 3)))
> #
> ['ABC', 'DEF']

This is deducible from seqchunk_odd(s, size)

> def strchunk_fill(s, size, fill):
>     # fill odd chunks
>     q, r = divmod(len(s), size)
>     even = size * q
>     for i in range(0, even, size):
>         yield s[i:i+size]
>     if size != even:
>         yield s[even:] + fill * (size - r)
>
> print(list(strchunk_fill('ABCDEFG', 3, 'x')))
> #
> ['ABC', 'DEF', 'Gxx']

Also deducible from seqchunk_odd(s, size)

> Because the 'fill' value is necessarily a sequence for strings,
> strchunk_fill would only work for lists and tuples if the fill value were
> either required to be given as a tuple or list of length 1 or if it were
> internally converted inside the function. Skipping that for now.
>
> Having written the fill version based on the even version, it is easy to
> select among the three behaviors by modifying the fill version.
>
> def strchunk(s, size, fill=NotImplemented):
>     # fill odd chunks
>     q, r = divmod(len(s), size)
>     even = size * q
>     for i in range(0, even, size):
>         yield s[i:i+size]
>     if size != even and fill is not NotImplemented:
>         yield s[even:] + fill * (size - r)
>
> print(*strchunk('ABCDEFG', 3))
> print(*strchunk('ABCDEFG', 3, ''))
> print(*strchunk('ABCDEFG', 3, 'x'))
> #
> ABC DEF
> ABC DEF G
> ABC DEF Gxx

I now don't even think that fill value is needed as argument.
if len(chunk) < size:
  chunk.extend( [fill] * ( size - len(chunk)) )

> I already described how something similar could be done by checking each
> grouper output tuple for a fill value, but that requires that the fill value
> be a sentinal that could not otherwise appear in the tuple. One could modify
> grouper to fill with a private object() and check the last item of each
> group for that sentinal and act accordingly (delete, truncate, or replace).
> A generic api needs some thought, though.

I just need to chunk strings and sequences. Generic API is too complex
without counting all usecases and iterating over them.

> An issue I did not previously mention is that people sometimes want
> overlapping chunks rather than contiguous disjoint chunks. The slice
> approach trivially adapts to that.
>
> def seqlap(s, size):
>     for i in range(len(s)-size+1):
>         yield s[i:i+size]
>
> print(*seqlap('ABCDEFG', 3))
> #
> ABC BCD CDE DEF EFG
>
> A sliding window for a generic iterable requires a deque or ring buffer
> approach that is quite different from the zip-longest -- grouper approach.

That's why I'd like to drastically reduce the scope of proposal.
itertools doesn't seem to be the best place anymore. How about
sequence method?

   string.chunks(size)  -> ABC DEF G
   list.chunks(size) -> [A,B,C], [C,D,E],[G]

If somebody needs a keyword argument - this can come later without
breaking compatibility.


From steve at pearwood.info  Thu Jul  5 17:57:17 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 06 Jul 2012 01:57:17 +1000
Subject: [Python-ideas] [Python-Dev]  itertools.chunks(iterable, size,
 fill=None)
In-Reply-To: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>	<jsl3cb$knn$1@dough.gmane.org>	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>	<jt225u$iih$1@dough.gmane.org>
	<CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>
Message-ID: <4FF5B95D.3060208@pearwood.info>

anatoly techtonik wrote:
> On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote:

>> A sliding window for a generic iterable requires a deque or ring buffer
>> approach that is quite different from the zip-longest -- grouper approach.
> 
> That's why I'd like to drastically reduce the scope of proposal.
> itertools doesn't seem to be the best place anymore. How about
> sequence method?
> 
>    string.chunks(size)  -> ABC DEF G
>    list.chunks(size) -> [A,B,C], [C,D,E],[G]

-1

This is a fairly trivial problem to solve, and there are many variations on 
it. Many people will not find the default behaviour helpful, and will need to 
write their own. Why complicate the API for all sequence types with this?

I don't believe that we should enshrine one variation as a built-in method, 
without any evidence that it is the most useful or common variation. Even if 
there is one variation far more useful than the others, that doesn't 
necessarily mean we ought to make it a builtin method unless it is a 
fundamental sequence operation, has wide applicability, and is genuinely hard 
to write. I don't believe chunking meets *any* of those criteria, let alone 
all three.

Not every six line function needs to be a builtin.

I believe that splitting a sequence (or a string) into fixed-size chunks is 
more of a programming exercise problem than a genuinely useful tool. That does 
not mean that there is never any real use-cases for splitting into fixed-size 
chunks, only that this is the function that *seems* more useful in theory than 
it turns out in practice.

Compare this with more useful sequence/iteration tools, like (say) zip. You 
can hardly write a hundred lines of code without using zip at least once. But 
I bet you can write tens of thousands of lines of code without needing to 
split sequences into fixed chunks like this.

Besides, the name "chunks" is more general than how you are using it. For 
example, I consider chunking to be splitting a sequence up at a various 
delimiters or separators, not at fixed character positions. E.g. "the third 
word of item two of the fourth line" is a chunk.

This fits more with the non-programming use of the term chunk or chunking, and 
has precedence in Apple's Hypertalk language, which literally allowed you to 
talk about words, items and lines of text, each of which are described as chunks.

This might be a good candidate for a utility module made up of assorted useful 
functions, but not for the string and sequence APIs.



-- 
Steven



From steve at pearwood.info  Thu Jul  5 18:09:44 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 06 Jul 2012 02:09:44 +1000
Subject: [Python-ideas] [Python-Dev]  itertools.chunks(iterable, size,
 fill=None)
In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>	<jsl3cb$knn$1@dough.gmane.org>
	<4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
Message-ID: <4FF5BC48.9080302@pearwood.info>

anatoly techtonik wrote:
>>>> Which is the 33th most voted Python question on SO -
>>>>
>>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>> I am curious how you get that number. I do note that there are about 15
>> other Python SO questions that seem to be variations on the theme. There
>> might be more if 'blocks' and 'groups' were searched for.
> 
> It's easy:
> 1. Go http://stackoverflow.com/
> 2. Search [python]
> 3. Click `votes` tab
> 4. Choose `30 per page` at the bottom
> 5. Jump to the second page, there it is 4th from the top:
> http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30

Yes. I don't think this is particularly significant. Have a look at some of 
the questions with roughly the same number of votes:

#26 "How can I remove (chomp) a newline in Python?" 176 votes

#33 "How do you split a list into evenly sized chunks in Python?" 149 votes

#36 "Accessing the index in Python for loops" 144 votes


Being 33rd most voted question doesn't really mean much.


By the way, why is this discussion going to both python-dev and python-ideas?



-- 
Steven


From techtonik at gmail.com  Thu Jul  5 21:41:29 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 22:41:29 +0300
Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev]
 itertools.chunks(iterable, size, fill=None))
Message-ID: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>

On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> anatoly techtonik, 05.07.2012 15:36:
>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>>> From Raymond's first message on http://bugs.python.org/issue6021 , add
>>> grouper:
>>>
>>> "This has been rejected before.
>>
>> I quite often see such arguments and I can't stand to repeat that
>> these are not arguments. It is good to know, but when people use that
>> as a reason to close tickets - that's just disgusting.
>
> The *real* problem is that people keep bringing up topics (and even spell
> them out in the bug tracker) without searching for existing discussions
> and/or tickets first. That's why those who do such a search (or who know
> what they are talking about anyway) close these tickets with the remark
> "this has been rejected before", instead of repeating an entire heap of
> arguments all over again to feed a discussion that would only lead to the
> same result as it did before, often several times before.

Make the bloody FAQ and summarize this stuff? Why waste each others
time? If people don't enjoy repeating themselves over and over - there
is a bloody wiki. What should happen to people to start extracting
gems of knowledge from piles of dusty sheets called list "archives"
for others to admire.

No, it is easier to say "it was already discussed many times", "why
don't you Google yourself", "so far you're only complaining", etc. If
people can't find anything - why everybody thinks they are ignorant
and lazy. Even if it so, why nobody thinks that maybe that bloody
Xapian index is dead again for a bloody amount of moons nobody knows
why and how many exactly? Why nobody thinks that lazy coders can also
help with development? Maybe that laziness is the primary reason some
major groups actually prefer Python to Java, C++ and other more
interesting languages (such as PHP) when it comes to typing? Make it
easy and the patches will follow. Answers like "this was discussed
before" don't make it easy to understand, and leaving users rereading
old 19xx archives that people don't reread themselves will likely make
users bounce and never (NEVER!) come up with some proposal again. An
"organic" way to keep traffic low.

Miscommunication is a bad experience for users, bad experience for
developers, everybody is annoyed and as a result such nice language as
Python loses points on TIOBE (and convenient chunk() functions to
munch-munch on the sequence data).

Wheew. :-F


From techtonik at gmail.com  Thu Jul  5 21:55:26 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 22:55:26 +0300
Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping
Message-ID: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>

This one is practical. I am looking at NaCl SDK download page:
https://developers.google.com/native-client/sdk/download

"you need Python installed", "download SDK update utility"

What makes me sad that update utility is a Python script in a zip file
- nacl_sdk.zip
which includes shell script and a .bat file for launching this Python script.

This makes me kind of sad. You have Python installed. Why can't you
just crossplatformly do:

  mkdir nacl
  cd nacl
  python -m urllib get
http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py
  python update_sdk.py


From amauryfa at gmail.com  Thu Jul  5 22:24:27 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 5 Jul 2012 22:24:27 +0200
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
Message-ID: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>

2012/7/5 anatoly techtonik <techtonik at gmail.com>:
> This makes me kind of sad. You have Python installed. Why can't you
> just crossplatformly do:
>
>   mkdir nacl
>   cd nacl
>   python -m urllib get
> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py
>   python update_sdk.py

I'm sure there is already a way with standard python tools. Something
along these lines:

python -c "from urllib.request import urlretrieve; urlretrieve('URL',
'update_sdk.zip')"
python -m update_sdk.zip

The second command will work if the zip file has a __main__.py.
Do you think we need other tools?

-- 
Amaury Forgeot d'Arc


From simon.sapin at kozea.fr  Thu Jul  5 22:15:10 2012
From: simon.sapin at kozea.fr (Simon Sapin)
Date: Thu, 05 Jul 2012 22:15:10 +0200
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
Message-ID: <4FF5F5CE.8020701@kozea.fr>

Le 05/07/2012 21:55, anatoly techtonik a ?crit :
> This one is practical. I am looking at NaCl SDK download page:
> https://developers.google.com/native-client/sdk/download
>
> "you need Python installed", "download SDK update utility"
>
> What makes me sad that update utility is a Python script in a zip file
> - nacl_sdk.zip
> which includes shell script and a .bat file for launching this Python script.
>
> This makes me kind of sad. You have Python installed. Why can't you
> just crossplatformly do:
>
>    mkdir nacl
>    cd nacl
>    python -m urllib get
> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py
>    python update_sdk.py

Hi,

"Sadness" drama aside, what?s your point? Is this a proposal to add a 
command-line API for urlretrieve()?

In that case, +1. (For what my vote is worth.)
The use cases are limited (just use the Python API) but they do exist, 
as you showed.

Regards,
-- 
Simon Sapin




From ned at nedbatchelder.com  Thu Jul  5 23:07:14 2012
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 05 Jul 2012 17:07:14 -0400
Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev]
 itertools.chunks(iterable, size, fill=None))
In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
Message-ID: <4FF60202.1040909@nedbatchelder.com>


On 7/5/2012 3:41 PM, anatoly techtonik wrote:
> Make the bloody FAQ and summarize this stuff? Why waste each others
> time? If people don't enjoy repeating themselves over and over - there
> is a bloody wiki. What should happen to people to start extracting
> gems of knowledge from piles of dusty sheets called list "archives"
> for others to admire.
>
> No, it is easier to say "it was already discussed many times", "why
> don't you Google yourself", "so far you're only complaining", etc. If
> people can't find anything - why everybody thinks they are ignorant
> and lazy. Even if it so, why nobody thinks that maybe that bloody
> Xapian index is dead again for a bloody amount of moons nobody knows
> why and how many exactly? Why nobody thinks that lazy coders can also
> help with development? Maybe that laziness is the primary reason some
> major groups actually prefer Python to Java, C++ and other more
> interesting languages (such as PHP) when it comes to typing? Make it
> easy and the patches will follow. Answers like "this was discussed
> before" don't make it easy to understand, and leaving users rereading
> old 19xx archives that people don't reread themselves will likely make
> users bounce and never (NEVER!) come up with some proposal again. An
> "organic" way to keep traffic low.
>
> Miscommunication is a bad experience for users, bad experience for
> developers, everybody is annoyed and as a result such nice language as
> Python loses points on TIOBE (and convenient chunk() functions to
> munch-munch on the sequence data).
Anatoly, miscommunication is a bad experience, yes.  This is an 
excellent example of it.  I honestly have no idea what you are 
advocating in this message, partly because sarcasm doesn't work well.  
You seem really upset, but I for one don't know what it is you want.

--Ned.

> Wheew. :-F
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>




From wuwei23 at gmail.com  Fri Jul  6 07:05:04 2012
From: wuwei23 at gmail.com (alex23)
Date: Thu, 5 Jul 2012 22:05:04 -0700 (PDT)
Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev]
	itertools.chunks(iterable, size, fill=None))
In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
Message-ID: <05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com>

On Jul 6, 5:41?am, anatoly techtonik <techto... at gmail.com> wrote:
> Why nobody thinks that lazy coders can also
> help with development? Maybe that laziness is the primary reason some
> major groups actually prefer Python to Java, C++ and other more
> interesting languages (such as PHP) when it comes to typing?

You make this criticism fairly regularly and yet you yourself
constantly make suggestions and refuse to provide implementations. I
have a hard time reconciling the implicit anger in this statement:

> Why waste each others time? If people don't enjoy repeating themselves over
> and over - there is a bloody wiki.

...with ones you have previously made defending your _own_
unwillingness to examine pre-existing discussion:

> It's too boring to live in a world of existing knowledge and expertise

You yourself feel you have no responsibility in understanding a
situation before demanding that it be raised for discussion. I'm not
sure _what_ others could put into place that would _make_ you perform
this action.

> Answers like "this was discussed
> before" don't make it easy to understand, and leaving users rereading
> old 19xx archives that people don't reread themselves will likely make
> users bounce and never (NEVER!) come up with some proposal again.

Once again you're putting the burden on _others_ to somehow come up
with mechanisms to force people to do the due diligence they should be
capable of doing as a programmer.

Frankly, I'm of the opinion that if someone cannot be bothered putting
the effort into understanding the pre-existing content for a proposal
they're making, then yes, they simply should not come up with that
proposal. I'm not entirely sure why you feel people should be entitled
to request far more than they're willing to contribute.


From wuwei23 at gmail.com  Fri Jul  6 07:08:09 2012
From: wuwei23 at gmail.com (alex23)
Date: Thu, 5 Jul 2012 22:08:09 -0700 (PDT)
Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev]
	itertools.chunks(iterable, size, fill=None))
In-Reply-To: <05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com>
References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
	<05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com>
Message-ID: <55df23be-bfb5-40f7-aa51-6b7e89c129e3@m2g2000pbv.googlegroups.com>

On Jul 6, 3:05?pm, alex23 <wuwe... at gmail.com> wrote:
>  the pre-existing content

That should actually be "context".


From stephen at xemacs.org  Fri Jul  6 07:26:57 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 06 Jul 2012 14:26:57 +0900
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
Message-ID: <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>

Annoying cross-post trimmed.

anatoly techtonik writes:

 > Before anything else I must apologize for significant lags in my
 > replies.

No, that's nothing you need to apologize for.  Taking time to
formulate a decent reply is common courtesy, and commendable.  We're
really *not* in a hurry here; Python is going to be around for a few
more decades at least.

What you need to apologize for is the major faux pas of cross-posting.
Please cut it out, and in particular trim your own address list when
replying, or use Reply-To (and maybe Mail-Followup-To) to redirect
others' replies to an appropriate list.  IMO, the right list for this
discussion is python-ideas, but at the very least choose one.

and again he writes:

 > Make the bloody FAQ and summarize this stuff? Why waste each others
 > time? If people don't enjoy repeating themselves over and over -
 > there is a bloody wiki. What should happen to people to start
 > extracting gems of knowledge from piles of dusty sheets called list
 > "archives" for others to admire.

This is a terrible idea for python-dev and python-ideas.  While it is
frustrating to get a "been there, done that, rejected with extreme
prejudice" reply, and there's no question that searching the archives
is a hit-and-mostly-miss kind of thing because of the difficulty of
choosing good search terms, it's really not that costly to come back
with "I'm sorry, I couldn't find the thread".  Rather than spend
effort on writing a FAQ that would rather quickly turn into a
monstrosity hardly more easy to search than the archives themselves,
and almost never be read, we should devote any effort to improving the
capability for searching archives (and the wiki and the issue
tracker).

The problem with trying to put everything into the FAQ is that it's a
terribly unrewarding thing to do.  Most of the stuff you will write
will be ignored and profit nobody.  It really needs to be selected, by
somebody with taste and knowledge of user needs.  Presumably O'Reilly
or somebody has a "Python Hacks" book -- go buy it, which will
encourage the author to keep up the good work.  If they don't, why
don't you propose it to them, and write it yourself?  Then you can
measure how good an idea it is by your royalties.

Or if you don't feel like writing it yourself, maybe you can convince
O'Reilly to come up with a big enough advance to interest somebody
like Terry Reedy or Steven d'Aprano or even Raymond Hettinger or David
Beazley.

Regards,


From mwm at mired.org  Fri Jul  6 08:00:30 2012
From: mwm at mired.org (Mike Meyer)
Date: Fri, 06 Jul 2012 02:00:30 -0400
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
	<87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com>

"Stephen J. Turnbull" <stephen at xemacs.org> wrote:
>anatoly techtonik writes:
> > Make the bloody FAQ and summarize this stuff? Why waste each others
> > time? If people don't enjoy repeating themselves over and over -
> > there is a bloody wiki. What should happen to people to start
> > extracting gems of knowledge from piles of dusty sheets called list
> > "archives" for others to admire.
>
>This is a terrible idea for python-dev and python-ideas.  While it is
>frustrating to get a "been there, done that, rejected with extreme
>prejudice" reply, and there's no question that searching the archives
>is a hit-and-mostly-miss kind of thing because of the difficulty of
>choosing good search terms, it's really not that costly to come back
>with "I'm sorry, I couldn't find the thread".  Rather than spend
>effort on writing a FAQ that would rather quickly turn into a
>monstrosity hardly more easy to search than the archives themselves,
>and almost never be read, we should devote any effort to improving the
>capability for searching archives (and the wiki and the issue
>tracker).

Python-dev i'm not familiar enough with to formulate an opinion, but I think it could work well for python-ideas.

While better search tools always help, a mail list is mostly unorganized, and a wiki need a purpose (possibly provided by a person or small group of people who take on the responsibility of maintaining the thing), or it becomes an even less organized mess.

A FAQ, on the other hand, *has* a purpose. It also has a natural source of new material in the list. Python.org already has multiple FAQs with an integrated search engine (that are hopefully integrated into the python.org search engine), and last time I looked (admitadly long ago), a tool for maintaining FAQs. While I don't think I'm quaified to write answer for python-ideas faqs, I believe I can decide whether or not something is appropriate as a python-ideas FAQ entry. If the tool for maintinaing them isn't a memory error and is still in use, I'd certainly be willing to do that.

But that's the easy job. The trick is getting the list members to play along. If they provide what is most people consider the definitive answer a couple of times, then the next time the question comes up, write it up as a FAQ entry and submit it as such as well to the list. The time after that, just provide a pointer to the FAQ entry.

This worked very well for the FreeBSD FAQ, which is a fairly large document. They didn't even have a spiffy tool for maintaining it, but had to take submissions and add or fix markup and then check them in to the document tree.

-- 
Sent from my Android tablet. Please excuse my swyping.


From stephen at xemacs.org  Fri Jul  6 11:22:03 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 06 Jul 2012 18:22:03 +0900
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
	<87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com>
Message-ID: <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > Python-dev i'm not familiar enough with to formulate an opinion,
 > but I think it could work well for python-ideas.

You're missing the point.  Improving the FAQ for python-*list* is a
great idea, but Mr. techtonik proposes a FAQ for python-*ideas*.  IOW,
if the question is actually on-topic, the answers are reasons why
something isn't going to ever be in Python.

True, often the explanation of why a proposed feature is inappropriate
is of the form "this three-line function does the job", but if the
question is really a FAQ, it presumably has been asked multiple times
on python-list, and should be picked up there.

 > This worked very well for the FreeBSD FAQ, which is a fairly large
 > document.

No, it didn't.  The FreeBSD FAQ is quite obviously not oriented to
telling developers what isn't going to go into the next version of
FreeBSD and why not.

If people want to move ahead with this discussion, it really ought to
move to python-list IMO.



From mwm at mired.org  Fri Jul  6 17:38:42 2012
From: mwm at mired.org (Mike Meyer)
Date: Fri, 6 Jul 2012 11:38:42 -0400
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
	<87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com>
	<8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20120706113842.28e7a3ae@bhuda.mired.org>

On Fri, 06 Jul 2012 18:22:03 +0900
"Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> Mike Meyer writes:
>  > Python-dev i'm not familiar enough with to formulate an opinion,
>  > but I think it could work well for python-ideas.
> You're missing the point.  Improving the FAQ for python-*list* is a
> great idea, but Mr. techtonik proposes a FAQ for python-*ideas*.  IOW,
> if the question is actually on-topic, the answers are reasons why
> something isn't going to ever be in Python.

That point I got.

> True, often the explanation of why a proposed feature is inappropriate
> is of the form "this three-line function does the job", but if the
> question is really a FAQ, it presumably has been asked multiple times
> on python-list, and should be picked up there.

That I didn't, and it's fair enough. But the onus to fix this *still*
falls on the people who are answering the same question over and
over. They need to start submitting the answers to the FAQ (there is a
history & design FAQ that would seem to be an appropriate place for
them) rather than just to the list.

>  > This worked very well for the FreeBSD FAQ, which is a fairly large
>  > document.
> No, it didn't.  The FreeBSD FAQ is quite obviously not oriented to
> telling developers what isn't going to go into the next version of
> FreeBSD and why not.

If you're claiming that the FreeBSD FAQ technique didn't work well for
it, then I disagree. That the FreeBSD FAQ has a different orientation
is true, but not clearly relevant.

> If people want to move ahead with this discussion, it really ought to
> move to python-list IMO.

Not in mine. The problem is repeated questions on -ideas. My proposed
solution has to be implemented by -ideas readers. They need to start
submitting frequently typed answers to the appropriate FAQ, not just
to the list.

   <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From jae+python at jaerhard.com  Fri Jul  6 20:14:04 2012
From: jae+python at jaerhard.com (=?iso-8859-1?B?SvxyZ2VuIEEu?= Erhard)
Date: Fri, 6 Jul 2012 20:14:04 +0200
Subject: [Python-ideas] itertools recipes: why not add them to the stdlib
	*somewhere*?
Message-ID: <20120706181404.GB3347@jaerhard.com>

I just now ran into (since learning about the recipes) having use for
a recipe (then I found out that I can't use it anyway, but that's a
different thing)

So... I consider it very strange and annoying that there's this code.
Ready to use.  But you have to copy-and-paste it into your code
*somewhere* instead of easily importing it.

As to the "it makes using Python harder to learn", I beg to
differ: an addition to the *language* (like, say, metaclasses) *can*
make it more complicated.  But additions to the stdlib?  What about
"batteries included"?  Not a motto anymore (I heard rumors, so maybe
that's the case)

Putting these in some official(!) iterutils (or name it what you want)
package would be a solution for so many people.  Yes, not everyone
needs grouper's current functionality.  But for the many who do, it'd
be there already.  And someone thought those recipes are useful,
didn't you?  So why not make them more easily available?

And then there's Alex Martelli at
http://stackoverflow.com/a/1641242/238406, and I agree with him
wholeheartedly.

Bye, J


From stephen at xemacs.org  Fri Jul  6 20:53:31 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 07 Jul 2012 03:53:31 +0900
Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None)
In-Reply-To: <20120706113842.28e7a3ae@bhuda.mired.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
	<87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp>
	<1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com>
	<8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20120706113842.28e7a3ae@bhuda.mired.org>
Message-ID: <87txxk67ro.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > That I didn't, and it's fair enough. But the onus to fix this *still*
 > falls on the people who are answering the same question over and
 > over. They need to start submitting the answers to the FAQ (there is a
 > history & design FAQ that would seem to be an appropriate place for
 > them) rather than just to the list.

I disagree that it's appropriate.  The History & Design FAQ isn't
intended to be encyclopedic, and I don't see that making it so would
be useful.

The point is that if "This has been rejected before" is sufficient
answer, the proponent withdraws, everybody happy.  And if not, they
have to go to the archives anyway. They need to answer the defects
brought up in the list discussion, point by point, or they're not
going to get a hearing.

 > If you're claiming that the FreeBSD FAQ technique didn't work well
 > for it, then I disagree.

No, I'm claiming that AFAICS the FreeBSD FAQ didn't even try to fix
the problem that you and anatoly percieve.

 > The problem is repeated questions on -ideas.

Indeed!<wink/>  But I think the solution is to encourage asking those
questions on python-list first.[1]  The experts there are both willing
to answer such questions, and have a better sense of FAQ-iness
(relatively few users interested in the question will propose a
general solution).  And they have an incentive to document answers if
they notice a FAQ.  If no suitable idiom appears, then work up a
change proposal and post to -ideas.

Footnotes: 
[1]  Yes, I'm being mildly ironic; people typically won't know it's a
FAQ until they ask.  Nevertheless, asking on python-list before
posting to either python-ideas or python-dev should be encouraged
("unless you're Dutch"<wink/>).



From g.brandl at gmx.net  Fri Jul  6 21:30:41 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 06 Jul 2012 21:30:41 +0200
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
	<CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
Message-ID: <jt7ebl$duo$2@dough.gmane.org>

On 05.07.2012 22:24, Amaury Forgeot d'Arc wrote:
> 2012/7/5 anatoly techtonik <techtonik at gmail.com>:
>> This makes me kind of sad. You have Python installed. Why can't you
>> just crossplatformly do:
>>
>>   mkdir nacl
>>   cd nacl
>>   python -m urllib get
>> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py
>>   python update_sdk.py
>
> I'm sure there is already a way with standard python tools. Something
> along these lines:
>
> python -c "from urllib.request import urlretrieve; urlretrieve('URL',
> 'update_sdk.zip')"
> python -m update_sdk.zip
>
> The second command will work if the zip file has a __main__.py.
> Do you think we need other tools?

The "python -m urllib" (don't think "get" is required) interface certainly
looks nice and is similar in style with many of the other __main__ stuff we
add to stdlib modules.

Georg



From anacrolix at gmail.com  Sat Jul  7 18:03:15 2012
From: anacrolix at gmail.com (Matt Joiner)
Date: Sun, 8 Jul 2012 00:03:15 +0800
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
	<CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
	<CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com>
	<09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net>
Message-ID: <CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com>

Sorry yes I meant block scoping. I assimilated terms from further up
in the thread I think.

I mean that there can be no temporary/block/localized scope for a name
to within the if statement, so there's no reason not to put it on the
line above. FWIW I've grown fond of function level scoping, it really
reduces confusion in closures and encourages smaller functions to
avoid name leaks.

Go has something like the OP asks for:

if x := SomeFunc(); x > 3 {
    // x scoped only within this block
}

Haskell also lets  you name expressions like:

someFunc y at x:xs = ...

But in Python I still think this is best:

x = some_func()
if x > 3:
    # no scoping :)

On Thu, Jul 5, 2012 at 1:30 AM, Masklinn <masklinn at masklinn.net> wrote:
> On 2012-07-04, at 18:35 , Devin Jeanpierre wrote:
>
>> On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote:
>>> There's no point doing this in Python since scoping is function-level
>>> and not lexical. Just move the assignment to the line preceding the if
>>> statement.
>>
>> I'm not sure what distinction you want to draw here between "lexical
>> scoping" and "function-level scoping". Is your issue that scopes
>> cannot be inferred during the lexing phase? Or is it something else?
>
> Guessing he misused "lexical scoping" for block-based scoping (as in
> e.g. C) or explicit scoping constructs (e.g. `let`-type constructs).
>
>> Generally, AIUI, because lexical scoping is only really interesting in
>> contrast to static scoping
>
> And I see you miswrite as well, "lexical scoping" and "static scoping"
> are the same thing, and opposed to "dynamic scoping" (where name
> resolution traverses the call stack).
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From masklinn at masklinn.net  Sat Jul  7 18:15:35 2012
From: masklinn at masklinn.net (Masklinn)
Date: Sat, 7 Jul 2012 18:15:35 +0200
Subject: [Python-ideas] the optional "as" statement inside "if"
	statements
In-Reply-To: <CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com>
References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com>
	<CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com>
	<20120630204852.4b898a0a@bhuda.mired.org>
	<CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com>
	<CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com>
	<09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net>
	<CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com>
Message-ID: <24E42BCE-2875-4DFB-9E60-1B8A1CF9A7AA@masklinn.net>


On 2012-07-07, at 18:03 , Matt Joiner wrote:
> 
> Haskell also lets  you name expressions like:
> 
> someFunc y at x:xs = ...

as-patterns have very little relation with what was described in the
thread, as they're about keeping the "packed" version of a
pattern-matched expression.

Guards are probably closer:

    case someFunc val of
        x | x > 3 -> ?



From christopherreay at gmail.com  Sat Jul  7 20:07:02 2012
From: christopherreay at gmail.com (Christopher Reay)
Date: Sat, 7 Jul 2012 19:07:02 +0100
Subject: [Python-ideas] Explanation on how to search the archives
Message-ID: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>

I have heard two suggestions on how to search the achives

one is:

If you have time to post and discuss an idea, you have time to search
gmane.comp.python.ideas at search.gmane.org and perhaps find out why not to
bother posting.

And the Other:

It's more a matter of working out how to point Google (or the search
engine of your choice) at the archives in a useful way. In this case:
https://www.google.com/search?q=inurl%3Apython-ideas%20site%3Amail.python.org%20embedded%20assignment

This is great information.

Is there a page suggesting how to search archives effectively?
Whilst much of that skill is generic, and clear concise explanation of what
is expected from the polite user of the mailing list (and other Python
mailing lists) could help.

A link to that page could be put in the footer of the python mailing list
emails.. and voila, a very easy thing to reference and get righteous about
people not having read.

What do you think?

Christopher

-- 

Be prepared to have your predictions come true
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120707/2bc25a93/attachment.html>

From steve at pearwood.info  Sat Jul  7 21:01:59 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 08 Jul 2012 05:01:59 +1000
Subject: [Python-ideas] Explanation on how to search the archives
In-Reply-To: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>
References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>
Message-ID: <4FF887A7.5070101@pearwood.info>

Christopher Reay wrote:

> Is there a page suggesting how to search archives effectively?
[...]
> What do you think?

I think that is an excellent idea for some person or persons to write up on 
their blogs or other websites. There's no need to wait for permission, or for 
The One True Official Page -- anyone can write up a page and link to it.

There is even a python.org wiki, sadly under-utilized in my opinion. Anyone 
with an account who wants to start a page there can.

Not every concept vaguely related to programming in Python falls under the 
responsibility of python.org. Teaching people how to search the Internet and 
interact with mailing lists without being annoying is not the job of the 
Python devs, but it's a good way for people to help out.


-- 
Steven





From mikegraham at gmail.com  Sun Jul  8 22:22:43 2012
From: mikegraham at gmail.com (Mike Graham)
Date: Sun, 8 Jul 2012 16:22:43 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
Message-ID: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>

A common stumbling block for new users is writing decorators that take
arguments. To create a decorator like

@timesn(n)
def f(y):
    ...

We write code like

def timesn(n)
    def decorator(f):
        def inner(y):
            return n * f(y)
        return inner
     return decorator

which confuses many users and can be a handful to type. I wonder if it
would be clearer for people to write

def timesn(n)(f):
    def inner(y):
        return n * f(y)
    return inner

which is more concise and looks a lot more like a non-parametric
decorator someone might have written already. The syntax is mostly
self-explaining and could potentially be useful in other contexts.

There exist tools like the decorator library to try to simplify this
already, but in my experience they mostly serve to confuse people
using decorators for the first time more.

One thing I didn't specify was whether `n` was nonlocal or not and the
behavior of something that keeps and reuses timesn(some_specific_n)
multiple times.

Does anyone think a feature like this may be useful?

Regards,
Mike


From jeanpierreda at gmail.com  Sun Jul  8 22:41:02 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 8 Jul 2012 16:41:02 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <CABicbJJ7dWNC1sAx=rqqBmHVmG-VbfXNh7H05z+zFete41iBTg@mail.gmail.com>

On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote:
> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.

Ooh, +1 Semantically the function and the decorator-params are both
"arguments" to the decorator, and this confuses a lot of people into
writing silly things like def decorator(arg1, arg2, f): ...

In actuality, these two are equivalent, and this new def syntax reflects that:

@decorator(arg):
def foo(...):
    ...

def foo(...):
    ...
foo = decorator(arg)(foo)

As opposed to the usual definition syntax, which is not really intuitive.

> One thing I didn't specify was whether `n` was nonlocal or not and the
> behavior of something that keeps and reuses timesn(some_specific_n)
> multiple times.

For the purposes of decorators I don't think it matters. I would guess
that keeping and reusing the first arg is more useful for
non-decorator purposes, and making them a nonlocal is more consistent
with the rest of Python, if we reuse the argument. Plus it'd align
better semantically with the old way of defining parametric
decorators.

-- Devin


From solipsis at pitrou.net  Sun Jul  8 23:52:46 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 8 Jul 2012 23:52:46 +0200
Subject: [Python-ideas] Syntax for defining parametric decorators
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <20120708235246.7eaa54e8@pitrou.net>

On Sun, 8 Jul 2012 16:22:43 -0400
Mike Graham <mikegraham at gmail.com> wrote:
> 
> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
> 
> def timesn(n)(f):
>     def inner(y):
>         return n * f(y)
>     return inner

But then, why not:

def timesn(n)(f)(y):
    return n * f(y)

?

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From ncoghlan at gmail.com  Mon Jul  9 00:54:07 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 08:54:07 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>

Step one is to stop referring to decorator factories as decorators.

Beyond that, no, this is too limited - it only helps when there's no extra
code in the outer scope which, in my experience, is the exception rather
than the rule. Closures are an advanced programming concept - there are
limits to how simple they are ever going to be.

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/1fb07743/attachment.html>

From grosser.meister.morti at gmx.net  Mon Jul  9 01:13:40 2012
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Mon, 09 Jul 2012 01:13:40 +0200
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <4FFA1424.30009@gmx.net>

On 07/08/2012 10:22 PM, Mike Graham wrote:
> A common stumbling block for new users is writing decorators that take
> arguments. To create a decorator like
>
> @timesn(n)
> def f(y):
>      ...
>
> We write code like
>
> def timesn(n)
>      def decorator(f):
>          def inner(y):
>              return n * f(y)
>          return inner
>       return decorator
>
> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
>
> def timesn(n)(f):
>      def inner(y):
>          return n * f(y)
>      return inner
>

Why not write this?:

def timesn(n)(f)(y):
     return n * f(y)

This would be a currified function. One could implement something like that like this:

def curry(f):
	for i in range(f.func_code.co_argcount-1):
		f = (lambda f: lambda *args: partial(f,*args))(f)
	return f

@curry
def timesn(n,f,y):
	return n * f(y)

> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.
>
> There exist tools like the decorator library to try to simplify this
> already, but in my experience they mostly serve to confuse people
> using decorators for the first time more.
>
> One thing I didn't specify was whether `n` was nonlocal or not and the
> behavior of something that keeps and reuses timesn(some_specific_n)
> multiple times.
>
> Does anyone think a feature like this may be useful?
>
> Regards,
> Mike


From jeanpierreda at gmail.com  Mon Jul  9 01:47:07 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 8 Jul 2012 19:47:07 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <4FFA1424.30009@gmx.net>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<4FFA1424.30009@gmx.net>
Message-ID: <CABicbJJBzqE_oBOhdb8R0S2N-uBBw7AkuuaeRQVdCgpce-rmUg@mail.gmail.com>

On Sun, Jul 8, 2012 at 7:13 PM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> This would be a currified function. One could implement something like that
> like this:
>
> def curry(f):
>         for i in range(f.func_code.co_argcount-1):
>                 f = (lambda f: lambda *args: partial(f,*args))(f)
>         return f
>
> @curry
> def timesn(n,f,y):
>         return n * f(y)

But then how do you do:

def linearcomposition(m, c=0)(f)(x):
    return m * f(x) + c

Python isn't Haskell, so we can't really currify every argument and
expect things to work out, esp. because of variadic functions
(defaults or even *args and **kwargs).

-- Devin


From steve at pearwood.info  Mon Jul  9 02:52:15 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 09 Jul 2012 10:52:15 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <4FFA2B3F.5010106@pearwood.info>

Mike Graham wrote:
> A common stumbling block for new users is writing decorators that take
> arguments. To create a decorator like

I question that assertion. Have you spent much time on the tutor at python.org 
mailing list? I have, and I can say that decorators is not something that many 
new users there are concerned about *at all*, let alone decorators which take 
arguments.

I believe that decorator factories, and closures in general, are a moderately 
advanced technique, and not a stumbling block for most new users. I can't say 
I remember the last time I've seen anyone ask for help writing decorator 
factories on either the tutor or python-list mailing list.

So I don't believe that making new syntax is neither needed or desirable.
-1 on the idea.

Some more issues/problems with the idea:


> We write code like
> 
> def timesn(n)
>     def decorator(f):
>         def inner(y):
>             return n * f(y)
>         return inner
>      return decorator


In my experience, most decorator factories (or factory-factories in general, 
not just for generators but any time you need to wrap a factory function in a 
closure) require at least one pre-processing step, and occasionally one or two 
post-processing steps as well:


def timesn(n)
     pre_process()
     def decorator(f):
         pre_process()
         @functools.wraps(f)
         def inner(y):
             return n * f(y)
         post_process()
         return inner
     post_process()
     return decorator

With your suggested syntax, you lose a scope, and it is unclear to me what 
this would do:


def timesn(n)(f):
     pre_process()
     @functools.wraps(f)
     def inner(f):
         return n * f(y)
     post_process()
     return inner


> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
> 
> def timesn(n)(f):
>     def inner(y):
>         return n * f(y)
>     return inner
> 
> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.

Concise, yes, but I disagree that it is self-explaining.

I'm having a lot of difficulty in thinking of how I would explain it to even a 
moderately experienced user except by expanding it out to the explicit nested 
function form. I'm not sure I can even explain it to myself.

What will happen when the user invariably writes something like this?

def ordinary(a)(b):
     return a + b


My guess is that they will get a syntax error, but it will be a syntax error 
in the wrong place: instead of clearly flagging the error (a)(b) as invalid 
syntax, as you get now

py> def ordinary(a)(b):
   File "<stdin>", line 1
     def ordinary(a)(b):
                    ^
SyntaxError: invalid syntax


Instead, the user will get a less useful syntax error on the following line, 
where the compiler sees that the function def is not followed immediately by 
another function def.


py> def ordinary(a)(b):
...     return a + b
   File "<stdin>", line 1
     return a + b
     ^
SyntaxError: invalid syntax


So whatever clarity you (allegedly) gain when writing decorator factories, you 
lose when making an error.



> There exist tools like the decorator library to try to simplify this
> already, but in my experience they mostly serve to confuse people
> using decorators for the first time more.

In my opinion, that's because the decorator and decorator factory concept are 
already as simple as they can possibly be. Anything you do to them adds 
complexity, not reduces it.

I mean, it's a simple concept: a decorator is a kind of function which returns 
a function. If you put that inside a third function, you have a function which 
returns a function which returns a function. Any syntax to disguise that 
simplicity is only going to make things more complex, not less.


I haven't given any thought to how this will effect class decorators, but I 
suspect it will run into the same problems.


-- 
Steven



From mikegraham at gmail.com  Mon Jul  9 03:11:00 2012
From: mikegraham at gmail.com (Mike Graham)
Date: Sun, 8 Jul 2012 21:11:00 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
Message-ID: <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>

On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Beyond that, no, this is too limited - it only helps when there's no extra
> code in the outer scope which, in my experience, is the exception rather
> than the rule.

I'm not sure I think that's the case. functools.wraps,
functools.lru_cache, and reprlib.recursive_repr are the only stdlib
decorator factories I could think of, and all of them could use this
syntax. So could the ones I could quickly find in my own code.

Mike


From mikegraham at gmail.com  Mon Jul  9 03:27:51 2012
From: mikegraham at gmail.com (Mike Graham)
Date: Sun, 8 Jul 2012 21:27:51 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <4FFA2B3F.5010106@pearwood.info>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<4FFA2B3F.5010106@pearwood.info>
Message-ID: <CAEBZo3Pq64GA2=1qxsZSg5-JRa7dG-hQh6Rz6WvLwKEQA_SrdQ@mail.gmail.com>

On Sun, Jul 8, 2012 at 8:52 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Mike Graham wrote:
>>
>> A common stumbling block for new users is writing decorators that take
>> arguments. To create a decorator like
>
> I question that assertion. Have you spent much time on the tutor at python.org
> mailing list? I have, and I can say that decorators is not something that
> many new users there are concerned about *at all*, let alone decorators
> which take arguments.

I haven't, but I have spend considerable time helping people learn
Python. I'm a gold-badge Python answerer on Stack Overflow, an
op/regular on the #python IRC channel on freenode, and have taught
people in the meatosphere. Decorators is certainly a topic that
interests some people as they're still getting to know Python, and my
most recent Stack Overflow question was from someone confused about
some code using decorators and as recently as this morning I saw
someone on #python say they "wanted to learn about decorators" in the
abstract. In any event, I didn't mean _people completely brand new to
Python_ when I talked about people who were new.

> In my experience, most decorator factories (or factory-factories in general,
> not just for generators but any time you need to wrap a factory function in
> a closure) require at least one pre-processing step, and occasionally one or
> two post-processing steps as well:

In surveying actual decorator factories in the stdlib and my own code,
I didn't notice it to be the case there was any preprocessing inside
the outermost function. Maybe I just found the wrong ones.

> Concise, yes, but I disagree that it is self-explaining.
>
> I'm having a lot of difficulty in thinking of how I would explain it to even
> a moderately experienced user except by expanding it out to the explicit
> nested function form. I'm not sure I can even explain it to myself.
>
> What will happen when the user invariably writes something like this?
>
> def ordinary(a)(b):
>     return a + b
>
>
> My guess is that they will get a syntax error, but it will be a syntax error
> in the wrong place: instead of clearly flagging the error (a)(b) as invalid
> syntax, as you get now
>
> py> def ordinary(a)(b):
>   File "<stdin>", line 1
>     def ordinary(a)(b):
>                    ^
> SyntaxError: invalid syntax
>
>
> Instead, the user will get a less useful syntax error on the following line,
> where the compiler sees that the function def is not followed immediately by
> another function def.
>
>
> py> def ordinary(a)(b):
> ...     return a + b
>   File "<stdin>", line 1
>     return a + b
>     ^
> SyntaxError: invalid syntax
>
>
> So whatever clarity you (allegedly) gain when writing decorator factories,
> you lose when making an error.

I'm glad to get a different perspective here. It was evidently much
more self-explanatory to the folks I discussed this with before
posting.

I wouldn't have expected this to require the def to be immediately
followed by another def or for that to be the only (or most-common)
case. I would expect that to be proper syntax under this idea, and for
ordinary(3)(4) to be 7. I don't know if I said anything to suggest
that the syntax should be quite so specialcased, but I hope not.

Mike


From jeanpierreda at gmail.com  Mon Jul  9 03:45:11 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 8 Jul 2012 21:45:11 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <4FFA2B3F.5010106@pearwood.info>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<4FFA2B3F.5010106@pearwood.info>
Message-ID: <CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com>

On Sun, Jul 8, 2012 at 8:52 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Concise, yes, but I disagree that it is self-explaining.

I think the idea is that just as

  def foo(a): ...

can be called as foo(x) to execute the body,

  def foo(a)(b): ...

can be called as foo(x)(y) to execute the body.

> I'm having a lot of difficulty in thinking of how I would explain it to even
> a moderately experienced user except by expanding it out to the explicit
> nested function form. I'm not sure I can even explain it to myself.

What's wrong with explaining it by expansion?

Also, I doubt it's as hard as you claim. Something similar -- curried
functions -- have been a staple of functional programming languages
since before Python ever existed. This should also help explain it to
lots of new users with wider backgrounds.

> What will happen when the user invariably writes something like this?
>
> def ordinary(a)(b):
>     return a + b

Surely it'd define a curried function s.t. ordinary(1)(2) == 3?

-- Devin


From ncoghlan at gmail.com  Mon Jul  9 03:46:16 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 11:46:16 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
	<CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
Message-ID: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>

On Mon, Jul 9, 2012 at 11:11 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Beyond that, no, this is too limited - it only helps when there's no extra
>> code in the outer scope which, in my experience, is the exception rather
>> than the rule.
>
> I'm not sure I think that's the case.

- functools.wraps

Not really, as it uses functools.partial, not a nested def (sure you
could rewrite it to use a nested def instead, but why?)

> functools.lru_cache

No, as it has pre- and post-processing steps around the nested
function definition.

- reprlib.recursive_repr

Yes, this one would qualify.

However, the simplest possible complete definition of decorators is as follows:

- a decorator expression (the bit after "@") on a def statement must
produce a function decorator
- a decorator expression on a class statement must produce a class decorator
- a function decorator accepts a function and produces another object.
This is typically another function (or even the original function for
annotation and registration decorators), but may also be a different
kind of object such as a descriptor (e.g. property, classmethod,
staticmethod) or context manager (e.g. contextlib.contextmanager)
- a class decorator is similar, but accepts a class rather than a function
- a decorator factory is any callable that returns a decorator
- function decorators are often written as functions that return a
nested function (1 level of lexical nesting)
- function decorator factories are often written as functions that
return a function that returns a functions (2 levels of lexical
nesting)

Creating a dedicated syntax for the special case of function decorator
factories written to use lexical nesting makes the language as a whole
*more* complicated rather than less. The only way to actually make
them *simpler* would be to eliminate one or more of the bullet points
from the above list, and that can't be done while retaining the
current functionality.

Yes, it means that to write efficient custom decorator factories you
need to understand both the decoration process and lexical closures.
That's *OK* - it just means writing custom decorator factories (as
opposed to using those written by others) is a moderately advanced
metaprogramming technique (it's still a lot simpler than writing a
custom metaclass, though).

Some concepts are just hard to get your head around, but that doesn't
mean we should be creating dedicated syntax for a very specialised use
case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From greg.ewing at canterbury.ac.nz  Mon Jul  9 00:24:58 2012
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 09 Jul 2012 10:24:58 +1200
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <4FFA08BA.8090504@canterbury.ac.nz>

Mike Graham wrote:

> def timesn(n)(f):
>     def inner(y):
>         return n * f(y)
>     return inner

+1 from me on this. Scheme has an analogous feature, btw, which is
very elegant. I'd love to see this in Python.

As has been pointed out, you can go further and collapse this
into

def timesn(n)(f)(y):
    return n * f(y)

This means it would also be useful for non-parametric decorators
as well, e.g.

def fivetimes(f)(x):
    return 5 * f(x)

-- 
Greg


From jeanpierreda at gmail.com  Mon Jul  9 03:55:09 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 8 Jul 2012 21:55:09 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
	<CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
	<CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
Message-ID: <CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com>

On Sun, Jul 8, 2012 at 9:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, the simplest possible complete definition of decorators is as follows:
>
> - a decorator expression (the bit after "@") on a def statement must
> produce a function decorator
> - a decorator expression on a class statement must produce a class decorator
> - a function decorator accepts a function and produces another object.
> This is typically another function (or even the original function for
> annotation and registration decorators), but may also be a different
> kind of object such as a descriptor (e.g. property, classmethod,
> staticmethod) or context manager (e.g. contextlib.contextmanager)
> - a class decorator is similar, but accepts a class rather than a function
> - a decorator factory is any callable that returns a decorator
> - function decorators are often written as functions that return a
> nested function (1 level of lexical nesting)
> - function decorator factories are often written as functions that
> return a function that returns a functions (2 levels of lexical
> nesting)

Mmm, this distinction of "kinds of decorators" is silly. The only
requirement for something to be a decorator is that it takes one
argument. The type of the argument could be anything.

def dumb_decorator(x):
    return x + 2

@dumb_decorator
@apply
def foo():
    return 5

> Creating a dedicated syntax for the special case of function decorator
> factories written to use lexical nesting makes the language as a whole
> *more* complicated rather than less. The only way to actually make
> them *simpler* would be to eliminate one or more of the bullet points
> from the above list, and that can't be done while retaining the
> current functionality.

Nothing about the OP's suggestion was specific to "function decorator
factories". Observe a class decorator factory:

def stupid_augmented_subclass_decorator(**kwargs)(cls):
    class MyClass(cls):
        pass
    MyClass.__dict__.update(**kwargs)
    return MyClass

-- Devin


From ncoghlan at gmail.com  Mon Jul  9 04:00:41 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 12:00:41 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<4FFA2B3F.5010106@pearwood.info>
	<CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com>
Message-ID: <CADiSq7f3JCVQmgyshroiAJX91NWzeXgQz-8OJwPf95WzATT6Lg@mail.gmail.com>

On Mon, Jul 9, 2012 at 11:45 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
>> I'm having a lot of difficulty in thinking of how I would explain it to even
>> a moderately experienced user except by expanding it out to the explicit
>> nested function form. I'm not sure I can even explain it to myself.
>
> What's wrong with explaining it by expansion?
>
> Also, I doubt it's as hard as you claim. Something similar -- curried
> functions -- have been a staple of functional programming languages
> since before Python ever existed. This should also help explain it to
> lots of new users with wider backgrounds.

OK, I think it would be *much* better to approach the problem from that angle.

    def incremental(x)(f)(y):
        return x + f(y)

As equivalent to:

    # implied names are not legal identifiers - the compiler gets to do that
    # because of its privileged role in naming things
    def incremental(x):
        def <incremental:1>(f):
            def <incremental:2>(y):
                return x +f(y)
        return <incremental:1>

I'm still not convinced of the general applicability (since it breaks
down as soon as you want to decorate or otherwise pre- or post-process
any of the steps, thus forcing people to learn the full
"callable-returning-a-callable" idiom anyway), but it's probably worth
writing up as a PEP in the 3.4 timeframe.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Mon Jul  9 04:02:48 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 12:02:48 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
	<CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
	<CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
	<CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com>
Message-ID: <CADiSq7eAwYDeduayqhk-=M2cJ_+sZ4D6rtZECPoSY2=GLGO_rw@mail.gmail.com>

On Mon, Jul 9, 2012 at 11:55 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> Nothing about the OP's suggestion was specific to "function decorator
> factories".

The sole motivating use case presented was "A common stumbling block
for new users is writing decorators that take
arguments."

It's a terrible motivating use case for curried functions, because
they only work for the most trivial of decorator factories.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From haoyi.sg at gmail.com  Mon Jul  9 04:04:14 2012
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Sun, 8 Jul 2012 19:04:14 -0700
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
	<CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
	<CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
Message-ID: <CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com>

Just another point of data, Scala has exactly what is being proposed:
multiple parameter lists as synctactic sugar for nested return functions

http://stackoverflow.com/questions/4697404/scala-currying-by-nested-functions-or-by-multiple-parameter-lists


It also works with all the non-trivial edge-casey stuff python has (args,
default/named args, varargs, kwargs, splats).

My experience with this is that it actually works pretty well, the number
of "function returning function" types which require pre- or post-
processing is relatively small, and often this pre/post processing can
simply be moved into the body of the function for no real cost.

The cases where that cannot be done, it usually is because apart from
simply returning a new function, I'm also enclosing over some mutable state
in the outer function. In these cases I've found that this implicitly
captured mutable state is better handled with an object-based decorator,
making the stateful nature explicit. The lru_cache decorator seems like a
good candidate for using explicit objects to hold the cache, rather
cunningly enclosed mutable state.

I agree that as a "decorator factory helper", having extra syntax is
unwarranted. However, I think that in the general case, being able to
define curried functions in one line, a.l.a.

def func(a)(b)(c) = a * b * c

rather than

def func(a):
   def wrapper_two(b):
      def wrapper_one(c):
         return a * b * c
      return wrapper_one
   return wrapper_two

would in fact be very nice. From the Zen of Python, I would say these are
big wins for

- Flat is better than nested
- Readability Counts
- Beautiful is better than ugly

whether it's a win or lose for Explicit is better than Implicit is
debatable:

In the implementation domain, you're being less explicit about what's
actually happening (function returning functions)

but in the "how-to-use-me" domain you're being more explicit about what you
actually want (call this function with multiple parameter lists).

def f(a)(b)(c)

looks much more similar to how you would use it:

f(1)(2)(3)

than that horrible triple-nested monstrosity above.

-Haoyi

On Sun, Jul 8, 2012 at 6:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Mon, Jul 9, 2012 at 11:11 AM, Mike Graham <mikegraham at gmail.com> wrote:
> > On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> Beyond that, no, this is too limited - it only helps when there's no
> extra
> >> code in the outer scope which, in my experience, is the exception rather
> >> than the rule.
> >
> > I'm not sure I think that's the case.
>
> - functools.wraps
>
> Not really, as it uses functools.partial, not a nested def (sure you
> could rewrite it to use a nested def instead, but why?)
>
> > functools.lru_cache
>
> No, as it has pre- and post-processing steps around the nested
> function definition.
>
> - reprlib.recursive_repr
>
> Yes, this one would qualify.
>
> However, the simplest possible complete definition of decorators is as
> follows:
>
> - a decorator expression (the bit after "@") on a def statement must
> produce a function decorator
> - a decorator expression on a class statement must produce a class
> decorator
> - a function decorator accepts a function and produces another object.
> This is typically another function (or even the original function for
> annotation and registration decorators), but may also be a different
> kind of object such as a descriptor (e.g. property, classmethod,
> staticmethod) or context manager (e.g. contextlib.contextmanager)
> - a class decorator is similar, but accepts a class rather than a function
> - a decorator factory is any callable that returns a decorator
> - function decorators are often written as functions that return a
> nested function (1 level of lexical nesting)
> - function decorator factories are often written as functions that
> return a function that returns a functions (2 levels of lexical
> nesting)
>
> Creating a dedicated syntax for the special case of function decorator
> factories written to use lexical nesting makes the language as a whole
> *more* complicated rather than less. The only way to actually make
> them *simpler* would be to eliminate one or more of the bullet points
> from the above list, and that can't be done while retaining the
> current functionality.
>
> Yes, it means that to write efficient custom decorator factories you
> need to understand both the decoration process and lexical closures.
> That's *OK* - it just means writing custom decorator factories (as
> opposed to using those written by others) is a moderately advanced
> metaprogramming technique (it's still a lot simpler than writing a
> custom metaclass, though).
>
> Some concepts are just hard to get your head around, but that doesn't
> mean we should be creating dedicated syntax for a very specialised use
> case.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120708/d1ce8f89/attachment.html>

From wuwei23 at gmail.com  Mon Jul  9 06:16:51 2012
From: wuwei23 at gmail.com (alex23)
Date: Sun, 8 Jul 2012 21:16:51 -0700 (PDT)
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <20120706181404.GB3347@jaerhard.com>
References: <20120706181404.GB3347@jaerhard.com>
Message-ID: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>

On Jul 7, 4:14?am, J?rgen A. Erhard <jae+pyt... at jaerhard.com> wrote:
> So... I consider it very strange and annoying that there's this code.
> Ready to use. ?But you have to copy-and-paste it into your code
> *somewhere* instead of easily importing it.

I find it strange and annoying when people expect the standard library
to contain _everything_. What you gain in the convenience of readily
imported code imposes a cost on the ongoing maintenance and support of
the standard lib. Are you volunteering to write the PEP for its
inclusion, provide the implementation and support it until the end of
time?

> As to the "it makes using Python harder to learn", I beg to
> differ: an addition to the *language* (like, say, metaclasses) *can*
> make it more complicated. ?But additions to the stdlib? ?What about
> "batteries included"? ?Not a motto anymore (I heard rumors, so maybe
> that's the case)

Every single function added to the library makes the library broader
and thus more difficult to retain in memory.

It _is_ possible to present a case without snide allusions based on
hearsay: http://www.python.org/about/

> Putting these in some official(!) iterutils (or name it what you want)
> package would be a solution for so many people. ?Yes, not everyone
> needs grouper's current functionality. ?But for the many who do, it'd
> be there already.

So in the extremely likely outcome that a piece of functionality has
multiple implementations with different interpretations of edge cases,
who gets to decide which one is "more standard" than the other? If
such a decision has to be made, then it's not really "standard".

> And someone thought those recipes are useful,
> didn't you? ?So why not make them more easily available?

Because not everything needs to be in the standard library.

Because searching somewhere like Activestate's cookbook and copying
the implementations you find useful really isn't an onerous task.

Because if *you* feel there's an obvious lack in the stdlib then roll
your own package that covers it and put it up on PyPI. If it becomes
insanely popular, then you have a lot more leverage for insisting that
something would be beneficial to be included other than gut feeling &
convenience.


From ncoghlan at gmail.com  Mon Jul  9 06:58:06 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 14:58:06 +1000
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
Message-ID: <CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com>

On Mon, Jul 9, 2012 at 2:16 PM, alex23 <wuwei23 at gmail.com> wrote:
> So in the extremely likely outcome that a piece of functionality has
> multiple implementations with different interpretations of edge cases,
> who gets to decide which one is "more standard" than the other? If
> such a decision has to be made, then it's not really "standard".

This is often what lies at the heart of "not every 3 line function
needs to be <a builtin>/<in the standard library>".

It's very easy to hit a point of diminishing returns where the number
of possible alternatives mean that attempting to create an abstraction
layer ends up creating a UI that is *more complicated* than just
writing your own utility function that does exactly what you want.
Grouping is one such API (fixed length? Variable length with a
delimiter? Pad last group? Drop last group? Error early? Accept
arbitrary iterables? Accept sequences only? Support overlapping
windows?). Recursive descent into arbitrary collections is another
(Error on reference loops? Treat strings/bytes/bytearray as atomic
types? Treat any implementor of the buffer interface as an atomic
type? Support only hashable objects?).

There are a few key reasons why things end up in the standard library:

1. They're closely coupled to the language definition and
implementation and should be updated in sync with it (e.g. sys, imp,
gc, importlib, dis, inspect)
2. We want to use them in other parts of the standard library or its
test suite (e.g. collections, many of the unittest enhancements)
3. They solve a common problem that is otherwise prone to being
handled with incomplete solutions that lead to incorrect code (e.g.
ipaddress instead of regex based recipes for processing IP addresses)
4. It's an old module that would probably be left to PyPI these days,
but was added in earlier times when stdlib inclusion criteria were
less strict (but isn't broken so there's no harm in keeping it
around).

Sometimes a problem is sufficiently common, and has few enough
variations, that it's worth creating and providing a standard version.
That's the general aim of the itertools module. Other times, the core
problem is difficult enough that it's worth providing a standard
solution, even if it does mean including a vast array of configuration
options (e.g. subprocess. Popen).

However, sometimes, the correct answer to "Hey, this is a really
common pattern" is not "We should provide an API that uses that
pattern internally" but "we should document this pattern, so people
know it's a common idiom and can tailor it to their specific use case
and preferences".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From stephen at xemacs.org  Mon Jul  9 07:48:37 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 09 Jul 2012 14:48:37 +0900
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com>
	<CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com>
	<CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com>
	<CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com>
Message-ID: <87obnp5vt6.fsf@uwakimon.sk.tsukuba.ac.jp>

Haoyi Li writes:

 > However, I think that in the general case, being able to define
 > curried functions in one line, a.l.a.
[...]
 > def f(a)(b)(c)
 > 
 > looks much more similar to how you would use it:
 > 
 > f(1)(2)(3)

I don't understand.  I wouldn't use it that way!  I would think if
that were common usage, and the curried form only occasional,

    def f(a,b,c):
        pass

    f(1,2,3)

would be more natural in Python (and use functools.partial for the
occasional currying).  Presumably if you were using this syntax, you'd
be doing something more like

    def f(a)(b)(c):
        pass

    def g(h, x):
        h(x)

    g(f(1)(2), 3)

which doesn't look very nice to me, nor do I find the def to be a
particularly intuitive way of reminding me that the proper usage is a
call of a function in curried form (specifically, I would not be
reminded of whether the expected curried form is a 1st-, 2nd-, or
3rd-order function).

I also suppose that you wouldn't be able to do

    def f(a)(b)(c):
        pass

    def g(x,y):
        pass

    h(f(1))
    h(g)

(ie, without LBYL or EAFP constructs in h, which would simply be
throwing the complexity into the caller's backyard).  Even though

    def bar(a)(b):
        pass

    def baz(x):
        pass

    quux(bar(1))
    quux(baz)

would work fine.

So to me, there may be something in this syntax, but my initial
impression is that it's trying to turn Python into something it's not.


From haoyi.sg at gmail.com  Mon Jul  9 11:07:49 2012
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Mon, 9 Jul 2012 02:07:49 -0700
Subject: [Python-ideas] Syntax for defining parametric decorators
Message-ID: <2237588560696344575@unknownmsgid>

I admit calling it all at once would be very strange, but i still think
the "call me three times to get result" comes across waaaay more
clearly than with nested defs.

Multiple arguments in one list is about the same, it's true, until you
really need the curried version, like in decorators, then you end up
with the nasty tower-of-defs which sparked this discussion.

Regarding "not knowing how many times it should be called", the idea is
that you break it up when you have a meaningful, reusable unit. That's
to say you wouldn't do a

def f(a)(b)(c)

Unless

f
f(1)
f(1)(2)
f(1)(2)(3)

Are all meaningful, useful functions. Doesn't happen very often,
granted, but when it does (e.g. for decorator factories and decorators)
it'd be nice to have.

I view curried defs like default parameters: Doesnt add anything that
cant already be done (you can emulate default args with telescoping
methods) but cuts boilerplate, makes things clearer and overall nice to
have. Whether it's nice enough to justify extra syntax is of course
open for debate.

From: Stephen J. Turnbull
Sent: 7/9/2012 1:48 AM
To: Haoyi Li
Cc: Nick Coghlan; Python-Ideas
Subject: Re: [Python-ideas] Syntax for defining parametric decorators
Haoyi Li writes:

 > However, I think that in the general case, being able to define
 > curried functions in one line, a.l.a.
[...]
 > def f(a)(b)(c)
 >
 > looks much more similar to how you would use it:
 >
 > f(1)(2)(3)

I don't understand.  I wouldn't use it that way!  I would think if
that were common usage, and the curried form only occasional,

    def f(a,b,c):
        pass

    f(1,2,3)

would be more natural in Python (and use functools.partial for the
occasional currying).  Presumably if you were using this syntax, you'd
be doing something more like

    def f(a)(b)(c):
        pass

    def g(h, x):
        h(x)

    g(f(1)(2), 3)

which doesn't look very nice to me, nor do I find the def to be a
particularly intuitive way of reminding me that the proper usage is a
call of a function in curried form (specifically, I would not be
reminded of whether the expected curried form is a 1st-, 2nd-, or
3rd-order function).

I also suppose that you wouldn't be able to do

    def f(a)(b)(c):
        pass

    def g(x,y):
        pass

    h(f(1))
    h(g)

(ie, without LBYL or EAFP constructs in h, which would simply be
throwing the complexity into the caller's backyard).  Even though

    def bar(a)(b):
        pass

    def baz(x):
        pass

    quux(bar(1))
    quux(baz)

would work fine.

So to me, there may be something in this syntax, but my initial
impression is that it's trying to turn Python into something it's not.


From stefan_ml at behnel.de  Mon Jul  9 11:47:48 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 09 Jul 2012 11:47:48 +0200
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <jte9c4$2ej$1@dough.gmane.org>

Mike Graham, 08.07.2012 22:22:
> A common stumbling block for new users is writing decorators that take
> arguments. To create a decorator like
> 
> @timesn(n)
> def f(y):
>     ...
> 
> We write code like
> 
> def timesn(n)
>     def decorator(f):
>         def inner(y):
>             return n * f(y)
>         return inner
>      return decorator
> 
> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
> 
> def timesn(n)(f):
>     def inner(y):
>         return n * f(y)
>     return inner
> 
> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.

>From an innocent look, I have no idea what the syntax is supposed to mean.
Clearly doesn't hint at a factory for me.

Stefan



From mwm at mired.org  Mon Jul  9 11:56:55 2012
From: mwm at mired.org (Mike Meyer)
Date: Mon, 9 Jul 2012 05:56:55 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <jte9c4$2ej$1@dough.gmane.org>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
Message-ID: <20120709055655.49a978c7@bhuda.mired.org>

On Mon, 09 Jul 2012 11:47:48 +0200
Stefan Behnel <stefan_ml at behnel.de> wrote:
> Mike Graham, 08.07.2012 22:22:
> > def timesn(n)(f):
> >     def inner(y):
> >         return n * f(y)
> >     return inner
> > 
> > which is more concise and looks a lot more like a non-parametric
> > decorator someone might have written already. The syntax is mostly
> > self-explaining and could potentially be useful in other contexts.
> 
> >From an innocent look, I have no idea what the syntax is supposed to mean.
> Clearly doesn't hint at a factory for me.

Does to me. timesn is clearly a function of one argument that is going
to return a callable object that takes one argument. So:

x5 = times(5)
print x5(7)

should print 35.

I'd be interested in seeing an implementation of partial that used
this mechanism. Should be straightforward, but I'm in the midst of a
crunch.

	<mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From steve at pearwood.info  Mon Jul  9 13:32:59 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 09 Jul 2012 21:32:59 +1000
Subject: [Python-ideas] itertools recipes: why not add them to
 the	stdlib *somewhere*?
In-Reply-To: <CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com>
Message-ID: <4FFAC16B.7030206@pearwood.info>

Nick Coghlan wrote:

> It's very easy to hit a point of diminishing returns where the number
> of possible alternatives mean that attempting to create an abstraction
> layer ends up creating a UI that is *more complicated* than just
> writing your own utility function that does exactly what you want.
[...]

+1 on everything Nick says here.

One additional comment:

> However, sometimes, the correct answer to "Hey, this is a really
> common pattern" is not "We should provide an API that uses that
> pattern internally" but "we should document this pattern, so people
> know it's a common idiom and can tailor it to their specific use case
> and preferences".

Well, maybe, but documenting software patterns are not necessarily "our" job 
(whoever "our" is -- the Python devs, or python.org, or the PSI, or whoever 
people think is responsible).

Python has not been a one-man project for many, many years. There is a rich 
Python ecosystem now, not just Guido or the Python devs, but StackOverflow, 
comp.lang.python, dozens of books, more bloggers than you can poke a stick at, 
#python, etc etc etc. It's not "our" responsibility to teach every Python 
programmer every conceivable pattern. At some point we can just draw a line 
and say "Yeah, that's a useful pattern, but we're not going to document it. 
Somebody else can do it."

Even if that line is somewhat arbitrary.



-- 
Steven


From ironfroggy at gmail.com  Mon Jul  9 14:28:58 2012
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Mon, 9 Jul 2012 08:28:58 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com>

On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote:
> A common stumbling block for new users is writing decorators that take
> arguments. To create a decorator like
>
> @timesn(n)
> def f(y):
>     ...
>
> We write code like
>
> def timesn(n)
>     def decorator(f):
>         def inner(y):
>             return n * f(y)
>         return inner
>      return decorator
>
> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
>
> def timesn(n)(f):
>     def inner(y):
>         return n * f(y)
>     return inner
>
> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.
>
> There exist tools like the decorator library to try to simplify this
> already, but in my experience they mostly serve to confuse people
> using decorators for the first time more.
>
> One thing I didn't specify was whether `n` was nonlocal or not and the
> behavior of something that keeps and reuses timesn(some_specific_n)
> multiple times.
>
> Does anyone think a feature like this may be useful?

While I'm a bit taken aback by the syntax, I recognize this is mostly
because I'm
not used to it. Objectively, it makes enough sense that I think it
fits. I think even
if the only use case is decorator factories, they are a useful enough
thing that this
is okay. We have a specific syntax to support using decorators, why
not better support
writing them?

Still, I'd like to see any ideas people have for alternative uses for
this syntax.

> Regards,
> Mike
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From jsbueno at python.org.br  Mon Jul  9 15:14:26 2012
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Mon, 9 Jul 2012 10:14:26 -0300
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com>
Message-ID: <CAH0mxTROfkmJdtsX1eY-7wTPa_n3SPOq_O3n3bx_ZAax5=YpiQ@mail.gmail.com>

On 9 July 2012 09:28, Calvin Spealman <ironfroggy at gmail.com> wrote:
> On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote:
>> A common stumbling block for new users is writing decorators that take
>> arguments. To create a decorator like
>>
>> @timesn(n)
>> def f(y):
>>     ...


-1 for the new syntax, since a  decorator for trivial decorator factories
(rather, parameterized decorators) can be written in a total of 5 lines:


from functools import partial

def parameterized(decorator):
    def part_decorator(*args, **kw):
        return partial(decorator, *args, **kw)
    return part_decorator

>>> @parameterized
... def times(n, func):
...     def new_func(*args, **kw):
...         return n * func(*args, **kw)
...     return new_func
...
>>>
>>> @times(3)
... def add(x ,y):
...     return x + y
...
>>> add(1,1)
6
-------
For more complex cases, requiring pre processing, post processing, and
so on, the normal syntax can cut it.
And, certainly, a more complex such decorator could be written, so
that it accepts some arguments itself.

(I would be +1 for such a decorator in functools)

  js
 -><-


From aquavitae69 at gmail.com  Mon Jul  9 17:18:23 2012
From: aquavitae69 at gmail.com (David Townshend)
Date: Mon, 9 Jul 2012 17:18:23 +0200
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
Message-ID: <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com>

On Jul 9, 2012 6:17 AM, "alex23" <wuwei23 at gmail.com> wrote:
>
> On Jul 7, 4:14 am, J?rgen A. Erhard <jae+pyt... at jaerhard.com> wrote:
> > So... I consider it very strange and annoying that there's this code.
> > Ready to use.  But you have to copy-and-paste it into your code
> > *somewhere* instead of easily importing it.
>
> I find it strange and annoying when people expect the standard library
> to contain _everything_. What you gain in the convenience of readily
> imported code imposes a cost on the ongoing maintenance and support of
> the standard lib. Are you volunteering to write the PEP for its
> inclusion, provide the implementation and support it until the end of
> time?

While I agree that these recipes are not suitable for inclusion in stdlib
(for reasons made elsewhere), I don't agree with your (somewhat aggressive)
reasoning. The implementation has already been provided, and is being
maintained in the form of documentation. Simply copying the recipes into a
module is hardly an onerous task. And fairly often people mailing to this
list *are* volunteering to write a PEP, provide an implementation, etc.

>
> > As to the "it makes using Python harder to learn", I beg to
> > differ: an addition to the *language* (like, say, metaclasses) *can*
> > make it more complicated.  But additions to the stdlib?  What about
> > "batteries included"?  Not a motto anymore (I heard rumors, so maybe
> > that's the case)
>
> Every single function added to the library makes the library broader
> and thus more difficult to retain in memory.
>
> It _is_ possible to present a case without snide allusions based on
> hearsay: http://www.python.org/about/
>
> > Putting these in some official(!) iterutils (or name it what you want)
> > package would be a solution for so many people.  Yes, not everyone
> > needs grouper's current functionality.  But for the many who do, it'd
> > be there already.
>
> So in the extremely likely outcome that a piece of functionality has
> multiple implementations with different interpretations of edge cases,
> who gets to decide which one is "more standard" than the other? If
> such a decision has to be made, then it's not really "standard".
>
> > And someone thought those recipes are useful,
> > didn't you?  So why not make them more easily available?
>
> Because not everything needs to be in the standard library.
>
> Because searching somewhere like Activestate's cookbook and copying
> the implementations you find useful really isn't an onerous task.
>
> Because if *you* feel there's an obvious lack in the stdlib then roll
> your own package that covers it and put it up on PyPI. If it becomes
> insanely popular, then you have a lot more leverage for insisting that
> something would be beneficial to be included other than gut feeling &
> convenience.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/1fbf8dc5/attachment.html>

From raymond.hettinger at gmail.com  Mon Jul  9 18:18:54 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 9 Jul 2012 09:18:54 -0700
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com>
Message-ID: <A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com>


On Jul 9, 2012, at 8:18 AM, David Townshend wrote:

> . Simply copying the recipes into a module is hardly an onerous task.

In fact, it has already been done :-)
http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search



Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/3765f429/attachment.html>

From tjreedy at udel.edu  Mon Jul  9 21:47:10 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 09 Jul 2012 15:47:10 -0400
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com>
	<A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com>
Message-ID: <jtfcge$8o1$2@dough.gmane.org>

On 7/9/2012 12:18 PM, Raymond Hettinger wrote:
>
> On Jul 9, 2012, at 8:18 AM, David Townshend wrote:
>
>> . Simply copying the recipes into a module is hardly an onerous task.
>
> In fact, it has already been done :-)

Multiple times. Possible addition to the doc in 9.1.2 after first sentence:

These and similar recipes have been incorporated in various third-party 
modules. See ...

> http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search


-- 
Terry Jan Reedy





From jeanpierreda at gmail.com  Mon Jul  9 22:12:22 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 9 Jul 2012 16:12:22 -0400
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
Message-ID: <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>

On Mon, Jul 9, 2012 at 12:16 AM, alex23 <wuwei23 at gmail.com> wrote:
> I find it strange and annoying when people expect the standard library
> to contain _everything_. What you gain in the convenience of readily
> imported code imposes a cost on the ongoing maintenance and support of
> the standard lib. Are you volunteering to write the PEP for its
> inclusion, provide the implementation and support it until the end of
> time?

Are you saying the recipes are unsupported?

-- Devin


From wuwei23 at gmail.com  Tue Jul 10 00:28:41 2012
From: wuwei23 at gmail.com (wu wei)
Date: Tue, 10 Jul 2012 08:28:41 +1000
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
Message-ID: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>

On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda at gmail.com>
 wrote:

> Are you saying the recipes are unsupported?
>

This is getting ridiculous. I'm not sure what is so controversial about not
wanting to scrape up every piece of crap code off the web that some lazy
coder can't be bothered to implement contextually and shove them into the
_standard library_. Are _you_ seriously saying that every one of those
random bits of code _is_ supported? That they will all be kept up to date
as the language evolves? That they won't turn into another module that
people complain is "stale"? Or another "why tkinter and not <x>" argument?

Every inclusion to the library adds overhead to finding and using elements
of the library.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120710/3814f28c/attachment.html>

From phd at phdru.name  Tue Jul 10 00:36:26 2012
From: phd at phdru.name (Oleg Broytman)
Date: Tue, 10 Jul 2012 02:36:26 +0400
Subject: [Python-ideas] itertools recipes: why not add them to the
 stdlib *somewhere*?
In-Reply-To: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
Message-ID: <20120709223626.GB1279@iskra.aviel.ru>

On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23 at gmail.com> wrote:
> Every inclusion to the library adds overhead to finding and using elements
> of the library.

   And supporting it. It's much harder to remove bits from stdlib than
to add to it because stdlib have to maintain backward compatibility.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.


From alexandre.zani at gmail.com  Tue Jul 10 00:42:52 2012
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Mon, 9 Jul 2012 15:42:52 -0700
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <20120709223626.GB1279@iskra.aviel.ru>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
	<20120709223626.GB1279@iskra.aviel.ru>
Message-ID: <CAJVMpoqh=66p+uYEVi_4og4gFT6r0XFZJ1kzFgm_Ei+DRs-rOQ@mail.gmail.com>

On Mon, Jul 9, 2012 at 3:36 PM, Oleg Broytman <phd at phdru.name> wrote:
> On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23 at gmail.com> wrote:
>> Every inclusion to the library adds overhead to finding and using elements
>> of the library.
>
>    And supporting it. It's much harder to remove bits from stdlib than
> to add to it because stdlib have to maintain backward compatibility.
>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            phd at phdru.name
>            Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

It is a good point that those recipes because they are in the official
docs are already requiring support and maintenance. However, as
somebody above mentioned, there are many possible variations on those
recipes and so it makes a lot more sense to tell people to implement
their own version of grouper that will match their specific needs
rather than create an infinitely configurable version of grouper in
the stdlib. Furthermore, if the maintenance of those recipes ever
becomes undesirable, it is a simple matter to strike them from the
docs. The same cannot be said if they were inducted in the stdlib.

Alexandre Zani


From jeanpierreda at gmail.com  Tue Jul 10 00:45:55 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 9 Jul 2012 18:45:55 -0400
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
Message-ID: <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com>

On Mon, Jul 9, 2012 at 6:28 PM, wu wei <wuwei23 at gmail.com> wrote:
> On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda at gmail.com>
> wrote:
>>
>> Are you saying the recipes are unsupported?
>
>
> This is getting ridiculous. I'm not sure what is so controversial about not
> wanting to scrape up every piece of crap code off the web that some lazy

This is code from the official documentation.

> coder can't be bothered to implement contextually and shove them into the
> _standard library_. Are _you_ seriously saying that every one of those
> random bits of code _is_ supported?

No, I am suggesting that example code that exists in the documentation
is supported.

I think you are confused.

-- Devin


From tjreedy at udel.edu  Tue Jul 10 03:52:55 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 09 Jul 2012 21:52:55 -0400
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
	<CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com>
Message-ID: <jtg1u7$9nc$1@dough.gmane.org>

On 7/9/2012 6:45 PM, Devin Jeanpierre wrote:
> No, I am suggesting that example code that exists in the documentation
> is supported.

Well, in the sense that we try to make sure that they work as advertised 
*with the example data*, yes. However, that is the end of the support. 
Examples are just examples, put there to illustrate a didactic point. 
They are often simplified to serve the specific purpose. They are not 
intended to be production code, and they are not supported as that. 
Error handling is eliminated for clarity.# The api has not necessarily 
been thought through very well and is subject to change.# The multiple 
examples are not particularly meant to work together.

#Whereas, for example, the new in 3.3 ipaddress module, while based on 
existing 3rd party code, has had its api reexamined to make sure it well 
defined, documented, and something we will (likely) stick with. Nick has 
added a lot of error checking code with messages that try to say what is 
wrong rather than just 'you goofed'. And oh yes, the test suite has been 
expanded and reviewed with an aim to 100% coverage.

-- 
Terry Jan Reedy




From jeanpierreda at gmail.com  Tue Jul 10 05:38:38 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 9 Jul 2012 23:38:38 -0400
Subject: [Python-ideas] itertools recipes: why not add them to the
	stdlib *somewhere*?
In-Reply-To: <jtg1u7$9nc$1@dough.gmane.org>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
	<CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com>
	<jtg1u7$9nc$1@dough.gmane.org>
Message-ID: <CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com>

On Mon, Jul 9, 2012 at 9:52 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Well, in the sense that we try to make sure that they work as advertised
> *with the example data*, yes. However, that is the end of the support.
> Examples are just examples, put there to illustrate a didactic point. They
> are often simplified to serve the specific purpose. They are not intended to
> be production code, and they are not supported as that.

I've never assumed that (obviously, I guess), and that isn't stated in
the docs. If that's the level of support of those recipes, this should
be made clear inside the itertools documentation, because they _are_
used in peoples' code. For example, see the third party module linked
earlier in the thread.

On the contrary, the docs state the recipes are for creating an
"extended toolset", implying that they can be used freely the same as
the rest of the code.

-- Devin


From stephen at xemacs.org  Tue Jul 10 06:39:50 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 10 Jul 2012 13:39:50 +0900
Subject: [Python-ideas] itertools recipes: why not add them to
	the	stdlib *somewhere*?
In-Reply-To: <CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com>
References: <20120706181404.GB3347@jaerhard.com>
	<206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com>
	<CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com>
	<CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com>
	<CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com>
	<jtg1u7$9nc$1@dough.gmane.org>
	<CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com>
Message-ID: <87fw905iw9.fsf@uwakimon.sk.tsukuba.ac.jp>

Devin Jeanpierre writes:

 > I've never assumed that (obviously, I guess), and that isn't stated in
 > the docs. If that's the level of support of those recipes, this should
 > be made clear inside the itertools documentation,

It already is clear.  The very name "recipe" indicates that it may not
be to everyone's taste, that it's not carefully designed to handle
corner cases, and that people should review the code for their own
use.  The (very sparse) documentation associated with the recipes,
which is clearly oriented to advising you how you can (a) produce
similar code for different functions yourself and (b) make it better
further emphasizes what the name indicates.

Of course, all that may not be obvious to non-native speakers, but
that's a completely different issue from the starting point of this
thread, which was that these recipes belong in the stdlib rather than
in its documentation.  I have nothing to add to Terry's post
explaining why that is not appropriate for these recipes "as is."

 > because they _are_ used in peoples' code.

Of course they are.  That's what they're there for.



From techtonik at gmail.com  Tue Jul 10 08:22:11 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 10 Jul 2012 09:22:11 +0300
Subject: [Python-ideas] Explanation on how to search the archives
In-Reply-To: <4FF887A7.5070101@pearwood.info>
References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>
	<4FF887A7.5070101@pearwood.info>
Message-ID: <CAPkN8xJg-wZ8A3_iranDCfmhpWYFu9xms0U9kGgg90Hi-7qZyw@mail.gmail.com>

On Sat, Jul 7, 2012 at 10:01 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>
> Teaching people how to search the Internet and
> interact with mailing lists without being annoying is not the job of the
> Python devs

Aren't they primarily Python devs who are annoyed and complain that
people don't search properly?

I don't think your proposal to just write somewhere on the Internet,
but not on python.org helps. Even wiki.python.org doesn't help if
people don't have a chance to read it before posting. The idea about
giving a convenient search tool is pretty valid. The true problem is
that *.python.org is unable to do anything (not even a search form
here - http://mail.python.org/mailman/listinfo/python-ideas). For
usenet there were auto announcements, for Google Groups and forums
there are sticky threads with advices.
--
anatoly t.


From ncoghlan at gmail.com  Tue Jul 10 08:32:24 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jul 2012 16:32:24 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <jte9c4$2ej$1@dough.gmane.org>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
Message-ID: <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>

On Mon, Jul 9, 2012 at 7:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> >From an innocent look, I have no idea what the syntax is supposed to mean.
> Clearly doesn't hint at a factory for me.

I should also mention that I have a different proposal that affects
the way one would write functions-that-returns-functions. I've been
messing around with the idea of statement local namespaces for years
(see PEP 3150) trying to find something that I consider better than
the status quo, and PEP 403's statement local function and class
definitions (http://www.python.org/dev/peps/pep-0403/) are the current
incarnation.

With those, the number of statements in a simple wrapping decorator
factory doesn't change, but the return statements can be moved above
their respective function definitions:

    def notify_on_call(callback, *cb_args, **cb_kwds):
        in return decorator
        def decorator(f):
            in return wrapped
            @functools.wraps(f)
            def wrapped(*args, **kwds):
                callback(*cb_args, cb_kwds)
                return f(*args, **kwds)

Rather than the current out-of-order:

    def notify_on_call(callback, *cb_args, **cb_kwds):
        def decorator(f):
            @functools.wraps(f)
            def wrapped(*args, **kwds):
                callback(*cb_args, cb_kwds)
                return f(*args, **kwds)
            return wrapped
        return decorator

(Note: I haven't updated the PEP in a while, so it currently still
disallows combining the in statement with decorators - I think that's
a mistake, and will be updating it some time post 3.3)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From techtonik at gmail.com  Tue Jul 10 09:01:02 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 10 Jul 2012 10:01:02 +0300
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
	<CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
Message-ID: <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com>

On Thu, Jul 5, 2012 at 11:24 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2012/7/5 anatoly techtonik <techtonik at gmail.com>:
>> This makes me kind of sad. You have Python installed. Why can't you
>> just crossplatformly do:
>>
>>   mkdir nacl
>>   cd nacl
>>   python -m urllib get
>> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py
>>   python update_sdk.py
>
> I'm sure there is already a way with standard python tools. Something
> along these lines:
>
> python -c "from urllib.request import urlretrieve; urlretrieve('URL',
> 'update_sdk.zip')"
> python -m update_sdk.zip
>
> The second command will work if the zip file has a __main__.py.
> Do you think we need other tools?

Wow! Nice, but still a hack. I doubt many people wouls accept magic
files messing in the root dir of the package. The only way it will
look good is:

    python -m update_sdk.zip sdk_update[.py]

but I don't know if that's supported.
--
anatoly t.


From amauryfa at gmail.com  Tue Jul 10 09:51:25 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 10 Jul 2012 09:51:25 +0200
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
	<CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
	<CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com>
Message-ID: <CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com>

2012/7/10 anatoly techtonik <techtonik at gmail.com>:
> Wow! Nice, but still a hack. I doubt many people wouls accept magic
> files messing in the root dir of the package. The only way it will
> look good is:
>
>     python -m update_sdk.zip sdk_update[.py]
>
> but I don't know if that's supported.

-m does work with zip files (with a __main__.py file) and command line
arguments are passed.
Or did you expect another kind of support?

-- 
Amaury Forgeot d'Arc


From christopherreay at gmail.com  Tue Jul 10 09:58:31 2012
From: christopherreay at gmail.com (Christopher Reay)
Date: Tue, 10 Jul 2012 08:58:31 +0100
Subject: [Python-ideas] Explanation on how to search the archives
In-Reply-To: <4FF887A7.5070101@pearwood.info>
References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>
	<4FF887A7.5070101@pearwood.info>
Message-ID: <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com>

Im thinking something like:

Python-ideas mailing list
Python-ideas at python.org
http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
How to Search the Archives:
http://mail.python.org/dontReinventTheWheel
Searching the archives is the *first* place you should go when exploring
new ideas. They are full of (mostly!) excellent ideas, as well as showing
how to explain your idea well, and helping you to not reinvent the wheel...
or the tabloid. Please search for relevant keywords around your idea before
posting to python-ideas. Thanks!


As the footer of the emails.

This is a small addition which provides some great meta information to
"people with great ideas" and also allows a terse and simple response when
a veteran detects a repeated idea.

-- 

Be prepared to have your predictions come true
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120710/9d66f2b5/attachment.html>

From ncoghlan at gmail.com  Tue Jul 10 11:11:58 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 10 Jul 2012 19:11:58 +1000
Subject: [Python-ideas] Python as a tool to download stuff for
	bootstrapping
In-Reply-To: <CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com>
References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com>
	<CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com>
	<CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com>
	<CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com>
Message-ID: <CADiSq7dKfrNfDr5uv__GQe8CvJvETrU8wiDCW3RW_PuC_xeWhA@mail.gmail.com>

On Tue, Jul 10, 2012 at 5:51 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2012/7/10 anatoly techtonik <techtonik at gmail.com>:
>> Wow! Nice, but still a hack. I doubt many people wouls accept magic
>> files messing in the root dir of the package. The only way it will
>> look good is:
>>
>>     python -m update_sdk.zip sdk_update[.py]
>>
>> but I don't know if that's supported.
>
> -m does work with zip files (with a __main__.py file) and command line
> arguments are passed.
> Or did you expect another kind of support?

-m doesn't handle zip files - you just execute them directly:

python <script> = source files, bytecode files, directories, zip files
(requires a top level __main__.py in the latter two cases)
python -m <module> = modules, packages (requires a __main__ submodule
for the latter case)

The switch is only to tell the interpreter whether you're providing a
filesystem path (default) or a Python module name (-m switch).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From steve at pearwood.info  Wed Jul 11 03:12:10 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 11 Jul 2012 11:12:10 +1000
Subject: [Python-ideas] Explanation on how to search the archives
In-Reply-To: <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com>
References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>	<4FF887A7.5070101@pearwood.info>
	<CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com>
Message-ID: <4FFCD2EA.7090705@pearwood.info>

Christopher Reay wrote:
> Im thinking something like:
[...]
> How to Search the Archives:
> http://mail.python.org/dontReinventTheWheel
> Searching the archives is the *first* place you should go when exploring
> new ideas. ...

> As the footer of the emails.
> 
> This is a small addition which provides some great meta information to
> "people with great ideas" and also allows a terse and simple response when
> a veteran detects a repeated idea.

More bloat for every single email which will be utterly ignored. Case in 
point: the mail digest for the python-dev mailing list already states:

     When replying, please edit your Subject line so it is more specific
     than "Re: Contents of Python-Dev digest..."

and similar for python-list, tutor, etc. And this message is at the *top* of 
the digest, not the bottom, and a mere two lines, not a big wall of text[1]. 
Yet there is still a steady stream of posts like these:

http://mail.python.org/pipermail/python-ideas/2012-June/015547.html
http://mail.python.org/pipermail/python-dev/2012-July/120910.html
http://mail.python.org/pipermail/tutor/2012-July/090213.html

Don't believe me? How often do you read the "This email is Confidential blah 
blah blah yadda watermelon blah" footers at the bottom of oh-so-many corporate 
emails?

Everybody, sender and receiver, carries the cost of additional bloat in every 
email, and the people who most need the reminder that Python is twenty years 
old and this "brilliant idea" most likely has been thought of a dozen times 
before are the ones who are least likely to read instructions or pay any 
attention to them even if they do.



[1] Standards of what counts as big blocks of text have been steadily falling 
since the Internet became popular. I'm just waiting for people to start 
replying to twitter posts with "TL;DR", at which point I intend to turn my 
computer off and move to a cave in the desert.



-- 
Steven


From ben+python at benfinney.id.au  Thu Jul 12 03:31:57 2012
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 12 Jul 2012 11:31:57 +1000
Subject: [Python-ideas] Explanation on how to search the archives
References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com>
	<4FF887A7.5070101@pearwood.info>
	<CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com>
	<4FFCD2EA.7090705@pearwood.info>
Message-ID: <87y5mpojci.fsf@benfinney.id.au>

Steven D'Aprano <steve at pearwood.info> writes:

> [1] Standards of what counts as big blocks of text have been steadily
> falling since the Internet became popular. I'm just waiting for people
> to start replying to twitter posts with "TL;DR", at which point I
> intend to turn my computer off and move to a cave in the desert.

I used to read ?TL;DR?, but now I don't have the time to read five
characters.

-- 
 \            ?Simplicity is prerequisite for reliability.? ?Edsger W. |
  `\                                                          Dijkstra |
_o__)                                                                  |
Ben Finney



From steve at pearwood.info  Thu Jul 12 03:37:00 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Jul 2012 11:37:00 +1000
Subject: [Python-ideas] Moving return above statements [was Re: Syntax for
 defining parametric decorators]
In-Reply-To: <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>	<jte9c4$2ej$1@dough.gmane.org>
	<CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
Message-ID: <4FFE2A3C.1090905@pearwood.info>

Nick Coghlan wrote:
> On Mon, Jul 9, 2012 at 7:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> >From an innocent look, I have no idea what the syntax is supposed to mean.
>> Clearly doesn't hint at a factory for me.
> 
> I should also mention that I have a different proposal that affects
> the way one would write functions-that-returns-functions. I've been
> messing around with the idea of statement local namespaces for years
> (see PEP 3150) trying to find something that I consider better than
> the status quo, and PEP 403's statement local function and class
> definitions (http://www.python.org/dev/peps/pep-0403/) are the current
> incarnation.
> 
> With those, the number of statements in a simple wrapping decorator
> factory doesn't change, but the return statements can be moved above
> their respective function definitions:
> 
>     def notify_on_call(callback, *cb_args, **cb_kwds):
>         in return decorator
>         def decorator(f):
>             in return wrapped
>             @functools.wraps(f)
>             def wrapped(*args, **kwds):
>                 callback(*cb_args, cb_kwds)
>                 return f(*args, **kwds)


I *really* don't like the way that the "in return" statement reads like it 
creates a new block, but the following lines are not indented. If I were 
writing this as pseudo-code for a human reader, I would surely indent the 
following lines.

Even if the part following the "in return" were limited to a single 
expression, I would prefer to indent it if it appears on another line. Using 
lambda as an example:


# Best
lambda x: expression

# Acceptable
lambda x:\
     expression


# Unacceptable
lambda x:\
expression


Looking back at "in return", here's a simple example which doesn't use nested 
functions. Compare:


def function(arg):
     in return value
     value = process(arg)
     print("Value is", value)

versus:

def function(arg, verbose=False):
     in return value:
         value = process(arg)
         print("Value is", value)


The lack of indentation (and trailing colon) makes the first extremely 
unpythonic -- everything else that creates a new block is indented. 
Consequently I hate the first one and am merely cold to the second.



> Rather than the current out-of-order:
> 
>     def notify_on_call(callback, *cb_args, **cb_kwds):
>         def decorator(f):
>             @functools.wraps(f)
>             def wrapped(*args, **kwds):
>                 callback(*cb_args, cb_kwds)
>                 return f(*args, **kwds)
>             return wrapped
>         return decorator


I would not describe that as "out-of-order". Seems to me that it is precisely 
in order: first you create the object (a function), then you return it. You 
can't return something before it exists.

It seems to me that this proposal, and the older PEP 3150, are the ones which 
are out-of-order: you use things before they are defined.

For what it's worth, I have slightly warmed to PEP 3150 and would give it a 
very tentative +0.125:

def notify_on_call(callback, *cb_args, **cb_kwds):
     return decorator given:
         def decorator(f):
             @functools.wraps(f)
             def wrapped(*args, **kwds):
                 callback(*cb_args, cb_kwds)
                 return f(*args, **kwds)
             return wrapped


I don't think the above is any improvement at all on the status quo, but 3150 
would allow you to write maths expressions more mathematically:


def func(x):
     return a**2 + 3*a given:
         a = 1/(sin(x*pi)) + (cos(x)-1)/2


which I'm not sure will help people used to reading code, but it should at 
least be familiar territory to mathematicians and maths geeks.


(Aside: I actually prefer that bikeshed to be called "where" rather than "given".)



-- 
Steven


From tjreedy at udel.edu  Thu Jul 12 04:19:01 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 11 Jul 2012 22:19:01 -0400
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
Message-ID: <jtlc79$195$1@dough.gmane.org>

On 7/8/2012 4:22 PM, Mike Graham wrote:
> A common stumbling block for new users is writing decorators that take
> arguments. To create a decorator like
>
> @timesn(n)
> def f(y):
>      ...
>
> We write code like
>
> def timesn(n)
>      def decorator(f):
>          def inner(y):
>              return n * f(y)
>          return inner
>       return decorator
>
> which confuses many users and can be a handful to type. I wonder if it
> would be clearer for people to write
>
> def timesn(n)(f):
>      def inner(y):
>          return n * f(y)
>      return inner
>
> which is more concise and looks a lot more like a non-parametric
> decorator someone might have written already. The syntax is mostly
> self-explaining and could potentially be useful in other contexts.

As others have noted, this is a proposal for introducing optional 
automatic function currying to python. Parametric decorators are an 
obvious use case, because they require double function nesting, but 
neither the interpreter not the function, such as timesn, would know or 
care how the function will be used. If you are really concerned 
specifically about decorator expressions, I think someone suggestion of 
a new decorator function in functools would be the better way to go.

For people 'raised' with imperative languages without nested functions 
and currying, function currying is definately *not* 'self-explaining'. 
The difficultly is conceptual, not syntactical. I am 99% sure it would 
result in more confusion, not less.

> There exist tools like the decorator library to try to simplify this
> already, but in my experience they mostly serve to confuse people
> using decorators for the first time more.
>
> One thing I didn't specify was whether `n` was nonlocal or not and the
> behavior of something that keeps and reuses timesn(some_specific_n)
> multiple times.
>
> Does anyone think a feature like this may be useful?

Just about every proposal would be 'useful' to someone in some 
situations. Is it useful enough to justify the costs?  My brief answer 
is that the syntactic sugar of automatic currying only applies to a few 
situations and does not add any new functionality. It would be another 
thing to learn, which most people would not get until they learned the 
more general explicit nesting. For some people, say from always-curried 
languages, it might be an attractive nuisance in that they would use it 
and incur the added overhead when it is pointless.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Thu Jul 12 05:49:14 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 12 Jul 2012 13:49:14 +1000
Subject: [Python-ideas] Moving return above statements [was Re: Syntax
 for defining parametric decorators]
In-Reply-To: <4FFE2A3C.1090905@pearwood.info>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
	<CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
	<4FFE2A3C.1090905@pearwood.info>
Message-ID: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>

On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Looking back at "in return", here's a simple example which doesn't use
> nested functions. Compare:
>
>
> def function(arg):
>     in return value
>     value = process(arg)
>     print("Value is", value)

Like a decorator line, the "in" clause is effectively part of the
subsequent function/class definition - it's not a separate statement.
That's why there's no ending colon. However, I'm now thinking a
leading @ might be appropriate in order to make the decorator parallel
crystal clear.

> I would not describe that as "out-of-order". Seems to me that it is
> precisely in order: first you create the object (a function), then you
> return it. You can't return something before it exists.

No, that's implementation order - it's not "pattern of thought" order.
It's why English permits phrase like "this statement is true, given
that this particular assumption is true". You can forward reference
incidental details and fill them in later.

Sorting is generally a better example:

    x = sorted(container)

OK, this is sorted smallest to largest.

Now, I want to sort according to the second item. I can do this out of
order by introducing a key function before it's clear why I need it:

    def _key(v):
        return v[1]
    x = sorted(container, key=_key)

Or, we can use the awkward itemgetter construct to get things back in
the right order and restore the assignment and sorting operation to
it's rightful place of prominence:

    from operator import itemgetter
    x = sorted(container, key=itemgetter(1))

Using a lambda expression achieves the same goal:

    x = sorted(container, key=lambda v: v[1])

Those are both ugly though, and are restricted to a single expression.
More complicated sorting logic is forced back into using the out of
order form.

PEP 403 is designed to let you pull the sort key out into a trailing
full-fledged function definition:

    @in x = sorted(container, key=f)
    def f(v):
        return v[1]

The important part here is that the container is being sorted and the
result assigned to 'x' - the exact sort key used isn't important to
the overall flow of the algorithm.


> It seems to me that this proposal, and the older PEP 3150, are the ones
> which are out-of-order: you use things before they are defined.

Yes, both PEP 403 and 3150 are out of order from a code *execution*
point of view - but they're designed to match *patterns of thought*
that work that way. It's especially prevalent in callback based
programming - you end up with a lot of one-shot functions that aren't
actually all that important in their own right, what really matters is
the call you're handing them over to.

Sometimes you're only defining a separate function because that's the
only syntax Python has for passing a piece of code to a function, even
though the code is only used once.

Decorators eliminated this problem for the "f = deco(f)" and "C =
deco(C)" cases by baking the ability to make such calls into function
and class definitions. PEP 403 is solely about extending that same
capability to arbitrary simple statements.

> For what it's worth, I have slightly warmed to PEP 3150 and would give it a
> very tentative +0.125:

PEP 3150 died mainly due to "If the implementation is hard to explain,
it's a bad idea", but also due to the fact that you regularly ended up
with double-indents for callback based code. PEP 403 only has to a
support a statement local reference to a single name, which is
comparatively easy.

> (Aside: I actually prefer that bikeshed to be called "where" rather than
> "given".)

PEP 3150 has an entire section devoted to the reasons "where" isn't a
viable choice (not that that PEP stands any realistic chance of ever
being resurrected).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From raymond.hettinger at gmail.com  Thu Jul 12 05:19:20 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 11 Jul 2012 20:19:20 -0700
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <jtlc79$195$1@dough.gmane.org>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jtlc79$195$1@dough.gmane.org>
Message-ID: <6343E5B9-66E7-44B4-B6AE-D16D1B064C56@gmail.com>


On Jul 11, 2012, at 7:19 PM, Terry Reedy wrote:

> For people 'raised' with imperative languages without nested functions and currying, function currying is definately *not* 'self-explaining'. The difficultly is conceptual, not syntactical. I am 99% sure it would result in more confusion, not less.



FWIW, I've taught many Python classes this year and have found the current style easy to teach.

I agree with Terry's assessment that introducing new syntactic trickery would cause more problems than it would solve.

I also agree with Nick's sentiment that we would be better-off with a vocabulary that clearly distinguishes between between decorators and decorator factories.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120711/0fb7e1c9/attachment.html>

From ubershmekel at gmail.com  Thu Jul 12 07:23:22 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 12 Jul 2012 08:23:22 +0300
Subject: [Python-ideas] Moving return above statements [was Re: Syntax
 for defining parametric decorators]
In-Reply-To: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
	<CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
	<4FFE2A3C.1090905@pearwood.info>
	<CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>
Message-ID: <CANSw7KxAfTFBrVdpQNdqamM0xLLtsCacj851OAA=ezVeh_i7EQ@mail.gmail.com>

On Thu, Jul 12, 2012 at 6:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> > Looking back at "in return", here's a simple example which doesn't use
> > nested functions. Compare:
> >
> >
> > def function(arg):
> >     in return value
> >     value = process(arg)
> >     print("Value is", value)
>
> Like a decorator line, the "in" clause is effectively part of the
> subsequent function/class definition - it's not a separate statement.
> That's why there's no ending colon. However, I'm now thinking a
> leading @ might be appropriate in order to make the decorator parallel
> crystal clear.
>
> > I would not describe that as "out-of-order". Seems to me that it is
> > precisely in order: first you create the object (a function), then you
> > return it. You can't return something before it exists.
>
> No, that's implementation order - it's not "pattern of thought" order.
> It's why English permits phrase like "this statement is true, given
> that this particular assumption is true". You can forward reference
> incidental details and fill them in later.
>
> Sorting is generally a better example:
>
>     x = sorted(container)
>
> OK, this is sorted smallest to largest.
>
> Now, I want to sort according to the second item. I can do this out of
> order by introducing a key function before it's clear why I need it:
>
>     def _key(v):
>         return v[1]
>     x = sorted(container, key=_key)
>
> Or, we can use the awkward itemgetter construct to get things back in
> the right order and restore the assignment and sorting operation to
> it's rightful place of prominence:
>
>     from operator import itemgetter
>     x = sorted(container, key=itemgetter(1))
>
> Using a lambda expression achieves the same goal:
>
>     x = sorted(container, key=lambda v: v[1])
>
> Those are both ugly though, and are restricted to a single expression.
> More complicated sorting logic is forced back into using the out of
> order form.
>
> PEP 403 is designed to let you pull the sort key out into a trailing
> full-fledged function definition:
>
>     @in x = sorted(container, key=f)
>     def f(v):
>         return v[1]
>
> The important part here is that the container is being sorted and the
> result assigned to 'x' - the exact sort key used isn't important to
> the overall flow of the algorithm.
>
>
> > It seems to me that this proposal, and the older PEP 3150, are the ones
> > which are out-of-order: you use things before they are defined.
>
> Yes, both PEP 403 and 3150 are out of order from a code *execution*
> point of view - but they're designed to match *patterns of thought*
> that work that way. It's especially prevalent in callback based
> programming - you end up with a lot of one-shot functions that aren't
> actually all that important in their own right, what really matters is
> the call you're handing them over to.
>
> Sometimes you're only defining a separate function because that's the
> only syntax Python has for passing a piece of code to a function, even
> though the code is only used once.
>
> Decorators eliminated this problem for the "f = deco(f)" and "C =
> deco(C)" cases by baking the ability to make such calls into function
> and class definitions. PEP 403 is solely about extending that same
> capability to arbitrary simple statements.
>
> > For what it's worth, I have slightly warmed to PEP 3150 and would give
> it a
> > very tentative +0.125:
>
> PEP 3150 died mainly due to "If the implementation is hard to explain,
> it's a bad idea", but also due to the fact that you regularly ended up
> with double-indents for callback based code. PEP 403 only has to a
> support a statement local reference to a single name, which is
> comparatively easy.
>
> > (Aside: I actually prefer that bikeshed to be called "where" rather than
> > "given".)
>
> PEP 3150 has an entire section devoted to the reasons "where" isn't a
> viable choice (not that that PEP stands any realistic chance of ever
> being resurrected).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


The way I understand this proposal is similar to how MATLAB defines
functions where the names of the variables that are returned are in/near
the function declaration. It's useful on the occasions where you decide
another variable is needed by the caller. Also, it's nice to have
consistency in whatever's returned. The "return" keyword would be strange
though.

There are a few unobvious edge cases here. So I'd have to say -0.1

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120712/5f940d97/attachment.html>

From steve at pearwood.info  Thu Jul 12 09:05:02 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 12 Jul 2012 17:05:02 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <jtlc79$195$1@dough.gmane.org>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jtlc79$195$1@dough.gmane.org>
Message-ID: <20120712070502.GA29188@ando>

On Wed, Jul 11, 2012 at 10:19:01PM -0400, Terry Reedy wrote:

> For people 'raised' with imperative languages without nested functions 
> and currying, function currying is definately *not* 'self-explaining'. 
> The difficultly is conceptual, not syntactical. I am 99% sure it would 
> result in more confusion, not less.

I was raised on Pascal, Fortran and Hypertalk, with some Forth-like 
languages (Forth and HP's RPL). None of which are functional, and I had 
never even heard of currying until *long* after starting with Python.

But the idea of a function creating and returning a function was easy to 
understand once it was suggested to me. Once you have the right language 
to talk about a concept, some pretty complex concepts become simple.

I suspect that most of the difficulty people have is because they don't 
clearly distinguish between a decorator and a decorator-factory. Once 
you make the conceptual leap to a function making a function (a 
decorator, which is a kind of function-factory), it is easy to think 
about a decorator-factory, a decorator-factory-factory, and so on. But 
if you don't cleanly distinguish them, in my opinion it becomes 
confusing and unclear whether you need one inner function or two.

Of course, the book-keeping needed to make this all work is not 
necessarily trivial, but it's just book-keeping. New syntax won't make 
that easier.

Singling out decorator-factories for special treatment is, in my 
opinion, a no-go. If this idea is to be sold, decorator-factories may be 
given as just one use-case for syntax for currying.

But frankly, I don't think we need it, or should want it. Python is not 
Haskell, and while it's great to borrow ideas from Haskell (e.g. list 
comprehensions), it is notable that Python hasn't used the exact same 
syntax as Haskell. Compare:

[toUpper c | c <- s]  -- Haskell
[c.upper() for c in s]  # Python

and tell me that "for c in s" isn't more readable than "| c <- s".

I think syntax for currying

def func(a)(b)(c)(d):
    ...

f = func(1)(2)(3)
f(4)

is too Haskell-ish for my taste. I prefer the Python way:

def func(a, b, c, d):
    ...

f = functools.partial(func, 1, 2, 3)
f(4)

even though it is less concise.

I prefer a stronger tool set in functools, including a way to use 
partial on arguments from the right. Perhaps there is a clean API for 
simplifying decorator factories. That could start as a published recipe 
on (say) ActiveState, and then move to functools if there was demand 
for it.

But as usual, the bar to justify new syntax is rightly set much higher 
than the bar to justify new functionality.


-- 
Steven


From ncoghlan at gmail.com  Thu Jul 12 09:17:06 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 12 Jul 2012 17:17:06 +1000
Subject: [Python-ideas] Syntax for defining parametric decorators
In-Reply-To: <20120712070502.GA29188@ando>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jtlc79$195$1@dough.gmane.org> <20120712070502.GA29188@ando>
Message-ID: <CADiSq7e63BcXKtt+JseEbiqgYf_jLRwbRDwb+Sx8LZugGpKc5A@mail.gmail.com>

On Thu, Jul 12, 2012 at 5:05 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> I prefer a stronger tool set in functools, including a way to use
> partial on arguments from the right.

FWIW, my hope is that the provision of PEP 362 in Python 3.3 will
encourage third parties to explore some richer options for
manipulating callables now that it's much easier to get a clear
definition of their signature.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From stephen at xemacs.org  Thu Jul 12 11:32:45 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 12 Jul 2012 18:32:45 +0900
Subject: [Python-ideas] Moving return above statements [was Re: Syntax
 for defining parametric decorators]
In-Reply-To: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
	<CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
	<4FFE2A3C.1090905@pearwood.info>
	<CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>
Message-ID: <87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:
 > On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info> wrote:
 > > Looking back at "in return", here's a simple example which doesn't use
 > > nested functions. Compare:
 > >
 > >
 > > def function(arg):
 > >     in return value
 > >     value = process(arg)
 > >     print("Value is", value)
 > 
 > Like a decorator line, the "in" clause is effectively part of the
 > subsequent function/class definition - it's not a separate statement.
 > That's why there's no ending colon. However, I'm now thinking a
 > leading @ might be appropriate in order to make the decorator parallel
 > crystal clear.

Indeed I would prefer the leading "@", because the in clause
(syntactically) "decorates" the following definition.

But isn't Steven's example just plain invalid on two counts?  That is,
in the PEP (1) the "in" modifies a function or class definition, not
an arbitrary statement, and (2) it does not have suite scope, it has
statement scope.  Another way of expressing (2) is that the print
statement above is dead code.

With respect to the idea itself, I'm a little concerned that in the
case of the decorator, the idea is to deemphasize the decoration in
favor of focusing on the decorated definition, while in the case of
the "in" clause, the idea is to deemphasize the definition in favor of
focusing on the "decorating" statement.

Perhaps in practice this won't be a problem, though.  For the "in"
clause, the order of presentation will provide the emphasis on the
"decorating" statement.  OTOH, for the decorators, the order of
presentation is no big deal compared to the avoidance of DRY
violation.



From ncoghlan at gmail.com  Thu Jul 12 11:39:44 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 12 Jul 2012 19:39:44 +1000
Subject: [Python-ideas] Moving return above statements [was Re: Syntax
 for defining parametric decorators]
In-Reply-To: <87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com>
	<jte9c4$2ej$1@dough.gmane.org>
	<CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com>
	<4FFE2A3C.1090905@pearwood.info>
	<CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com>
	<87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7cRbuWLhkLMwCNQgCquRu1OzfAeMfOzrt5=6i6r-DoSiw@mail.gmail.com>

On Thu, Jul 12, 2012 at 7:32 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> With respect to the idea itself, I'm a little concerned that in the
> case of the decorator, the idea is to deemphasize the decoration in
> favor of focusing on the decorated definition, while in the case of
> the "in" clause, the idea is to deemphasize the definition in favor of
> focusing on the "decorating" statement.

There are many reasons PEP 403 is something I pick up and improve as
the urge strikes me rather than something I treat with any urgency :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From namdn at socbay.com  Sun Jul 15 08:18:59 2012
From: namdn at socbay.com (=?utf-8?B?xJBpbmg=?= Nho Nam)
Date: Sun, 15 Jul 2012 13:18:59 +0700 (ICT)
Subject: [Python-ideas] How can I get own instance of generator in itself
	code
Message-ID: <33429676.215905.1342333139344.JavaMail.root@socbay.com>

Hi Python Software Foundation!
I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects.
Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? 
detail pseudo code example:

def invoke_callback(url):
	do something here
	self = get myself (invoke_callback generator) instance #how can we do as this line????
	data = yield asynchronous_dowload(url, callback = self.next)
	do something with data

How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list.

def invoke_callback(url, wrapper):
	self = wrapper[0]
	data = yield asynchronous_dowload(url, callback = self.next)
	do something with data

wrapper = []
instance = invoke_callback(url, wrapper)
wrapper.append(instance)
instance.next()

The other way is sending instance to itself, and we invoke twice.

def invoke_callback(url):
	self = yield
	data = yield asynchronous_dowload(url, callback = self.next)
	do something with data
instance = invoke_callback(url)
instance.next()	#first
instance.send(instance)	#second

But the code in both of ways is not nice. Can you add this feature in the next version of python???
Regards
Namdn


From simon.sapin at kozea.fr  Sun Jul 15 11:36:27 2012
From: simon.sapin at kozea.fr (Simon Sapin)
Date: Sun, 15 Jul 2012 11:36:27 +0200
Subject: [Python-ideas] How can I get own instance of generator in
 itself code
In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
Message-ID: <50028F1B.1070105@kozea.fr>

Le 15/07/2012 08:18, ?inh Nho Nam a ?crit :
> def invoke_callback(url):
> 	do something here
> 	self = get myself (invoke_callback generator) instance #how can we do as this line????
> 	data = yield asynchronous_dowload(url, callback = self.next)
> 	do something with data
>
> How can we do as this code. How can I get self variable?. Does the
> language support this feature? I cannot find it in any document in
> python.org. So, I have to solve it by wrapper it in a list.

Hi,

Maybe you could achieve that by inspecting the stack frame, but this is 
generally not recommended.

But taking a step back, I think that the primitive you are looking for 
is a more general coroutine. greenlet can do this, and gevent builds on 
it to add higher level async networking:

http://greenlet.readthedocs.org/en/latest/index.html
http://www.gevent.org/

Regards,
-- 
Simon Sapin


From grosser.meister.morti at gmx.net  Sun Jul 15 15:42:09 2012
From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=)
Date: Sun, 15 Jul 2012 15:42:09 +0200
Subject: [Python-ideas] How can I get own instance of generator in
 itself code
In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
Message-ID: <5002C8B1.5030006@gmx.net>

On 07/15/2012 08:18 AM, ?inh Nho Nam wrote:
> I think that this feature is not supported in classic language as C++, Java, C#..

C# and Mozillas non-standard JavaScript also support generators (maybe not called "generators") and 
the yield statement.


From alexandre.zani at gmail.com  Sun Jul 15 17:58:10 2012
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Sun, 15 Jul 2012 08:58:10 -0700
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
Message-ID: <CAJVMporuyk=HVS1cfarU91WfBEmH1nVuX6WcOS2VZxf6q8Hxog@mail.gmail.com>

On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote:
> Hi Python Software Foundation!
> I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects.
> Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code?
> detail pseudo code example:
>
> def invoke_callback(url):
>         do something here
>         self = get myself (invoke_callback generator) instance #how can we do as this line????
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
>
> How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list.
>
> def invoke_callback(url, wrapper):
>         self = wrapper[0]
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
>
> wrapper = []
> instance = invoke_callback(url, wrapper)
> wrapper.append(instance)
> instance.next()
>
> The other way is sending instance to itself, and we invoke twice.
>
> def invoke_callback(url):
>         self = yield
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
> instance = invoke_callback(url)
> instance.next() #first
> instance.send(instance) #second
>
> But the code in both of ways is not nice. Can you add this feature in the next version of python???
> Regards
> Namdn
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

I would suggest creating a decorator for this:

def self_decorator(f):
  @functools.wraps(f)
  def decorated(*args, **kwargs):
    return f(f, *args, **kwargs)

@self_decorator
def do_stuff(self, a, b, c):
  pass

Should work for any kind of function.

And C/C++ do support such functionality in the sense that you can keep
state in between function invocations:

// This will return 1 the first time it is called, 2, the next time and so on...
int generator() {
  static int num = 0;
  num += 1;
  return num;
}

On a final note, this question more appropriate for python-list.
python-ideas is for the discussion of ideas to change python itself.

Alex


From pyideas at rebertia.com  Mon Jul 16 04:03:54 2012
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 15 Jul 2012 19:03:54 -0700
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
Message-ID: <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>

On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote:
> Hi Python Software Foundation!
> I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects.
> Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code?
<snip>
> How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org.
<snip>
> Can you add this feature in the next version of python???

Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which
would have similarly added nice ways to get the current module, class,
and regular function, was rejected, I wouldn't hold out much hope for
such a feature being added any time soon.
(Wrapping via decorators introduces nontrivial subtlety regarding
whether you really want to refer to the decorated or undecorated
version of an entity.)

Cheers,
Chris


From aquavitae69 at gmail.com  Mon Jul 16 09:12:32 2012
From: aquavitae69 at gmail.com (David Townshend)
Date: Mon, 16 Jul 2012 09:12:32 +0200
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
Message-ID: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>

The problem with PEP 3130 was the PEP itself, not the idea.  I believe Eric
Snow was working on a patch to implement this (see
http://code.activestate.com/lists/python-ideas/11234/ and
http://bugs.python.org/issue12857), but I'm not sure what its current
status is.

David
 On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote:

> On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote:
> > Hi Python Software Foundation!
> > I have learned Python for 2 years, and I love this language very much.
> The structure and grammar are simple and beautiful. I used this language in
> most of my projects.
> > Now, I learn to use generator. This architecture is very intelligent. It
> allows programmer can pause and resume function. I think that this feature
> is not supported in classic language as C++, Java, C#.. In my design, we
> use 1 generator in both tasks: invoke and callback when download one url.
> But I have a small problem when getting instance of generator. How can I
> get own instance of generator in itself code?
> <snip>
> > How can we do as this code. How can I get self variable?. Does the
> language support this feature? I cannot find it in any document in
> python.org.
> <snip>
> > Can you add this feature in the next version of python???
>
> Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which
> would have similarly added nice ways to get the current module, class,
> and regular function, was rejected, I wouldn't hold out much hope for
> such a feature being added any time soon.
> (Wrapping via decorators introduces nontrivial subtlety regarding
> whether you really want to refer to the decorated or undecorated
> version of an entity.)
>
> Cheers,
> Chris
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/05d7206f/attachment.html>

From michelelacchia at gmail.com  Mon Jul 16 10:55:12 2012
From: michelelacchia at gmail.com (Michele Lacchia)
Date: Mon, 16 Jul 2012 01:55:12 -0700 (PDT)
Subject: [Python-ideas] Where has packaging gone?
Message-ID: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com>

Hi,
I read this:
http://www.gossamer-threads.com/lists/python/dev/992979

but at the end nobody posted to it anymore. I would like to know what will
now happen, since as far as I understood, the development of packaging
has stopped and Bento (David Cournapeau library) has appeared.

So what will happen now? Packaging will continue the development? Or
will be Bento replace it? Personally I don't care which tool will become 
standard,
but I do feel the need for a standard to be created. So I would like to 
point out
that to me these three PEPs seem really important, and hope that will be 
taken
into account:

- PEP 345 (http://www.python.org/dev/peps/pep-0345/)
- PEP 376 (http://www.python.org/dev/peps/pep-0376/)
- PEP 386 (http://www.python.org/dev/peps/pep-0386/)

What do you think about all this?

Regards,
Michele Lacchia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/25064587/attachment.html>

From ncoghlan at gmail.com  Mon Jul 16 14:08:30 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Jul 2012 22:08:30 +1000
Subject: [Python-ideas]  Where has packaging gone?
In-Reply-To: <CADiSq7dAyxfA36FaW=GHy56En9udeir1MR+EDWGRcRviY+U4Hg@mail.gmail.com>
References: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com>
	<CADiSq7dAyxfA36FaW=GHy56En9udeir1MR+EDWGRcRviY+U4Hg@mail.gmail.com>
Message-ID: <CADiSq7d9r5OotU-Y=Ki_dG9pfTH56cFf=m_y4cF+DDmAYCpWNw@mail.gmail.com>

(the list address on the original email I replied to was broken)

On Mon, Jul 16, 2012 at 6:55 PM, Michele Lacchia
<michelelacchia at gmail.com> wrote:
> Hi,
> I read this:
> http://www.gossamer-threads.com/lists/python/dev/992979
>
> but at the end nobody posted to it anymore. I would like to know what will
> now happen, since as far as I understood, the development of packaging
> has stopped and Bento (David Cournapeau library) has appeared.

No, the distutils2 project continues and is still being actively
developed. (distutils2 *is* packaging - the name on PyPI is just
different from the name that will be used when it is added to the
standard library). Bento is a project that aims to tackle part of
distutils that distutils2 hasn't really looked at yet: creating a more
flexible build system (or, more accurately, making it easier to hook
into *existing* build systems rather than defining yet another one).

(In fact, the lack of attention to the build system flaws is one of
the big reasons distutils2/packaging was deemed unready for stdlib
inclusion in 3.3)

> So what will happen now? Packaging will continue the development? Or
> will be Bento replace it? Personally I don't care which tool will become
> standard,
> but I do feel the need for a standard to be created.

The point of having documented standards in PEP form is that there can
(and will) be multiple projects that produce and consume those
standard formats.

> - PEP 345 (http://www.python.org/dev/peps/pep-0345/)
> - PEP 376 (http://www.python.org/dev/peps/pep-0376/)
> - PEP 386 (http://www.python.org/dev/peps/pep-0386/)

Those PEPs are *already* standards and are actively supported by
external projects today. One key missing piece is standard library
support, which the packaging module is intended to provide. However,
it was deemed that the project was not quite mature enough for 3.3,
thus it will continue to be developed (with a faster iteration cycle)
as distutils2, with a view to including a more mature version of the
API for 3.4. This will likely take the form of additional PEPs that
more clearly define what is to be added and *why* it is being added,
with one focus being making it easier to build additional tools that
produce and consume metadata in the standard formats, rather then
being a comparatively unwieldy monolithic system like the original
distutils.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From sven at marnach.net  Mon Jul 16 16:49:52 2012
From: sven at marnach.net (Sven Marnach)
Date: Mon, 16 Jul 2012 15:49:52 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
Message-ID: <20120716144952.GK14370@bagheera>

Currently, there is no obvious way to encode a filename in the default
filesystem encoding.  To pipe some filenames to the stdin of a
subprocess, I effectively used

    encoded_name = file_name.encode(sys.getfilesystemencoding())

which mostly worked.  There are cases where this fails, though: on
Linux with LANG=C and filenames that contain non-ASCII characters, for
example, or in any situation where the default filesystem encoding
can't decode a filename.

The correct way to do this seems to be something like

    if sys.platform == "nt":
        errors = "strict"
    else:
        errors = "surrogateescape"
    encoded_name = file_name.encode(sys.getfilesystemencoding()
                                    errors=errors)

I think there should be (1) some documentation on the issue and (2) a
more obvious way to do encode filenames.

1. The most useful reference I could find in the docs is

       http://docs.python.org/dev/c-api/unicode.html#file-system-encoding

   and there is a short paragraph at

       http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables

   The filename encoding applies to basically all Python library
   functions (including built-ins like `open()`) and should probably
   be documented at a more prominent spot.  The "surrogateescape"
   error handler isn't mentioned here

       http://docs.python.org/dev/howto/unicode.html#unicode-filenames

2. There should be some way to access the C API functions for decoding
   and encoding filenames from Python.  I don't have a good idea how
   to do this ? maybe by adding a meta-encoding "filesystem", or by
   adding functions to the standard library.

Did I miss something?  Any thoughts?

Cheers,
    Sven


From solipsis at pitrou.net  Mon Jul 16 17:49:56 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2012 17:49:56 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
References: <20120716144952.GK14370@bagheera>
Message-ID: <20120716174956.162a2df2@pitrou.net>

On Mon, 16 Jul 2012 15:49:52 +0100
Sven Marnach <sven at marnach.net> wrote:
> Currently, there is no obvious way to encode a filename in the default
> filesystem encoding.  To pipe some filenames to the stdin of a
> subprocess, I effectively used
> 
>     encoded_name = file_name.encode(sys.getfilesystemencoding())

Well, how about os.fsencode() and os.fsdecode()?

http://docs.python.org/dev/library/os.html#os.fsencode

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From ericsnowcurrently at gmail.com  Mon Jul 16 18:28:48 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 16 Jul 2012 10:28:48 -0600
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
	<CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
Message-ID: <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com>

Yeah, this is still valuable.  However, at this point it would be a 3.4
addition (_if_ it gets committed).  Updating the patch in the tracker is on
my todo for the next few months.

-eric
On Jul 16, 2012 1:13 AM, "David Townshend" <aquavitae69 at gmail.com> wrote:

> The problem with PEP 3130 was the PEP itself, not the idea.  I believe
> Eric Snow was working on a patch to implement this (see
> http://code.activestate.com/lists/python-ideas/11234/ and
> http://bugs.python.org/issue12857), but I'm not sure what its current
> status is.
>
> David
>  On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote:
>
>> On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote:
>> > Hi Python Software Foundation!
>> > I have learned Python for 2 years, and I love this language very much.
>> The structure and grammar are simple and beautiful. I used this language in
>> most of my projects.
>> > Now, I learn to use generator. This architecture is very intelligent.
>> It allows programmer can pause and resume function. I think that this
>> feature is not supported in classic language as C++, Java, C#.. In my
>> design, we use 1 generator in both tasks: invoke and callback when download
>> one url. But I have a small problem when getting instance of generator. How
>> can I get own instance of generator in itself code?
>> <snip>
>> > How can we do as this code. How can I get self variable?. Does the
>> language support this feature? I cannot find it in any document in
>> python.org.
>> <snip>
>> > Can you add this feature in the next version of python???
>>
>> Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which
>> would have similarly added nice ways to get the current module, class,
>> and regular function, was rejected, I wouldn't hold out much hope for
>> such a feature being added any time soon.
>> (Wrapping via decorators introduces nontrivial subtlety regarding
>> whether you really want to refer to the decorated or undecorated
>> version of an entity.)
>>
>> Cheers,
>> Chris
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/1e714c3b/attachment.html>

From sven at marnach.net  Mon Jul 16 19:04:41 2012
From: sven at marnach.net (Sven Marnach)
Date: Mon, 16 Jul 2012 18:04:41 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <20120716174956.162a2df2@pitrou.net>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
Message-ID: <20120716170441.GL14370@bagheera>

Antoine Pitrou schrieb am Mon, 16. Jul 2012, um 17:49:56 +0200:
> On Mon, 16 Jul 2012 15:49:52 +0100
> Sven Marnach <sven at marnach.net> wrote:
> > Currently, there is no obvious way to encode a filename in the default
> > filesystem encoding.  To pipe some filenames to the stdin of a
> > subprocess, I effectively used
> > 
> >     encoded_name = file_name.encode(sys.getfilesystemencoding())
> 
> Well, how about os.fsencode() and os.fsdecode()?
> 
> http://docs.python.org/dev/library/os.html#os.fsencode

Oh, great, there they are!  I think these functions should be
mentioned in these sections to make them easier to find:

[1]: http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables
[2]: http://docs.python.org/dev/library/sys.html#sys.getfilesystemencoding
[3]: http://docs.python.org/dev/howto/unicode.html#unicode-filenames

I'll post an issue on the issue tracker.

Cheers,
    Sven


From victor.stinner at gmail.com  Mon Jul 16 19:23:59 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Jul 2012 19:23:59 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <20120716170441.GL14370@bagheera>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
Message-ID: <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>

>> Well, how about os.fsencode() and os.fsdecode()?
>>
>> http://docs.python.org/dev/library/os.html#os.fsencode
>
> Oh, great, there they are!  I think these functions should be
> mentioned in these sections to make them easier to find:
>
> [1]: http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables
> [2]: http://docs.python.org/dev/library/sys.html#sys.getfilesystemencoding
> [3]: http://docs.python.org/dev/howto/unicode.html#unicode-filenames
>
> I'll post an issue on the issue tracker.

Hi,

I wrote these functions when I worked in this topic for Python 3. Yes,
it would be great if you write a patch to mention these functions in
the doc.

Someone also complained that the surrogateescape error handler is not
mentionned in any FS related function.

Victor


From and-dev at doxdesk.com  Tue Jul 17 01:00:32 2012
From: and-dev at doxdesk.com (And Clover)
Date: Tue, 17 Jul 2012 00:00:32 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
Message-ID: <50049D10.5040903@doxdesk.com>

On 16/07/12 18:23, Victor Stinner wrote:
 > I wrote these functions when I worked in this topic for Python 3. Yes,
 > it would be great if you write a patch to mention these functions in
 > the doc.

Sure.

But should we be encouraging their use on Windows? I would have thought 
it the best thing to stick with the Unicode string for paths on NT, so 
that the native Win32 Unicode APIs are used instead of the 
ANSI-code-page-bound C stdio. Encoding down to the fsencoding for 
Windows just means that any path including a character that isn't in the 
ANSI CP will fail.

In lieu of some kind of abstract filepath object thatcould represent 
either bytes or str (depending on platform), how about a function that 
takes a str and only encodes it to bytes if the platform requires it?

cheers,

-- 
And Clover
mailto:and at doxdesk.com
http://www.doxdesk.com/
gtalk:chat?jid=bobince at gmail.com



From solipsis at pitrou.net  Tue Jul 17 01:41:24 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 17 Jul 2012 01:41:24 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
Message-ID: <20120717014124.2e824d59@pitrou.net>

On Tue, 17 Jul 2012 00:00:32 +0100
And Clover <and-dev at doxdesk.com> wrote:

> On 16/07/12 18:23, Victor Stinner wrote:
>  > I wrote these functions when I worked in this topic for Python 3. Yes,
>  > it would be great if you write a patch to mention these functions in
>  > the doc.
> 
> Sure.
> 
> But should we be encouraging their use on Windows? I would have thought 
> it the best thing to stick with the Unicode string for paths on NT, so 
> that the native Win32 Unicode APIs are used instead of the 
> ANSI-code-page-bound C stdio. Encoding down to the fsencoding for 
> Windows just means that any path including a character that isn't in the 
> ANSI CP will fail.

Well even under Unix, these functions are only useful for very
specialized cases. For normal usage, PEP 383 guarantees that all
filenames, including theoretically undecodable ones, pass through
properly. When piping filenames between Python processes, you can use
whatever encoding you want (or you can also use json or pickle).

The only remaining use case is sending some filenames to an external
(non-Python) program over a bytes stream, or reading some filenames
emitted by such a program. Here, you need bytes under Windows as well.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From victor.stinner at gmail.com  Tue Jul 17 03:03:24 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Jul 2012 03:03:24 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <50049D10.5040903@doxdesk.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
Message-ID: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>

2012/7/17 And Clover <and-dev at doxdesk.com>:
> But should we be encouraging their use on Windows? I would have thought it
> the best thing to stick with the Unicode string for paths on NT, so that the
> native Win32 Unicode APIs are used instead of the ANSI-code-page-bound C
> stdio. Encoding down to the fsencoding for Windows just means that any path
> including a character that isn't in the ANSI CP will fail.

os.fsencode() should not be used explicitly on Windows.

> In lieu of some kind of abstract filepath object thatcould represent either
> bytes or str (depending on platform), how about a function that takes a str
> and only encodes it to bytes if the platform requires it?

You can use the str (Unicode) type on all platforms with Python 3, so
use os.fsdecode(). os.listdir(str) does return str filenames on any
platform for example.

Victor


From stefan_ml at behnel.de  Tue Jul 17 06:57:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 06:57:43 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
	<CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
Message-ID: <ju2rc7$omg$1@dough.gmane.org>

Victor Stinner, 17.07.2012 03:03:
> 2012/7/17 And Clover:
>> But should we be encouraging their use on Windows? I would have thought it
>> the best thing to stick with the Unicode string for paths on NT, so that the
>> native Win32 Unicode APIs are used instead of the ANSI-code-page-bound C
>> stdio. Encoding down to the fsencoding for Windows just means that any path
>> including a character that isn't in the ANSI CP will fail.
> 
> os.fsencode() should not be used explicitly on Windows.
> 
>> In lieu of some kind of abstract filepath object thatcould represent either
>> bytes or str (depending on platform), how about a function that takes a str
>> and only encodes it to bytes if the platform requires it?
> 
> You can use the str (Unicode) type on all platforms with Python 3, so
> use os.fsdecode(). os.listdir(str) does return str filenames on any
> platform for example.

That's not the main use case I see, though. When talking to C libraries,
for example, they will usually require a byte encoded file path and also
return one. Getting the encoding right in this case is really not trivial.

I would expect that the above functions do "the right thing" also on
Windows here, unless the library really has a win32 specific file API (and
that's not likely).

Stefan



From breamoreboy at yahoo.co.uk  Tue Jul 17 15:20:57 2012
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Tue, 17 Jul 2012 14:20:57 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
	<CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
Message-ID: <ju3or0$rrj$1@dough.gmane.org>

On 17/07/2012 02:03, Victor Stinner wrote:
>
> os.fsencode() should not be used explicitly on Windows.
>
> Victor
>

Should there be a note in the docs to this effect?

-- 
Cheers.

Mark Lawrence.





From ned at nedbatchelder.com  Tue Jul 17 15:59:32 2012
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Tue, 17 Jul 2012 09:59:32 -0400
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <20120716174956.162a2df2@pitrou.net>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
Message-ID: <50056FC4.6040305@nedbatchelder.com>


On 7/16/2012 11:49 AM, Antoine Pitrou wrote:
> On Mon, 16 Jul 2012 15:49:52 +0100
> Sven Marnach <sven at marnach.net> wrote:
>> Currently, there is no obvious way to encode a filename in the default
>> filesystem encoding.  To pipe some filenames to the stdin of a
>> subprocess, I effectively used
>>
>>      encoded_name = file_name.encode(sys.getfilesystemencoding())
> Well, how about os.fsencode() and os.fsdecode()?
>
> http://docs.python.org/dev/library/os.html#os.fsencode
It's too bad these are not called os.path.encode() and os.path.decode(), 
since they fit so nicely into os.path's charter of manipulating strings 
representing file paths.

--Ned.

> Regards
>
> Antoine.
>
>




From victor.stinner at gmail.com  Tue Jul 17 16:31:57 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Jul 2012 16:31:57 +0200
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <50056FC4.6040305@nedbatchelder.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<50056FC4.6040305@nedbatchelder.com>
Message-ID: <CAMpsgwadeOEdQTnFgxQpJBV2bX40wBsFUkO5a_cX_z6OyNDiVg@mail.gmail.com>

>> Well, how about os.fsencode() and os.fsdecode()?
>>
>> http://docs.python.org/dev/library/os.html#os.fsencode
>
> It's too bad these are not called os.path.encode() and os.path.decode(),
> since they fit so nicely into os.path's charter of manipulating strings
> representing file paths.

os.fsencode()/fsdecode() are not specific to filesystems: you can use
these functions to encode/decode command line arguments, environment
variable, text from/to a console (sys.std*), etc.

The "fs" letters from the name comes from the encoding used by these
functions: sys.get*filesystem*encoding().

For example, os.fsencode() used by the subprocess module and
posixpath.expanduser() modules, and os.fsdecode() is used by
os.get_exec_path() and shutil.rmtree().

Victor


From sven at marnach.net  Tue Jul 17 23:25:42 2012
From: sven at marnach.net (Sven Marnach)
Date: Tue, 17 Jul 2012 22:25:42 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
	<CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
Message-ID: <20120717212542.GP14370@bagheera>

Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200:
> os.fsencode() should not be used explicitly on Windows.

What else should I do to pipe filenames to another process?  At least,
os.fsencode() seems to work, even with cyrillic filenames.

Cheers,
    Sven


From python at mrabarnett.plus.com  Tue Jul 17 23:52:57 2012
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 17 Jul 2012 22:52:57 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <20120717212542.GP14370@bagheera>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
	<CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
	<20120717212542.GP14370@bagheera>
Message-ID: <5005DEB9.3020406@mrabarnett.plus.com>

On 17/07/2012 22:25, Sven Marnach wrote:
> Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200:
>> os.fsencode() should not be used explicitly on Windows.
>
> What else should I do to pipe filenames to another process?  At least,
> os.fsencode() seems to work, even with cyrillic filenames.
>
Encode to UTF-8?


From namdn at socbay.com  Wed Jul 18 04:39:53 2012
From: namdn at socbay.com (=?utf-8?B?xJBpbmg=?= Nho Nam)
Date: Wed, 18 Jul 2012 09:39:53 +0700 (ICT)
Subject: [Python-ideas] How can I get own instance of generator in
 itself code
In-Reply-To: <CAJVMporuyk=HVS1cfarU91WfBEmH1nVuX6WcOS2VZxf6q8Hxog@mail.gmail.com>
Message-ID: <10274497.257110.1342579193494.JavaMail.root@socbay.com>

Yes, in my project, i define a decorator to call twice as I said. The code below:

def asynchronous(gen_func):
	class _generator(object):
		def __init__(self, *args, **kwargs):
			self.__generator = gen_func(*args, **kwargs)
			self.__generator.next()
			self.__generator.send(self.__generator)

		#override generator method: __iter__, next, send, ...	
			
		def __iter__(self):
			return self.__generator
		
		def next(self):
			return self.__generator.next()
			
		def send(self, arg):
			return self.__generator.send(arg)

		def close():
			return self.__generator.close()
		
		def __repr__(self):
			return repr(self.__generator)
		
	return _generator
	
	
@asynchronous
def invoke_callback(url):
	code here
	
and we only call is as function, not need call send or next method

g = invoke_callback(url)

But i think the code is still not nice, because any we have to define @asynchronous in any generator.
	




----- Original Message -----
From: "Alexandre Zani" <alexandre.zani at gmail.com>
To: "?inh Nho Nam" <namdn at socbay.com>
Cc: python-ideas at python.org
Sent: Sunday, July 15, 2012 10:58:10 PM
Subject: Re: [Python-ideas] How can I get own instance of generator in itself code

On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote:
> Hi Python Software Foundation!
> I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects.
> Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code?
> detail pseudo code example:
>
> def invoke_callback(url):
>         do something here
>         self = get myself (invoke_callback generator) instance #how can we do as this line????
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
>
> How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list.
>
> def invoke_callback(url, wrapper):
>         self = wrapper[0]
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
>
> wrapper = []
> instance = invoke_callback(url, wrapper)
> wrapper.append(instance)
> instance.next()
>
> The other way is sending instance to itself, and we invoke twice.
>
> def invoke_callback(url):
>         self = yield
>         data = yield asynchronous_dowload(url, callback = self.next)
>         do something with data
> instance = invoke_callback(url)
> instance.next() #first
> instance.send(instance) #second
>
> But the code in both of ways is not nice. Can you add this feature in the next version of python???
> Regards
> Namdn
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

I would suggest creating a decorator for this:

def self_decorator(f):
  @functools.wraps(f)
  def decorated(*args, **kwargs):
    return f(f, *args, **kwargs)

@self_decorator
def do_stuff(self, a, b, c):
  pass

Should work for any kind of function.

And C/C++ do support such functionality in the sense that you can keep
state in between function invocations:

// This will return 1 the first time it is called, 2, the next time and so on...
int generator() {
  static int num = 0;
  num += 1;
  return num;
}

On a final note, this question more appropriate for python-list.
python-ideas is for the discussion of ideas to change python itself.

Alex


From sven at marnach.net  Wed Jul 18 12:36:54 2012
From: sven at marnach.net (Sven Marnach)
Date: Wed, 18 Jul 2012 11:36:54 +0100
Subject: [Python-ideas] Introduce some obvious way to encode and decode
 filenames from Python code
In-Reply-To: <5005DEB9.3020406@mrabarnett.plus.com>
References: <20120716144952.GK14370@bagheera>
	<20120716174956.162a2df2@pitrou.net>
	<20120716170441.GL14370@bagheera>
	<CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com>
	<50049D10.5040903@doxdesk.com>
	<CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com>
	<20120717212542.GP14370@bagheera>
	<5005DEB9.3020406@mrabarnett.plus.com>
Message-ID: <20120718103654.GU14370@bagheera>

MRAB schrieb am Tue, 17. Jul 2012, um 22:52:57 +0100:
> On 17/07/2012 22:25, Sven Marnach wrote:
> >Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200:
> >>os.fsencode() should not be used explicitly on Windows.
> >
> >What else should I do to pipe filenames to another process?  At least,
> >os.fsencode() seems to work, even with cyrillic filenames.
> >
> Encode to UTF-8?

I don't have control over the other process (it's ExifTool in batch
mode), so I have to use whatever encoding is considered the standard
to encode filenames on Windows.  `os.fsencode()` works fine for this,
and Victor answered off-list that it would be fine in this case.

Cheers,
    Sven


From techtonik at gmail.com  Wed Jul 18 19:30:02 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Jul 2012 20:30:02 +0300
Subject: [Python-ideas] '' in 'abc' == True
Message-ID: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>

I've just spotted inconsistency between string and lists handling:

>>> '' in 'abc'
True
>>> '' in 'abc'.split()
False
>>> [] in ['a', 'b', 'c']
False

Why strings here behave differently than other sequence types? Is that
by design?


From matthew.lefavor at nasa.gov  Wed Jul 18 19:38:31 2012
From: matthew.lefavor at nasa.gov (Lefavor, Matthew  (GSFC-582.0)[MICROTEL LLC])
Date: Wed, 18 Jul 2012 12:38:31 -0500
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
Message-ID: <CC2C6BD7.F6AA%matthew.lefavor@nasa.gov>

Strings have a common use case which lists do not: finding
subsequences/substrings.

Consider the following:

>>> string = 'abcdefghijklmnop'
>>> 'def' in string
True
>>> list('def') in list(string)
False

The contains operator ("in") has a different meaning than the contains
operator for a list. A list contains an object if (and only if) that
object is a single element of the list. A string contains another string
if (and only if) the other string is a substring of the first string.


Matthew Lefavor

NASA GSFC [Microtel, LLC]
Mail Code 699.0/Org Code 582.0
matthew.lefavor at nasa.gov
(301) 614-6818 (Desk)
(443) 758-4891 (Cell)





On 7/18/12 1:30 PM, "anatoly techtonik" <techtonik at gmail.com> wrote:

>I've just spotted inconsistency between string and lists handling:
>
>>>> '' in 'abc'
>True
>>>> '' in 'abc'.split()
>False
>>>> [] in ['a', 'b', 'c']
>False
>
>Why strings here behave differently than other sequence types? Is that
>by design?
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>http://mail.python.org/mailman/listinfo/python-ideas



From masklinn at masklinn.net  Wed Jul 18 19:43:27 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 18 Jul 2012 19:43:27 +0200
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
Message-ID: <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>

On 2012-07-18, at 19:30 , anatoly techtonik wrote:

> I've just spotted inconsistency between string and lists handling:
> 
>>>> '' in 'abc'
> True
>>>> '' in 'abc'.split()
> False
>>>> [] in ['a', 'b', 'c']
> False
> 
> Why strings here behave differently than other sequence types? Is that
> by design?

Erm? yes? `in` would not be very useful for strings if you could only use
it to check for a single character would it?

From masklinn at masklinn.net  Wed Jul 18 19:58:17 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 18 Jul 2012 19:58:17 +0200
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
Message-ID: <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>

On 2012-07-18, at 19:43 , Masklinn wrote:
> On 2012-07-18, at 19:30 , anatoly techtonik wrote:
> 
>> I've just spotted inconsistency between string and lists handling:
>> 
>>>>> '' in 'abc'
>> True
>>>>> '' in 'abc'.split()
>> False
>>>>> [] in ['a', 'b', 'c']
>> False
>> 
>> Why strings here behave differently than other sequence types? Is that
>> by design?
> 
> Erm? yes? `in` would not be very useful for strings if you could only use
> it to check for a single character would it?

in fact, things used to work that way in older Python, this was
specifically changed to the current behavior *as noted in the documentation*:

> When s is a string or Unicode string object the in and not in
> operations act like a substring test. In Python versions before 2.3, x
> had to be a string of length 1. In Python 2.3 and beyond, x may be a
> string of any length.

A Python string, you may want to note, is a string. Not a sequence of
characters. The first item of a 1-character string is itself, all basic
(step-less) slices of a string are contained in itself (including itself
and the empty string), you can infinitely get the first item of a
non-empty string, and I'm sure I'm missing plenty.

From jeanpierreda at gmail.com  Wed Jul 18 20:06:59 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Wed, 18 Jul 2012 14:06:59 -0400
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
Message-ID: <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>

On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
> A Python string, you may want to note, is a string. Not a sequence of
> characters.

It's both (with the caveat that, in Python, a character is just a
string of length 1).

(See: http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
)

-- Devin


From masklinn at masklinn.net  Wed Jul 18 20:16:18 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 18 Jul 2012 20:16:18 +0200
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
Message-ID: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>

On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>> A Python string, you may want to note, is a string. Not a sequence of
>> characters.
> 
> It's both (with the caveat that, in Python, a character is just a
> string of length 1).

That's playing with words, especially comparing strings with Python 3
binaries which *do* actually have a separate "character" type
(reified to an integer).

So Python strings don't have reified characters, a string's item and a
slice of size 1 are essentially identical which is pretty much unique
to them (as far as my knowledge of Python's sequences go).

Which is not a bad thing, mind you, it makes working with strings much
more pleasant.


From jeanpierreda at gmail.com  Wed Jul 18 20:31:19 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Wed, 18 Jul 2012 14:31:19 -0400
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
Message-ID: <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com>

On Wed, Jul 18, 2012 at 2:16 PM, Masklinn <masklinn at masklinn.net> wrote:
>> It's both (with the caveat that, in Python, a character is just a
>> string of length 1).
>
> That's playing with words, especially comparing strings with Python 3
> binaries which *do* actually have a separate "character" type
> (reified to an integer).

No it isn't. Strings are adherents to the sequence protocol. The
Python datatype reference echoes what I said, nearly exactly.

http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy

> So Python strings don't have reified characters, a string's item and a
> slice of size 1 are essentially identical which is pretty much unique
> to them (as far as my knowledge of Python's sequences go).

Nothing about that feature makes them not-sequences; instead, it makes
them a rather special kind of sequence.

-- Devin


From masklinn at masklinn.net  Wed Jul 18 21:02:36 2012
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 18 Jul 2012 21:02:36 +0200
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com>
Message-ID: <285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net>

On 2012-07-18, at 20:31 , Devin Jeanpierre wrote:

> On Wed, Jul 18, 2012 at 2:16 PM, Masklinn <masklinn at masklinn.net> wrote:
>>> It's both (with the caveat that, in Python, a character is just a
>>> string of length 1).
>> 
>> That's playing with words, especially comparing strings with Python 3
>> binaries which *do* actually have a separate "character" type
>> (reified to an integer).
> 
> No it isn't. Strings are adherents to the sequence protocol. The
> Python datatype reference echoes what I said, nearly exactly.

This has no relevance to my messages, I have not claimed anywhere that
strings weren't sequences.

>> So Python strings don't have reified characters, a string's item and a
>> slice of size 1 are essentially identical which is pretty much unique
>> to them (as far as my knowledge of Python's sequences go).
> 
> Nothing about that feature makes them not-sequences; instead, it makes
> them a rather special kind of sequence.

I'm not sure why you're saying that. Again, I have never once claimed they
were not sequences (quite the opposite in fact). Why the strawmanning?


From ethan at stoneleaf.us  Wed Jul 18 21:32:20 2012
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 18 Jul 2012 12:32:20 -0700
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
Message-ID: <50070F44.5080509@stoneleaf.us>

Masklinn wrote:
> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>>> A Python string, you may want to note, is a string. Not a sequence of
>>> characters.
>> It's both (with the caveat that, in Python, a character is just a
>> string of length 1).
> 
> That's playing with words, especially comparing strings with Python 3
> binaries which *do* actually have a separate "character" type
> (reified to an integer).

Python 3 does not have a 'character' type; it has 'str' which is made up 
of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).

~Ethan~


From jeanpierreda at gmail.com  Wed Jul 18 22:02:29 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Wed, 18 Jul 2012 16:02:29 -0400
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com>
	<285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net>
Message-ID: <CABicbJJWh00=vuJCuMaXH+5BHC4fGFjK_X+GQUNg9kiQX_1t-g@mail.gmail.com>

On Wed, Jul 18, 2012 at 3:02 PM, Masklinn <masklinn at masklinn.net> wrote:
> This has no relevance to my messages, I have not claimed anywhere that
> strings weren't sequences.

Sorry, I misinterpreted what you said. I'm tired. :(

-- Devin


From aquavitae69 at gmail.com  Fri Jul 20 18:11:52 2012
From: aquavitae69 at gmail.com (David Townshend)
Date: Fri, 20 Jul 2012 18:11:52 +0200
Subject: [Python-ideas] __sort__ special member
Message-ID: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>

I was quite surprised to find that nobody seems to have suggested this
before, because it seems like an obvious idea. Basically, add a special
method __sort__ which, if specified, is used when sorted() is called. I can
think of two immediate use cases:

1. When an object wants sorted() to return something other than a list,
e.g. dict.__sort__ could return an OrderedDict.
2. When there is a more efficient method of sorting a specific sequence.
E.g. sorting a range object should be trivial.

Is there some obvious reason why nobody has suggested this before?  Is it
worth pursuing?

David.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/98e29be9/attachment.html>

From phd at phdru.name  Fri Jul 20 18:23:11 2012
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 20 Jul 2012 20:23:11 +0400
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
Message-ID: <20120720162311.GA13159@iskra.aviel.ru>

On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com> wrote:
> I was quite surprised to find that nobody seems to have suggested this
> before, because it seems like an obvious idea. Basically, add a special
> method __sort__ which, if specified, is used when sorted() is called. I can
> think of two immediate use cases:
> 
> 1. When an object wants sorted() to return something other than a list,
> e.g. dict.__sort__ could return an OrderedDict.
> 2. When there is a more efficient method of sorting a specific sequence.
> E.g. sorting a range object should be trivial.
> 
> Is there some obvious reason why nobody has suggested this before?  Is it
> worth pursuing?

   Because it's too application-specific?

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.


From matthew.lefavor at nasa.gov  Fri Jul 20 18:22:46 2012
From: matthew.lefavor at nasa.gov (Lefavor, Matthew  (GSFC-582.0)[MICROTEL LLC])
Date: Fri, 20 Jul 2012 11:22:46 -0500
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
Message-ID: <CC2EFC43.F815%matthew.lefavor@nasa.gov>

With regard to point 1: I think that changing the return type of sorted() would cause nasty surprises. If you can no longer assume that calling sorted() on any arbitrary container is going to return an indexed list, then we've lost all the advantages of Python's "duck typing."

With regard to point 2: How would this interact with the existing signature of the sorted() function? The sorted() function, as it is right now, is responsible not just for the sort algorithm, but also for defining how comparisons should work. If the comparison keys are changed, then you can't assume any sorting operation is trivial anymore. Consider the following, using a range object:

>>> import math
>>> sorted(range(10), key=math.sin)
[5, 4, 6, 0, 3, 9, 7, 1, 2, 8]

Matthew Lefavor

From: David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>>
Date: Friday, July 20, 2012 12:11 PM
To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>>
Subject: [Python-ideas] __sort__ special member


I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases:

1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict.
2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial.

Is there some obvious reason why nobody has suggested this before?  Is it worth pursuing?

David.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/0f8410f6/attachment.html>

From dreamingforward at gmail.com  Fri Jul 20 18:44:26 2012
From: dreamingforward at gmail.com (Mark Adam)
Date: Fri, 20 Jul 2012 11:44:26 -0500
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <20120720162311.GA13159@iskra.aviel.ru>
References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
	<20120720162311.GA13159@iskra.aviel.ru>
Message-ID: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com>

On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name> wrote:

> On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <
> aquavitae69 at gmail.com> wrote:
> > I was quite surprised to find that nobody seems to have suggested this
> > before, because it seems like an obvious idea. Basically, add a special
> > method __sort__ which, if specified, is used when sorted() is called. I
> can
> > think of two immediate use cases:
> >
> > 1. When an object wants sorted() to return something other than a list,
> > e.g. dict.__sort__ could return an OrderedDict.
> > 2. When there is a more efficient method of sorting a specific sequence.
> > E.g. sorting a range object should be trivial.
> >
> > Is there some obvious reason why nobody has suggested this before?  Is it
> > worth pursuing?
>
>    Because it's too application-specific?
>
> What are you talking about.  This would solve the issue of ordering for
dicts that became contentious a few years ago.  If people don't like the
arbitrary ordering, they can subclass it.

mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/8929f1e6/attachment.html>

From matthew.lefavor at nasa.gov  Fri Jul 20 18:54:43 2012
From: matthew.lefavor at nasa.gov (Lefavor, Matthew  (GSFC-582.0)[MICROTEL LLC])
Date: Fri, 20 Jul 2012 11:54:43 -0500
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com>
Message-ID: <CC2F04A9.F85B%matthew.lefavor@nasa.gov>

I'm confused: how is this related to the question of dictionary ordering? Calling sorted() on a dict orders all the keys as it stands right now. Arbitrary dictionary ordering is only an issue when iterating over a dictionary. That could already be solved by subclassing dict and overriding the __iter__ method to return the keys in some sort of order.

Matthew Lefavor


From: Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>>
Date: Friday, July 20, 2012 12:44 PM
To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>>
Subject: Re: [Python-ideas] __sort__ special member

On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name<mailto:phd at phdru.name>> wrote:
On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> wrote:
> I was quite surprised to find that nobody seems to have suggested this
> before, because it seems like an obvious idea. Basically, add a special
> method __sort__ which, if specified, is used when sorted() is called. I can
> think of two immediate use cases:
>
> 1. When an object wants sorted() to return something other than a list,
> e.g. dict.__sort__ could return an OrderedDict.
> 2. When there is a more efficient method of sorting a specific sequence.
> E.g. sorting a range object should be trivial.
>
> Is there some obvious reason why nobody has suggested this before?  Is it
> worth pursuing?

   Because it's too application-specific?

What are you talking about.  This would solve the issue of ordering for dicts that became contentious a few years ago.  If people don't like the arbitrary ordering, they can subclass it.

mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/c6ded7a8/attachment.html>

From matthew.lefavor at nasa.gov  Fri Jul 20 18:29:29 2012
From: matthew.lefavor at nasa.gov (Lefavor, Matthew  (GSFC-582.0)[MICROTEL LLC])
Date: Fri, 20 Jul 2012 11:29:29 -0500
Subject: [Python-ideas]  __sort__ special member
In-Reply-To: <CC2EFC43.F815%matthew.lefavor@nasa.gov>
Message-ID: <CC2EFF84.F82B%matthew.lefavor@nasa.gov>

(I apologize if this sent twice, my outbox is acting screwy).

With regard to point 1: I think that changing the return type of sorted() would cause nasty surprises. If you can no longer assume that calling sorted() on any arbitrary container is going to return an indexed list, then we've lost all the advantages of Python's "duck typing."

With regard to point 2: How would this interact with the existing signature of the sorted() function? The sorted() function, as it is right now, is responsible not just for the sort algorithm, but also for defining how comparisons should work. If the comparison keys are changed, then you can't assume any sorting operation is trivial anymore. Consider the following, using a range object:

>>> import math
>>> sorted(range(10), key=math.sin)
[5, 4, 6, 0, 3, 9, 7, 1, 2, 8]

Matthew Lefavor

From: David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>>
Date: Friday, July 20, 2012 12:11 PM
To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>>
Subject: [Python-ideas] __sort__ special member


I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases:

1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict.
2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial.

Is there some obvious reason why nobody has suggested this before?  Is it worth pursuing?

David.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/5f09290c/attachment.html>

From dreamingforward at gmail.com  Fri Jul 20 20:29:25 2012
From: dreamingforward at gmail.com (Mark Adam)
Date: Fri, 20 Jul 2012 13:29:25 -0500
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com>
References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
	<20120720162311.GA13159@iskra.aviel.ru>
	<CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com>
Message-ID: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>

On Fri, Jul 20, 2012 at 11:44 AM, Mark Adam <dreamingforward at gmail.com>wrote:

> On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name> wrote:
>
>> On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <
>> aquavitae69 at gmail.com> wrote:
>> > I was quite surprised to find that nobody seems to have suggested this
>> > before, because it seems like an obvious idea. Basically, add a special
>> > method __sort__ which, if specified, is used when sorted() is called. I
>> can
>> > think of two immediate use cases:
>> >
>> > 1. When an object wants sorted() to return something other than a list,
>> > e.g. dict.__sort__ could return an OrderedDict.
>> > 2. When there is a more efficient method of sorting a specific sequence.
>> > E.g. sorting a range object should be trivial.
>> >
>> > Is there some obvious reason why nobody has suggested this before?  Is
>> it
>> > worth pursuing?
>>
>>    Because it's too application-specific?
>>
>> What are you talking about.  This would solve the issue of ordering for
> dicts that became contentious a few years ago.  If people don't like the
> arbitrary ordering, they can subclass it.
>
> Sorry, I'll tone down my words.  But this makes more sense than sort() as
a built-in.  In theory, sort is for collections, not just any object.  So
it could belong to an abstract collections base type.   This would allow
users to select lexical orderings, for example, rather than ASCII.  It
seems strange that no one thought of it before(it that true?).

mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/c6c8d0f7/attachment.html>

From matthew.lefavor at nasa.gov  Fri Jul 20 20:55:34 2012
From: matthew.lefavor at nasa.gov (Lefavor, Matthew  (GSFC-582.0)[MICROTEL LLC])
Date: Fri, 20 Jul 2012 13:55:34 -0500
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>
Message-ID: <CC2F1EF4.F87E%matthew.lefavor@nasa.gov>

The sorted() function works on any object supporting an __iter__ method. It does not just sort "any object"; if the argument is not iterable, sorted() will raise a TypeError.

If you insist on talking about types, you could say it operates on any instance of the abstract class collections.Iterable. This includes generators. For example:

>>> import collections
>>> def gen():
...     yield 3
...     yield 2
...     yield 1
...
>>> isinstance(gen(), collections.Iterable)
True
>>> sorted(gen())
[1, 2, 3]

As for your other point, users can already support whatever ordering they wish using the "key" argument to the sorted() function. For example, lexicographic ordering is already supported by using str.lower as the key function. For example:

>>> test_list = ["python 3", "PYTHON 4", "Python 2", "pYtHoN 1"]
>>> sorted(test_list)
['PYTHON 4', 'Python 2', 'pYtHoN 1', 'python 3']
>>> sorted(test_list, key=str.lower)
['pYtHoN 1', 'Python 2', 'python 3', 'PYTHON 4']

If you have an object for which you wanted to specify your own default comparison behavior, you could do so by specifying your own implementations of __lt__ or __eq__.

Matthew Lefavor


From: Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>>
Date: Friday, July 20, 2012 2:29 PM
To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>>
Subject: Re: [Python-ideas] __sort__ special member

On Fri, Jul 20, 2012 at 11:44 AM, Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>> wrote:
On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name<mailto:phd at phdru.name>> wrote:
On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> wrote:
> I was quite surprised to find that nobody seems to have suggested this
> before, because it seems like an obvious idea. Basically, add a special
> method __sort__ which, if specified, is used when sorted() is called. I can
> think of two immediate use cases:
>
> 1. When an object wants sorted() to return something other than a list,
> e.g. dict.__sort__ could return an OrderedDict.
> 2. When there is a more efficient method of sorting a specific sequence.
> E.g. sorting a range object should be trivial.
>
> Is there some obvious reason why nobody has suggested this before?  Is it
> worth pursuing?

   Because it's too application-specific?

What are you talking about.  This would solve the issue of ordering for dicts that became contentious a few years ago.  If people don't like the arbitrary ordering, they can subclass it.

Sorry, I'll tone down my words.  But this makes more sense than sort() as a built-in.  In theory, sort is for collections, not just any object.  So it could belong to an abstract collections base type.   This would allow users to select lexical orderings, for example, rather than ASCII.  It seems strange that no one thought of it before(it that true?).

mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/7bb06210/attachment.html>

From g.brandl at gmx.net  Fri Jul 20 22:56:08 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 20 Jul 2012 22:56:08 +0200
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <50070F44.5080509@stoneleaf.us>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<50070F44.5080509@stoneleaf.us>
Message-ID: <jucgl8$lb$1@dough.gmane.org>

On 07/18/2012 09:32 PM, Ethan Furman wrote:
> Masklinn wrote:
>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>>>> A Python string, you may want to note, is a string. Not a sequence of
>>>> characters.
>>> It's both (with the caveat that, in Python, a character is just a
>>> string of length 1).
>> 
>> That's playing with words, especially comparing strings with Python 3
>> binaries which *do* actually have a separate "character" type
>> (reified to an integer).
> 
> Python 3 does not have a 'character' type; it has 'str' which is made up 
> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).

That's what he said.  Could we stop the annoying "but I know it better
than you without reading your message" please?

Georg



From ethan at stoneleaf.us  Fri Jul 20 23:18:20 2012
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 20 Jul 2012 14:18:20 -0700
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <jucgl8$lb$1@dough.gmane.org>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>	<50070F44.5080509@stoneleaf.us>
	<jucgl8$lb$1@dough.gmane.org>
Message-ID: <5009CB1C.4050201@stoneleaf.us>

Georg Brandl wrote:
> On 07/18/2012 09:32 PM, Ethan Furman wrote:
>> Masklinn wrote:
>>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
>>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>>>>> A Python string, you may want to note, is a string. Not a sequence of
>>>>> characters.
 >>>>
>>>> It's both (with the caveat that, in Python, a character is just a
>>>> string of length 1).
 >>>
>>> That's playing with words, especially comparing strings with Python 3
>>> binaries which *do* actually have a separate "character" type
>>> (reified to an integer).
 >>
>> Python 3 does not have a 'character' type; it has 'str' which is made up 
>> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).
> 
> That's what he said.  Could we stop the annoying "but I know it better
> than you without reading your message" please?

I am having trouble equating what I said with with Masklinn said. 
Perhaps you could explain how they say the same thing instead of 
assuming I didn't read his message?

~Ethan~


From andrew.svetlov at gmail.com  Fri Jul 20 23:19:36 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sat, 21 Jul 2012 00:19:36 +0300
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <jucgl8$lb$1@dough.gmane.org>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org>
Message-ID: <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com>

Masklinn's explanation is comprehensive clean to me.

On Fri, Jul 20, 2012 at 11:56 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> On 07/18/2012 09:32 PM, Ethan Furman wrote:
>> Masklinn wrote:
>>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
>>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>>>>> A Python string, you may want to note, is a string. Not a sequence of
>>>>> characters.
>>>> It's both (with the caveat that, in Python, a character is just a
>>>> string of length 1).
>>>
>>> That's playing with words, especially comparing strings with Python 3
>>> binaries which *do* actually have a separate "character" type
>>> (reified to an integer).
>>
>> Python 3 does not have a 'character' type; it has 'str' which is made up
>> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).
>
> That's what he said.  Could we stop the annoying "but I know it better
> than you without reading your message" please?
>
> Georg
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
Thanks,
Andrew Svetlov


From solipsis at pitrou.net  Fri Jul 20 23:27:10 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Jul 2012 23:27:10 +0200
Subject: [Python-ideas] '' in 'abc' == True
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org>
	<5009CB1C.4050201@stoneleaf.us>
Message-ID: <20120720232710.55a85f11@pitrou.net>

On Fri, 20 Jul 2012 14:18:20 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:

> Georg Brandl wrote:
> > On 07/18/2012 09:32 PM, Ethan Furman wrote:
> >> Masklinn wrote:
> >>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
> >>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
> >>>>> A Python string, you may want to note, is a string. Not a sequence of
> >>>>> characters.
>  >>>>
> >>>> It's both (with the caveat that, in Python, a character is just a
> >>>> string of length 1).
>  >>>
> >>> That's playing with words, especially comparing strings with Python 3
> >>> binaries which *do* actually have a separate "character" type
> >>> (reified to an integer).
>  >>
> >> Python 3 does not have a 'character' type; it has 'str' which is made up 
> >> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).
> > 
> > That's what he said.  Could we stop the annoying "but I know it better
> > than you without reading your message" please?
> 
> I am having trouble equating what I said with with Masklinn said. 
> Perhaps you could explain how they say the same thing instead of 
> assuming I didn't read his message?

"Python 3 binaries" probably means "Python 3 bytes objects" above.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From ncoghlan at gmail.com  Sat Jul 21 01:19:50 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 21 Jul 2012 09:19:50 +1000
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CC2F1EF4.F87E%matthew.lefavor@nasa.gov>
References: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>
	<CC2F1EF4.F87E%matthew.lefavor@nasa.gov>
Message-ID: <CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com>

Because the decision of how sorting should occur is up to the consumer, not
the provider. sorted() has the contract to produce a list, using an
algorithm that makes a lot of guarantees about the results. If objects are
allowed to override sorting completely, then those guarantees are no longer
possible.

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120721/888b5fdb/attachment.html>

From raymond.hettinger at gmail.com  Sat Jul 21 02:07:43 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 20 Jul 2012 17:07:43 -0700
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com>
References: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>
	<CC2F1EF4.F87E%matthew.lefavor@nasa.gov>
	<CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com>
Message-ID: <56A2427D-03FB-4CB2-BC38-4BDDBC49BEB3@gmail.com>


On Jul 20, 2012, at 4:19 PM, Nick Coghlan wrote:

> Because the decision of how sorting should occur is up to the consumer, not the provider. sorted() has the contract to produce a list, using an algorithm that makes a lot of guarantees about the results. If objects are allowed to override sorting completely, then those guarantees are no longer possible.

Well said.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/090710d0/attachment.html>

From steve at pearwood.info  Sat Jul 21 13:19:24 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 21 Jul 2012 21:19:24 +1000
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>	<50070F44.5080509@stoneleaf.us>
	<jucgl8$lb$1@dough.gmane.org>
	<CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com>
Message-ID: <500A903C.7020405@pearwood.info>

Andrew Svetlov wrote:
> Masklinn's explanation is comprehensive clean to me.

I'm glad that it's clear to someone, because to me the straight-forward, 
literal meaning of Masklinn's explanation (that Python 3 has a character type, 
and they're integers) is wrong. Python has no built-in "Char" type, under any 
spelling, let alone one which is also a subset of int. The non-literal meaning 
is hard to understand. I *guess* that Masklinn is trying to get across that 
Python 3 strings are Unicode strings, and characters in Unicode are actually 
code points, which are implemented at the C level as integers. If not that, I 
have no idea.

I've more or less forgotten why this was important, but I am enjoying watching 
people try to out-pedant each other :)



> On Fri, Jul 20, 2012 at 11:56 PM, Georg Brandl <g.brandl at gmx.net> wrote:
>> On 07/18/2012 09:32 PM, Ethan Furman wrote:
>>> Masklinn wrote:
>>>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote:
>>>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote:
>>>>>> A Python string, you may want to note, is a string. Not a sequence of
>>>>>> characters.
>>>>> It's both (with the caveat that, in Python, a character is just a
>>>>> string of length 1).
>>>> That's playing with words, especially comparing strings with Python 3
>>>> binaries which *do* actually have a separate "character" type
>>>> (reified to an integer).
>>> Python 3 does not have a 'character' type; it has 'str' which is made up
>>> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly).
>> That's what he said.  Could we stop the annoying "but I know it better
>> than you without reading your message" please?



-- 
Steven


From jeanpierreda at gmail.com  Sat Jul 21 15:59:05 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 21 Jul 2012 09:59:05 -0400
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <500A903C.7020405@pearwood.info>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>
	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>
	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>
	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>
	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>
	<50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org>
	<CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com>
	<500A903C.7020405@pearwood.info>
Message-ID: <CABicbJKpzgUcxH1u4O+SR9natMcDtJf7sytFpV8B3xtUqqFZFg@mail.gmail.com>

On Sat, Jul 21, 2012 at 7:19 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Andrew Svetlov wrote:
>>
>> Masklinn's explanation is comprehensive clean to me.
>
>
> I'm glad that it's clear to someone, because to me the straight-forward,
> literal meaning of Masklinn's explanation (that Python 3 has a character
> type, and they're integers) is wrong. Python has no built-in "Char" type,
> under any spelling, let alone one which is also a subset of int. The
> non-literal meaning is hard to understand. I *guess* that Masklinn is trying
> to get across that Python 3 strings are Unicode strings, and characters in
> Unicode are actually code points, which are implemented at the C level as
> integers. If not that, I have no idea.

You're pretty far off. He was talking about bytes objects, not str objects.

-- Devin


From matt at whoosh.ca  Sat Jul 21 20:59:52 2012
From: matt at whoosh.ca (Matt Chaput)
Date: Sat, 21 Jul 2012 14:59:52 -0400
Subject: [Python-ideas] Atomic file.get(offset, length)
Message-ID: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>

I wish Python binary file objects had an atomic seek-read method, so I wouldn't have to perform my own locking everywhere to prevent other threads from moving the file pointer between seek and read. Is this something that can be bubbled up from the underlying platform? I think the Linux C equivalent is pread. I also think Java has something like this but can't find a reference now.

Or does this exist and I missed it? (On a mmap file this is trivial, of course.) Has this been discussed before?

Cheers,

Matt



From tjreedy at udel.edu  Sat Jul 21 21:35:04 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 21 Jul 2012 15:35:04 -0400
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
Message-ID: <juf09k$njc$1@dough.gmane.org>

On 7/21/2012 2:59 PM, Matt Chaput wrote:
> I wish Python binary file objects had an atomic seek-read method, so
> I wouldn't have to perform my own locking everywhere to prevent other
> threads from moving the file pointer between seek and read.

If you are reading a file from multiple threads, I suggest you write 
your own seek_and_read_with_locks function that does exactly what you 
need in one place. Or add a .readx method to a subclass.

 > Is this something that can be bubbled up from the underlying
 > platform? I think the Linux C equivalent is pread.

If there is a standard posix function that is not yet wrapped in os, you 
can propose its addition. But some research to see has widespread and 
actually standardized it is.

-- 
Terry Jan Reedy



From storchaka at gmail.com  Sat Jul 21 21:36:17 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 21 Jul 2012 22:36:17 +0300
Subject: [Python-ideas] '' in 'abc' == True
In-Reply-To: <500A903C.7020405@pearwood.info>
References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com>	<CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net>	<8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net>	<CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com>	<192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net>	<50070F44.5080509@stoneleaf.us>
	<jucgl8$lb$1@dough.gmane.org>
	<CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com>
	<500A903C.7020405@pearwood.info>
Message-ID: <juf0bj$o0a$1@dough.gmane.org>

On 21.07.12 14:19, Steven D'Aprano wrote:
> I'm glad that it's clear to someone, because to me the straight-forward,
> literal meaning of Masklinn's explanation (that Python 3 has a character
> type, and they're integers) is wrong.

Yes, but Masklinn did not claim this.



From guido at python.org  Sat Jul 21 22:35:21 2012
From: guido at python.org (Guido van Rossum)
Date: Sat, 21 Jul 2012 13:35:21 -0700
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <juf09k$njc$1@dough.gmane.org>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
Message-ID: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>

On Sat, Jul 21, 2012 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/21/2012 2:59 PM, Matt Chaput wrote:
>>
>> I wish Python binary file objects had an atomic seek-read method, so
>> I wouldn't have to perform my own locking everywhere to prevent other
>> threads from moving the file pointer between seek and read.
>
>
> If you are reading a file from multiple threads, I suggest you write your
> own seek_and_read_with_locks function that does exactly what you need in one
> place. Or add a .readx method to a subclass.
>
>
>> Is this something that can be bubbled up from the underlying
>> platform? I think the Linux C equivalent is pread.
>
> If there is a standard posix function that is not yet wrapped in os, you can
> propose its addition. But some research to see has widespread and actually
> standardized it is.

"man pread" on OS/X suggests it exists there too. I presume the use
case is to have a large data file open for reading by multiple
threads. This is a reasonable use case and it makes some sense to
extend our binary readable streams (buffered and unbuffered) with an
API for this purpose.

However, it's probably just efficient to just have a separate open
stream per thread -- I doubt that open file descriptors are scarcer
resources than threads, and I presume the kernel will happily share
any buffering it does on behalf of multiple open files referencing the
same file. If you're worried about the buffer space, the default
buffer size is 8K, which is hardly worth mentioning compared to the
default thread stack allocation. Depending on your use case you may
get away with an unbuffered stream just fine.

This approach seems better than implementing something using locks
(since the locks create contention that is not inherent in the
problem) and is available right now, without waiting for Python 3.4 to
be released...

-- 
--Guido van Rossum (python.org/~guido)


From jeanpierreda at gmail.com  Sun Jul 22 03:08:50 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 21 Jul 2012 21:08:50 -0400
Subject: [Python-ideas] __sort__ special member
In-Reply-To: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>
References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com>
	<20120720162311.GA13159@iskra.aviel.ru>
	<CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com>
	<CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com>
Message-ID: <CABicbJK_Mgry5HzM5XMU4tUBJfTWiQ80YGpm9VbT+kY=i9=1WQ@mail.gmail.com>

On Fri, Jul 20, 2012 at 2:29 PM, Mark Adam <dreamingforward at gmail.com> wrote:
> Sorry, I'll tone down my words.  But this makes more sense than sort() as a
> built-in.  In theory, sort is for collections, not just any object.  So it
> could belong to an abstract collections base type.   This would allow users
> to select lexical orderings, for example, rather than ASCII.  It seems
> strange that no one thought of it before(it that true?).

FWIW the sort method (as in list.sort()) is polymorphic, just nobody
bothers implementing it.

And how many interesting specializations are there? At best I can only
think of linear-time sorts, but it's pretty rare you'll know your
collection only contains discrete values.

-- Devin


From michelelacchia at gmail.com  Sun Jul 22 21:58:34 2012
From: michelelacchia at gmail.com (Michele Lacchia)
Date: Sun, 22 Jul 2012 12:58:34 -0700 (PDT)
Subject: [Python-ideas] Where has packaging gone?
In-Reply-To: <CAPkN8xJ_776rx0nwA_zYoMCNLd82n9pnfepb+2rOrOQwNSFgCQ@mail.gmail.com>
References: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com>
	<CAPkN8xJ_776rx0nwA_zYoMCNLd82n9pnfepb+2rOrOQwNSFgCQ@mail.gmail.com>
Message-ID: <dc8dcbc1-627b-4d8a-a5b9-566b59247ae8@googlegroups.com>


>
> Thoughts attached.
>

Yeah that one is great! :) And true.

No, the distutils2 project continues and is still being actively 
> developed. (distutils2 *is* packaging - the name on PyPI is just 
> different from the name that will be used when it is added to the 
> standard library). Bento is a project that aims to tackle part of 
> distutils that distutils2 hasn't really looked at yet: creating a more 
> flexible build system (or, more accurately, making it easier to hook 
> into *existing* build systems rather than defining yet another one).


Ok. Now the distinction is clear. Good to hear disutils2 is still actively 
developed.

Those PEPs are *already* standards and are actively supported by 
> external projects today. 


I just found the wheel project:
https://bitbucket.org/dholth/wheel

It tries to implement those standards by patching distribute and pip. I run 
the
demo and I think the resulting package format is just amazing. In a 
nutshell is
a binary format (extensions get compiled) which in my opinion is really 
flexible.
I hope it will become standard and will be integrated in the major software 
someday.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120722/b36ba8e6/attachment.html>

From pyideas at rebertia.com  Sun Jul 22 22:33:53 2012
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 22 Jul 2012 13:33:53 -0700
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
	<CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
Message-ID: <CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com>

On Mon, Jul 16, 2012 at 12:12 AM, David Townshend <aquavitae69 at gmail.com> wrote:
> On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote:
<snip>
>> Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which
>> would have similarly added nice ways to get the current module, class,
>> and regular function, was rejected, I wouldn't hold out much hope for
>> such a feature being added any time soon.
>> (Wrapping via decorators introduces nontrivial subtlety regarding
>> whether you really want to refer to the decorated or undecorated
>> version of an entity.)
<snip>
> The problem with PEP 3130 was the PEP itself, not the idea.

Could you elaborate on how the PEP and the "idea" materially differed?

> I believe Eric
> Snow was working on a patch to implement this (see
> http://code.activestate.com/lists/python-ideas/11234/ and
> http://bugs.python.org/issue12857), but I'm not sure what its current status
> is.

I would hardly describe mucking around with frame objects as "nice"
(particularly compared against a dedicated keyword), so I would not
say that it implements the same idea; though, to be sure, what the
patch accomplishes is related, still useful, and could be used to
power the implementation of something like the PEP.

Belated regards,
Chris
--
Please avoid top-posting in the future :)


From victor.stinner at gmail.com  Sun Jul 22 23:25:59 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 22 Jul 2012 23:25:59 +0200
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
Message-ID: <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>

> "man pread" on OS/X suggests it exists there too

"man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
have been added to Python 3.3.

Victor


From guido at python.org  Mon Jul 23 00:36:44 2012
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Jul 2012 15:36:44 -0700
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
Message-ID: <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>

On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
>> "man pread" on OS/X suggests it exists there too
>
> "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
> have been added to Python 3.3.

Awesome. :-) But does the io module offer an API that uses it? It's
kind of awkward to have to call os.pread() with stream.fileno() as an
argument.

-- 
--Guido van Rossum (python.org/~guido)


From cs at zip.com.au  Mon Jul 23 01:04:30 2012
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 23 Jul 2012 09:04:30 +1000
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
References: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
Message-ID: <20120722230430.GA15682@cskk.homeip.net>

On 21Jul2012 13:35, Guido van Rossum <guido at python.org> wrote:
| On Sat, Jul 21, 2012 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote:
| > On 7/21/2012 2:59 PM, Matt Chaput wrote:
| >> I wish Python binary file objects had an atomic seek-read method, so
| >> I wouldn't have to perform my own locking everywhere to prevent other
| >> threads from moving the file pointer between seek and read.
[...]
| >> Is this something that can be bubbled up from the underlying
| >> platform? I think the Linux C equivalent is pread.
[...]
| "man pread" on OS/X suggests it exists there too. I presume the use
| case is to have a large data file open for reading by multiple
| threads. This is a reasonable use case and it makes some sense to
| extend our binary readable streams (buffered and unbuffered) with an
| API for this purpose.

On most Linux boxen you can say:

  man 3p pread

which will show you the POSIX man page, if it exists.

And it does!

So pread will exist on pretty much every UNIX platform, and I'd be
amazed if it wasn't on Windows.

In fact, it remarks that pread appeared in SysVr4, which is quite old.

| However, it's probably just efficient to just have a separate open
| stream per thread

It doubles the system call count per read (if pread is a system call,
which it ually will be (it is on Linux and MacOSX, and is hard to implement
otherwise without an annoying and slow locking scheme concealed inside
the C library).

I'd be +1 for adding pread and pwrite to the os module. It seems
reasonable and quite useful and should work on most platforms.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au>

Rimmer: It will be happened; it shall be going to be happening; it will be
    was an event that could will have been taken place in the future.
        - Red Dwarf, _Future Echoes_


From solipsis at pitrou.net  Mon Jul 23 01:47:10 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Jul 2012 01:47:10 +0200
Subject: [Python-ideas] Atomic file.get(offset, length)
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
Message-ID: <20120723014710.61fd5e12@pitrou.net>

On Sun, 22 Jul 2012 15:36:44 -0700
Guido van Rossum <guido at python.org> wrote:
> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
> >> "man pread" on OS/X suggests it exists there too
> >
> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
> > have been added to Python 3.3.
> 
> Awesome. :-) But does the io module offer an API that uses it? It's
> kind of awkward to have to call os.pread() with stream.fileno() as an
> argument.

It doesn't. I guess we could add an "offset" keyword-only argument to
read() and write(), but then we need to provide a Windows
implementation as well (it seems using overlapped I/O with
ReadFile() / WriteFile() could make it possible). Also, I'm not sure it
makes sense for buffered I/O, or only unbuffered.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From guido at python.org  Mon Jul 23 02:16:57 2012
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Jul 2012 17:16:57 -0700
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <20120723014710.61fd5e12@pitrou.net>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
	<20120723014710.61fd5e12@pitrou.net>
Message-ID: <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com>

On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 22 Jul 2012 15:36:44 -0700
> Guido van Rossum <guido at python.org> wrote:
>> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
>> <victor.stinner at gmail.com> wrote:
>> >> "man pread" on OS/X suggests it exists there too
>> >
>> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
>> > have been added to Python 3.3.
>>
>> Awesome. :-) But does the io module offer an API that uses it? It's
>> kind of awkward to have to call os.pread() with stream.fileno() as an
>> argument.
>
> It doesn't. I guess we could add an "offset" keyword-only argument to
> read() and write(), but then we need to provide a Windows
> implementation as well (it seems using overlapped I/O with
> ReadFile() / WriteFile() could make it possible). Also, I'm not sure it
> makes sense for buffered I/O, or only unbuffered.

Given that the use case is to avoid race conditions when more than one
thread is doing random-access reads on the same open file, I think it
makes some sense to implement it for both buffered and unbuffered
streams -- and even for text streams, since those support seek() as
well, so the race condition exists for those too.

But note that the pread() man page (at least the one I checked :-)
specifies that pread() doesn't affect the file pointer. So I suppose
it should also not affect the buffer. That may make it hard to
implement it for text streams (which IIRC rely quite heavy on
buffering for their implementation), but it should be easy for
buffered streams: it should just be passed on to the underlying
unbuffered stream.

(For those jumping in the middle of the thread: I know it's past the
feature freeze, so these considerations are for 3.4. Also, os.pread()
is in 3.3.)

-- 
--Guido van Rossum (python.org/~guido)


From g.rodola at gmail.com  Mon Jul 23 06:03:24 2012
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Mon, 23 Jul 2012 06:03:24 +0200
Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins
Message-ID: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>

This would be similar to 'key' argument already available for min(), max()
and sorted() and would let user decide what must be considered True and
what not.
Some use cases:


all(a, b, c, ... key=callable)
all(a, b, c, ... key=lambda x: isinstance(x, int))
any(a, b, c, ... key=lambda x: x < 10)

a = b = c = None
for x in foo:
    if cond1:
         a = 0
    elif cond2:
         b = 0
    elif cond3:
         c = 0
# meaning all cond* has been satisfied and it's ok to consider 0 a
meaningful value
assert all(a, b, c, key=lambda x: x != None)


Thoughts?

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/530d1aaf/attachment.html>

From raymond.hettinger at gmail.com  Mon Jul 23 06:36:28 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 22 Jul 2012 23:36:28 -0500
Subject: [Python-ideas] Provide a 'key' argument for all() and any()
	builtins
In-Reply-To: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
Message-ID: <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com>


On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote:

> This would be similar to 'key' argument already available for min(), max() and sorted() and would let user decide what must be considered True and what not.

There's no need.   We already have:

    >>> all(predicate(x) for x in iterable)
    >>> any(predicate(x) for x in iterable)


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120722/9c380f04/attachment.html>

From pyideas at rebertia.com  Mon Jul 23 06:39:13 2012
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 22 Jul 2012 21:39:13 -0700
Subject: [Python-ideas] Provide a 'key' argument for all() and any()
	builtins
In-Reply-To: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
Message-ID: <CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com>

On Sun, Jul 22, 2012 at 9:03 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
> This would be similar to 'key' argument already available for min(), max()
> and sorted() and would let user decide what must be considered True and what
> not.
> Some use cases:
>
>
> all(a, b, c, ... key=callable)

You're missing some brackets or parens there:
    all([a, b, c, ...], key=callable)

<snip>
> Thoughts?

I see negligible advantage over just writing the generator expression directly:
    all(callable(item) for item in iterable)
=== all(iterable, key=callable)
(which is about comparable to the difference between a map()
invocation vs. a list comprehension, and the choice there seems to be
largely a matter of taste)

min() & max() return the result object satisfying the constraint, so
the `key` argument makes sense for when you want to e.g. find the
cheapest house rather than the price of the cheapest house.
In contrast, any() & all() *always* just return a simple bool result,
not the object whose truthiness determined the predicate's result, so
`key` would be pointless since the result gets converted to a plain
bool anyway (or at least, we can conceptualize the implementation as
if it worked that way).

Cheers,
Chris


From aquavitae69 at gmail.com  Mon Jul 23 09:07:01 2012
From: aquavitae69 at gmail.com (David Townshend)
Date: Mon, 23 Jul 2012 09:07:01 +0200
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
	<CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
	<CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com>
Message-ID: <CAEgL-fdmxyerPVydU=v9Yvi9Nzm1iYjAF7Xgy+i8MBz580CCOw@mail.gmail.com>

>
> Could you elaborate on how the PEP and the "idea" materially differed?
>

IIRC, this was explained quite well in the thread I linked to.  Basically
the idea of accessing functions objects from within the function was fine,
but the PEP didn't adequately answer all the questions it should have and
tried to cover three ideas in one.  To quote the rejection notice from the
PEP:

    This PEP is rejected.  It is not clear how it should be
    implemented or what the precise semantics should be in edge cases,
    and there aren't enough important use cases given.  response has
    been lukewarm at best.


>  I would hardly describe mucking around with frame objects as "nice"
> (particularly compared against a dedicated keyword), so I would not
> say that it implements the same idea; though, to be sure, what the
> patch accomplishes is related, still useful, and could be used to
> power the implementation of something like the PEP.
>

Agreed, but it is a good start.


> Please avoid top-posting in the future :)
>

Sorry about that - tends to happen when I reply on my phone if I'm not
concentrating.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/0e1c2520/attachment.html>

From solipsis at pitrou.net  Mon Jul 23 11:52:16 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Jul 2012 11:52:16 +0200
Subject: [Python-ideas] Atomic file.get(offset, length)
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
	<20120723014710.61fd5e12@pitrou.net>
	<CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com>
Message-ID: <20120723115216.33cc8ed5@pitrou.net>

On Sun, 22 Jul 2012 17:16:57 -0700
Guido van Rossum <guido at python.org> wrote:

> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sun, 22 Jul 2012 15:36:44 -0700
> > Guido van Rossum <guido at python.org> wrote:
> >> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
> >> <victor.stinner at gmail.com> wrote:
> >> >> "man pread" on OS/X suggests it exists there too
> >> >
> >> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
> >> > have been added to Python 3.3.
> >>
> >> Awesome. :-) But does the io module offer an API that uses it? It's
> >> kind of awkward to have to call os.pread() with stream.fileno() as an
> >> argument.
> >
> > It doesn't. I guess we could add an "offset" keyword-only argument to
> > read() and write(), but then we need to provide a Windows
> > implementation as well (it seems using overlapped I/O with
> > ReadFile() / WriteFile() could make it possible). Also, I'm not sure it
> > makes sense for buffered I/O, or only unbuffered.
> 
> Given that the use case is to avoid race conditions when more than one
> thread is doing random-access reads on the same open file, I think it
> makes some sense to implement it for both buffered and unbuffered
> streams -- and even for text streams, since those support seek() as
> well, so the race condition exists for those too.
> 
> But note that the pread() man page (at least the one I checked :-)
> specifies that pread() doesn't affect the file pointer. So I suppose
> it should also not affect the buffer. That may make it hard to
> implement it for text streams (which IIRC rely quite heavy on
> buffering for their implementation), but it should be easy for
> buffered streams: it should just be passed on to the underlying
> unbuffered stream.

Indeed, it should not affect the buffer. That's why I'm questioning the
addition of this feature to buffered streams (whose whole point is their
implicit buffer management). Also, there are implementation subtleties
when e.g. reading from an area which overlaps the current buffer :-)

As you pointed out, I think a reasonable solution to the race condition
problem is to use several file descriptors. It may not work so well if
you also write to the file, though.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From g.rodola at gmail.com  Mon Jul 23 13:28:08 2012
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Mon, 23 Jul 2012 13:28:08 +0200
Subject: [Python-ideas] Provide a 'key' argument for all() and any()
	builtins
In-Reply-To: <CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com>
References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
	<CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com>
Message-ID: <CAFYqXL93gpRK0B0jxXqbVpEio4Kq--=DiUiP3hE=sTHmrt35iQ@mail.gmail.com>

2012/7/23 Chris Rebert <pyideas at rebertia.com>
>
> On Sun, Jul 22, 2012 at 9:03 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
> > This would be similar to 'key' argument already available for min(), max()
> > and sorted() and would let user decide what must be considered True and what
> > not.
> > Some use cases:
> >
> >
> > all(a, b, c, ... key=callable)
>
> You're missing some brackets or parens there:
>     all([a, b, c, ...], key=callable)
>
> <snip>
> > Thoughts?
>
> I see negligible advantage over just writing the generator expression directly:
>     all(callable(item) for item in iterable)

<snip>

> min() & max() return the result object satisfying the constraint, so
> the `key` argument makes sense for when you want to e.g. find the
> cheapest house rather than the price of the cheapest house.
> In contrast, any() & all() *always* just return a simple bool result,
> not the object whose truthiness determined the predicate's result, so
> `key` would be pointless since the result gets converted to a plain
> bool anyway (or at least, we can conceptualize the implementation as
> if it worked that way).


Agreed.
Please ignore my proposal then.

--- Giampaolo
 https://code.google.com/p/pyftpdlib/
https://code.google.com/p/psutil/
https://code.google.com/p/pysendfile/


From guido at python.org  Mon Jul 23 16:55:32 2012
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Jul 2012 07:55:32 -0700
Subject: [Python-ideas] Provide a 'key' argument for all() and any()
	builtins
In-Reply-To: <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com>
References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
	<B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com>
Message-ID: <CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com>

On Sun, Jul 22, 2012 at 9:36 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote:
>
> This would be similar to 'key' argument already available for min(), max()
> and sorted() and would let user decide what must be considered True and what
> not.
>
> There's no need.   We already have:
>
>     >>> all(predicate(x) for x in iterable)
>     >>> any(predicate(x) for x in iterable)

In addition, there's a reason why the key= parameter to sort, sorted,
max and min can't be replaced with a similar idiom: all these return
the *original* object(s) and the key= parameter affects only the
comparison; for example, try max(1,-2,3,-4, key=abs).

Whereas for the current proposal, the value returned by the key would
also determine the outcome -- and then there is no advantage over
using the comprehension. More general, if you can first apply the
transformation (the key function) and then the aggregation function
(min/max/etc.), there's no need to combine the transformation and the
aggregation. But in all the cases that have a key= parameter, this is
not so.

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Mon Jul 23 17:00:37 2012
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Jul 2012 08:00:37 -0700
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <20120723115216.33cc8ed5@pitrou.net>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
	<20120723014710.61fd5e12@pitrou.net>
	<CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com>
	<20120723115216.33cc8ed5@pitrou.net>
Message-ID: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com>

On Mon, Jul 23, 2012 at 2:52 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 22 Jul 2012 17:16:57 -0700
> Guido van Rossum <guido at python.org> wrote:
>
>> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Sun, 22 Jul 2012 15:36:44 -0700
>> > Guido van Rossum <guido at python.org> wrote:
>> >> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
>> >> <victor.stinner at gmail.com> wrote:
>> >> >> "man pread" on OS/X suggests it exists there too
>> >> >
>> >> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
>> >> > have been added to Python 3.3.
>> >>
>> >> Awesome. :-) But does the io module offer an API that uses it? It's
>> >> kind of awkward to have to call os.pread() with stream.fileno() as an
>> >> argument.
>> >
>> > It doesn't. I guess we could add an "offset" keyword-only argument to
>> > read() and write(), but then we need to provide a Windows
>> > implementation as well (it seems using overlapped I/O with
>> > ReadFile() / WriteFile() could make it possible). Also, I'm not sure it
>> > makes sense for buffered I/O, or only unbuffered.
>>
>> Given that the use case is to avoid race conditions when more than one
>> thread is doing random-access reads on the same open file, I think it
>> makes some sense to implement it for both buffered and unbuffered
>> streams -- and even for text streams, since those support seek() as
>> well, so the race condition exists for those too.
>>
>> But note that the pread() man page (at least the one I checked :-)
>> specifies that pread() doesn't affect the file pointer. So I suppose
>> it should also not affect the buffer. That may make it hard to
>> implement it for text streams (which IIRC rely quite heavy on
>> buffering for their implementation), but it should be easy for
>> buffered streams: it should just be passed on to the underlying
>> unbuffered stream.
>
> Indeed, it should not affect the buffer. That's why I'm questioning the
> addition of this feature to buffered streams (whose whole point is their
> implicit buffer management). Also, there are implementation subtleties
> when e.g. reading from an area which overlaps the current buffer :-)
>
> As you pointed out, I think a reasonable solution to the race condition
> problem is to use several file descriptors. It may not work so well if
> you also write to the file, though.

If you write and read a file from multiple threads you're crazy.

-- 
--Guido van Rossum (python.org/~guido)


From breamoreboy at yahoo.co.uk  Mon Jul 23 17:12:05 2012
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Mon, 23 Jul 2012 16:12:05 +0100
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
	<20120723014710.61fd5e12@pitrou.net>
	<CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com>
	<20120723115216.33cc8ed5@pitrou.net>
	<CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com>
Message-ID: <jujpgu$2pr$1@dough.gmane.org>

On 23/07/2012 16:00, Guido van Rossum wrote:
> On Mon, Jul 23, 2012 at 2:52 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Sun, 22 Jul 2012 17:16:57 -0700
>> Guido van Rossum <guido at python.org> wrote:
>>
>>> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>> On Sun, 22 Jul 2012 15:36:44 -0700
>>>> Guido van Rossum <guido at python.org> wrote:
>>>>> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner
>>>>> <victor.stinner at gmail.com> wrote:
>>>>>>> "man pread" on OS/X suggests it exists there too
>>>>>>
>>>>>> "man pread" or "import os; help(os.pread" ;-) pread() and pwrite()
>>>>>> have been added to Python 3.3.
>>>>>
>>>>> Awesome. :-) But does the io module offer an API that uses it? It's
>>>>> kind of awkward to have to call os.pread() with stream.fileno() as an
>>>>> argument.
>>>>
>>>> It doesn't. I guess we could add an "offset" keyword-only argument to
>>>> read() and write(), but then we need to provide a Windows
>>>> implementation as well (it seems using overlapped I/O with
>>>> ReadFile() / WriteFile() could make it possible). Also, I'm not sure it
>>>> makes sense for buffered I/O, or only unbuffered.
>>>
>>> Given that the use case is to avoid race conditions when more than one
>>> thread is doing random-access reads on the same open file, I think it
>>> makes some sense to implement it for both buffered and unbuffered
>>> streams -- and even for text streams, since those support seek() as
>>> well, so the race condition exists for those too.
>>>
>>> But note that the pread() man page (at least the one I checked :-)
>>> specifies that pread() doesn't affect the file pointer. So I suppose
>>> it should also not affect the buffer. That may make it hard to
>>> implement it for text streams (which IIRC rely quite heavy on
>>> buffering for their implementation), but it should be easy for
>>> buffered streams: it should just be passed on to the underlying
>>> unbuffered stream.
>>
>> Indeed, it should not affect the buffer. That's why I'm questioning the
>> addition of this feature to buffered streams (whose whole point is their
>> implicit buffer management). Also, there are implementation subtleties
>> when e.g. reading from an area which overlaps the current buffer :-)
>>
>> As you pointed out, I think a reasonable solution to the race condition
>> problem is to use several file descriptors. It may not work so well if
>> you also write to the file, though.
>
> If you write and read a file from multiple threads you're crazy.
>

Or Dutch? :)

-- 
Cheers.

Mark Lawrence.



From p.f.moore at gmail.com  Mon Jul 23 17:18:11 2012
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 23 Jul 2012 16:18:11 +0100
Subject: [Python-ideas] Provide a 'key' argument for all() and any()
	builtins
In-Reply-To: <CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com>
References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com>
	<B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com>
	<CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com>
Message-ID: <CACac1F-u=mkGQSeZFEVA0swdD4J4oq9dRjQcNKv7BCiP7rVddg@mail.gmail.com>

On 23 July 2012 15:55, Guido van Rossum <guido at python.org> wrote:
> On Sun, Jul 22, 2012 at 9:36 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>> On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote:
>>
>> This would be similar to 'key' argument already available for min(), max()
>> and sorted() and would let user decide what must be considered True and what
>> not.
>>
>> There's no need.   We already have:
>>
>>     >>> all(predicate(x) for x in iterable)
>>     >>> any(predicate(x) for x in iterable)
>
> In addition, there's a reason why the key= parameter to sort, sorted,
> max and min can't be replaced with a similar idiom: all these return
> the *original* object(s) and the key= parameter affects only the
> comparison; for example, try max(1,-2,3,-4, key=abs).
>
> Whereas for the current proposal, the value returned by the key would
> also determine the outcome -- and then there is no advantage over
> using the comprehension. More general, if you can first apply the
> transformation (the key function) and then the aggregation function
> (min/max/etc.), there's no need to combine the transformation and the
> aggregation. But in all the cases that have a key= parameter, this is
> not so.

To put it yet another way, the key= parameter encapsulates the
decorate-process-undecorate pattern, and that pattern doesn't apply to
any/all.

Paul.


From ubershmekel at gmail.com  Mon Jul 23 18:50:29 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Mon, 23 Jul 2012 19:50:29 +0300
Subject: [Python-ideas] Atomic file.get(offset, length)
In-Reply-To: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com>
References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca>
	<juf09k$njc$1@dough.gmane.org>
	<CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com>
	<CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com>
	<CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com>
	<20120723014710.61fd5e12@pitrou.net>
	<CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com>
	<20120723115216.33cc8ed5@pitrou.net>
	<CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com>
Message-ID: <CANSw7KzUDSpasvUutZekxpaBdu+q9EBn7Vj9VG+sRFnWaEBzkQ@mail.gmail.com>

On Mon, Jul 23, 2012 at 6:00 PM, Guido van Rossum <guido at python.org> wrote:

> If you write and read a file from multiple threads you're crazy.
>

Perhaps, though I could imagine a fine-grained-locking DB doing this with
constant sized data structures. Though that might be a good time to pull
this one out:
http://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/4a12fd4b/attachment.html>

From ericsnowcurrently at gmail.com  Tue Jul 24 02:59:21 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 23 Jul 2012 18:59:21 -0600
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
	<CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
	<CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com>
Message-ID: <CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com>

On Mon, Jul 16, 2012 at 10:28 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Yeah, this is still valuable.  However, at this point it would be a 3.4
> addition (_if_ it gets committed).  Updating the patch in the tracker is on
> my todo for the next few months.

FWIW, the possibility of exposing something like __function__ in the
function locals is less likely to garner support than the initial
f_func approach.  Said Guido[1]:

<quote>
If it were expected that people would start writing recursive
calls using __function__ routinely, in situations where a name
reference works, I'd be very unhappy with the new feature. (And if
someone wants to make the argument that recursive calls using
__function__ are actually better in some way I am willing to
filibuster.)
</quote>

So there's a metric that f_func passes much more readily.  That said,
the f_func patch would also make "__function__" easier to implement if
that feature passed muster.

-eric

[1] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html


From ncoghlan at gmail.com  Tue Jul 24 05:57:34 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Jul 2012 13:57:34 +1000
Subject: [Python-ideas] How can I get own instance of generator in
	itself code
In-Reply-To: <CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com>
References: <33429676.215905.1342333139344.JavaMail.root@socbay.com>
	<CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com>
	<CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com>
	<CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com>
	<CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com>
Message-ID: <CADiSq7etDhjrsNuXssbtP7CgkOY05_qZ9KXcAMKMsKaJ2Z8m_g@mail.gmail.com>

On Tue, Jul 24, 2012 at 10:59 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Mon, Jul 16, 2012 at 10:28 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> Yeah, this is still valuable.  However, at this point it would be a 3.4
>> addition (_if_ it gets committed).  Updating the patch in the tracker is on
>> my todo for the next few months.
>
> FWIW, the possibility of exposing something like __function__ in the
> function locals is less likely to garner support than the initial
> f_func approach.  Said Guido[1]:
>
> <quote>
> If it were expected that people would start writing recursive
> calls using __function__ routinely, in situations where a name
> reference works, I'd be very unhappy with the new feature. (And if
> someone wants to make the argument that recursive calls using
> __function__ are actually better in some way I am willing to
> filibuster.)
> </quote>
>
> So there's a metric that f_func passes much more readily.  That said,
> the f_func patch would also make "__function__" easier to implement if
> that feature passed muster.

Python 3 already has an implicit lexical __class__ reference (courtesy
of PEP 3135) and it refers to the inner *undecorated* version of the
class. References by name, on the other hand, refer to the
postdecorated version bound in the outer scope (which may or may not
be the same object, depending on the specific decorators).

A PEP for 3.4 that proposes a mechanism that more cleanly incorporates
__class__ into the language data model (thus addressing
http://bugs.python.org/issue12370) would be most welcome.

Such a PEP could also attempt to make a symmetry argument for a
__def__ reference that refers directly to the undecorated function
object (as it's a lexical reference, it's worth emphasising the
keyword over the kind of object). An implicit reference to the
*decorated* object would not be acceptable, as decorators need to be
able to call the function, even if it relies on the implicit
reference. (This is the same reason why __class__ refers to the
undecorated version: so that the class definition is complete while
the decorator is running, which means they can instantiate it and call
methods that use the new shorthand style super() invocation)

The module self reference aspect discussed in PEP 3130 should just be
dropped as using "mod = importlib.import_module(__name__)" is enough
to avoid the odd behaviour associated with trying to use "__import__"
directly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From fetchinson at googlemail.com  Tue Jul 24 15:23:15 2012
From: fetchinson at googlemail.com (Daniel Fetchinson)
Date: Tue, 24 Jul 2012 15:23:15 +0200
Subject: [Python-ideas] Python Package Collections
Message-ID: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>

Hi folks, here is an idea that came out of (not quite) thin air:

Pypi, setuptools, distribute & Co are all very useful for finding,
installing, upgrading, etc. of python packages. They even handle
dependencies! Yaaaay!

However, there is one more thing that could be built on top of them
that would make them even more useful. This is the <drum roll please>
Python Package Collections! This would be an online service to
organize packages into collections allowing one to download, install,
build, upgrade, etc. many packages at once. Say somebody finds
package1, package2, package3, package4 very useful and figure they
naturally belong together by topic. And so our test person always
wants to have them all installed so when he downloads a new python
version which will make his python arsenal not only 2.4 2.5 2.6 2.7
3.1 3.2 3.3 but also 3.4, he won't have to install all these packages
separately again but he can just install the collection with one
command or one click.

You might ask why can't our test person just do "install package1"
then "install package2" then "install package3" then "install
package4"? Well, certainly he could do that but if he found that these
four packages nicely go together chances are others will find it handy
to have them all too. So our brilliant test person will want to share
his collection with others telling them essentially "if you like
working with X" or "you frequently develop for Y" then "you can just
install my collection and you will be ready to go"!

Anybody could organize packages into collections and I bet there would
be very popular ones like "mathematics" including all sorts of math
packages from pypi, or "pdfgoodies" including everything pdf related
that one typically needs for work with pdf. These don't have to depend
on each other (dependencies are handled quite well by existing tools),
they just need to have the property that they are very handy to have
together.

To be honest the idea is not mine, take a look at
https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for
firefox are organized into collections. It's basically the same idea
but for python packages.

This whole thing could be integrated into pypi or initially the idea
could be tested on a separate site to see if it catches on.

What do you all think?

Cheers,
Daniel


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown


From ubershmekel at gmail.com  Tue Jul 24 15:26:19 2012
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Tue, 24 Jul 2012 16:26:19 +0300
Subject: [Python-ideas] Python Package Collections
In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
Message-ID: <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com>

On Tue, Jul 24, 2012 at 4:23 PM, Daniel Fetchinson <
fetchinson at googlemail.com> wrote:

> Hi folks, here is an idea that came out of (not quite) thin air:
>
> Pypi, setuptools, distribute & Co are all very useful for finding,
> installing, upgrading, etc. of python packages. They even handle
> dependencies! Yaaaay!
>
> However, there is one more thing that could be built on top of them
> that would make them even more useful. This is the <drum roll please>
> Python Package Collections! This would be an online service to
> organize packages into collections allowing one to download, install,
> build, upgrade, etc. many packages at once. Say somebody finds
> package1, package2, package3, package4 very useful and figure they
> naturally belong together by topic. And so our test person always
> wants to have them all installed so when he downloads a new python
> version which will make his python arsenal not only 2.4 2.5 2.6 2.7
> 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages
> separately again but he can just install the collection with one
> command or one click.
>
> You might ask why can't our test person just do "install package1"
> then "install package2" then "install package3" then "install
> package4"? Well, certainly he could do that but if he found that these
> four packages nicely go together chances are others will find it handy
> to have them all too. So our brilliant test person will want to share
> his collection with others telling them essentially "if you like
> working with X" or "you frequently develop for Y" then "you can just
> install my collection and you will be ready to go"!
>
> Anybody could organize packages into collections and I bet there would
> be very popular ones like "mathematics" including all sorts of math
> packages from pypi, or "pdfgoodies" including everything pdf related
> that one typically needs for work with pdf. These don't have to depend
> on each other (dependencies are handled quite well by existing tools),
> they just need to have the property that they are very handy to have
> together.
>
> To be honest the idea is not mine, take a look at
> https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for
> firefox are organized into collections. It's basically the same idea
> but for python packages.
>
> This whole thing could be integrated into pypi or initially the idea
> could be tested on a separate site to see if it catches on.
>
> What do you all think?
>
> Cheers,
> Daniel
>
>
> --
> Psss, psss, put it down! - http://www.cafepress.com/putitdown
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


Sounds like a great idea for a separate site.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/655b338c/attachment.html>

From ncoghlan at gmail.com  Tue Jul 24 15:33:57 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Jul 2012 23:33:57 +1000
Subject: [Python-ideas] Python Package Collections
In-Reply-To: <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com>
References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
	<CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com>
Message-ID: <CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com>

On Tue, Jul 24, 2012 at 11:26 PM, Yuval Greenfield
<ubershmekel at gmail.com> wrote:
> On Tue, Jul 24, 2012 at 4:23 PM, Daniel Fetchinson
> <fetchinson at googlemail.com> wrote:
>> This whole thing could be integrated into pypi or initially the idea
>> could be tested on a separate site to see if it catches on.
>
> Sounds like a great idea for a separate site.

Certainly sounds like a concept worth exploring. Similar to sites like
crate.io and pythonpackages, your best bet would be to start with a
separate site that uses the PyPI APIs to access the relevant
information.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From michelelacchia at gmail.com  Tue Jul 24 16:06:24 2012
From: michelelacchia at gmail.com (Michele Lacchia)
Date: Tue, 24 Jul 2012 07:06:24 -0700 (PDT)
Subject: [Python-ideas] Python Package Collections
In-Reply-To: <CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com>
References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
	<CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com>
	<CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com>
Message-ID: <78662cc3-2b86-404a-9a9b-28b35a165711@googlegroups.com>

Crate.io supports so-called lists of packages. But I don't know if you can 
share them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/0a92de02/attachment.html>

From bkabrda at redhat.com  Tue Jul 24 15:55:40 2012
From: bkabrda at redhat.com (Bohuslav Kabrda)
Date: Tue, 24 Jul 2012 09:55:40 -0400 (EDT)
Subject: [Python-ideas] Python Package Collections
In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
Message-ID: <924466335.1936859.1343138140069.JavaMail.root@redhat.com>

----- Original Message -----
> Hi folks, here is an idea that came out of (not quite) thin air:
> 
> Pypi, setuptools, distribute & Co are all very useful for finding,
> installing, upgrading, etc. of python packages. They even handle
> dependencies! Yaaaay!
> 
> However, there is one more thing that could be built on top of them
> that would make them even more useful. This is the <drum roll please>
> Python Package Collections! This would be an online service to
> organize packages into collections allowing one to download, install,
> build, upgrade, etc. many packages at once. Say somebody finds
> package1, package2, package3, package4 very useful and figure they
> naturally belong together by topic. And so our test person always
> wants to have them all installed so when he downloads a new python
> version which will make his python arsenal not only 2.4 2.5 2.6 2.7
> 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages
> separately again but he can just install the collection with one
> command or one click.
> 
> You might ask why can't our test person just do "install package1"
> then "install package2" then "install package3" then "install
> package4"? Well, certainly he could do that but if he found that
> these
> four packages nicely go together chances are others will find it
> handy
> to have them all too. So our brilliant test person will want to share
> his collection with others telling them essentially "if you like
> working with X" or "you frequently develop for Y" then "you can just
> install my collection and you will be ready to go"!
> 
> Anybody could organize packages into collections and I bet there
> would
> be very popular ones like "mathematics" including all sorts of math
> packages from pypi, or "pdfgoodies" including everything pdf related
> that one typically needs for work with pdf. These don't have to
> depend
> on each other (dependencies are handled quite well by existing
> tools),
> they just need to have the property that they are very handy to have
> together.
> 
> To be honest the idea is not mine, take a look at
> https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for
> firefox are organized into collections. It's basically the same idea
> but for python packages.
> 
> This whole thing could be integrated into pypi or initially the idea
> could be tested on a separate site to see if it catches on.
> 
> What do you all think?
> 
> Cheers,
> Daniel
> 

I think you can emulate this behaviour by creating a pypi "metapackage" - an empty package that only "install_requires" the others.
Having said that, I agree that it might be nice to support this as a new feature (by more light-weight solution that the mentioned "metapackages").

-- 
Regards,
Bohuslav "Slavek" Kabrda.


From ianb at colorstudy.com  Tue Jul 24 20:16:41 2012
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 24 Jul 2012 13:16:41 -0500
Subject: [Python-ideas] Python Package Collections
In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com>
Message-ID: <CAHtfchqAUFeZi59CbGr8a=L2aY4K81QN3b8+KVESE8=TW+3Cxw@mail.gmail.com>

You can also publish pip requirement files at a URL, and use:

  pip install -r http://pypackagecollections.org/scraping-tools.txt

It works with upgrade/uninstall/etc.  But typically if you want to get
someone started on a new task using a new set of tools, you want to also
generate or at least unpack starter files for them ? installing libraries
isn't helpful if you don't know what they do or how to use them, or
distinguish between libraries-you-use and libraries-your-libraries-use.
 Which is to say, I'm not surprised that a public index of requirement
files hasn't emerged, because it makes more sense for specific starter
guides to be written, and those guides may use other tools.



On Tue, Jul 24, 2012 at 8:23 AM, Daniel Fetchinson <
fetchinson at googlemail.com> wrote:

> Hi folks, here is an idea that came out of (not quite) thin air:
>
> Pypi, setuptools, distribute & Co are all very useful for finding,
> installing, upgrading, etc. of python packages. They even handle
> dependencies! Yaaaay!
>
> However, there is one more thing that could be built on top of them
> that would make them even more useful. This is the <drum roll please>
> Python Package Collections! This would be an online service to
> organize packages into collections allowing one to download, install,
> build, upgrade, etc. many packages at once. Say somebody finds
> package1, package2, package3, package4 very useful and figure they
> naturally belong together by topic. And so our test person always
> wants to have them all installed so when he downloads a new python
> version which will make his python arsenal not only 2.4 2.5 2.6 2.7
> 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages
> separately again but he can just install the collection with one
> command or one click.
>
> You might ask why can't our test person just do "install package1"
> then "install package2" then "install package3" then "install
> package4"? Well, certainly he could do that but if he found that these
> four packages nicely go together chances are others will find it handy
> to have them all too. So our brilliant test person will want to share
> his collection with others telling them essentially "if you like
> working with X" or "you frequently develop for Y" then "you can just
> install my collection and you will be ready to go"!
>
> Anybody could organize packages into collections and I bet there would
> be very popular ones like "mathematics" including all sorts of math
> packages from pypi, or "pdfgoodies" including everything pdf related
> that one typically needs for work with pdf. These don't have to depend
> on each other (dependencies are handled quite well by existing tools),
> they just need to have the property that they are very handy to have
> together.
>
> To be honest the idea is not mine, take a look at
> https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for
> firefox are organized into collections. It's basically the same idea
> but for python packages.
>
> This whole thing could be integrated into pypi or initially the idea
> could be tested on a separate site to see if it catches on.
>
> What do you all think?
>
> Cheers,
> Daniel
>
>
> --
> Psss, psss, put it down! - http://www.cafepress.com/putitdown
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/d09e2ea1/attachment.html>

From miki.tebeka at gmail.com  Wed Jul 25 22:51:25 2012
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Wed, 25 Jul 2012 13:51:25 -0700 (PDT)
Subject: [Python-ideas] compatibility triples for binary distributions
In-Reply-To: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com>
References: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com>
Message-ID: <92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com>

IMO  implementation names should be compatible with PEP-421<http://www.python.org/dev/peps/pep-0421/>. 


Also instead of a cryptic long string, maybe some structured one?. For 
example (JSON): {"implementation":"cpython","machine":"x86_64"}
It's longer but more readable IMO (and easier to parse).


On Wednesday, July 25, 2012 7:55:34 AM UTC-7, Daniel Holth wrote:
>
> I am working on a specification that might be useful in the PKG-INFO
> Platform or Supported-Platform tags. It tries to express which Python
> versions a built distribution works on with a triple
> (Implementation+Language Version, ABI, Architecture), anticipating an
> ABI that is not tied to a single CPython release.
>
> A typical tag would be py27-noabi-noarch or cp33-cp33dmu-linux_x86_64.
>
> cp27-noabi-noarch is a package that requires CPython 2.7 but doesn't
> include extension modules.
>
> The Python implementation is abbreviated. Each implementation has a
> two-letter code:
>
>     py: Generic Python
>     cp: CPython
>     ip: IronPython
>     pp: PyPy
>     jy: Jython
>
> concatenated with py_version_nodot ?27?. The minor version can be
> omitted ?py2? or ?py3? when appropriate.
>
> The ABI tag is an abbreviated SOABI ?cp33m?, or, for ?pure Python?
> packages, ?noabi?. For pypy it was more or less suggested that the ABI
> would have to be a hash of the source code revision and the
> translation/compilation options.
>
> The platform tag is distutils.util.get_platform() with all periods and
> hyphens replaced with underscore, or the string ?noarch?.
>
> I use the term "built distribution" instead of "binary distribution"
> because it includes pure python packages that for example have had
> 2to3 run ahead of time and are installable without setup.py.
>
> Will it work? The ABI in particular may be so messy that it should be 
> lumped with architecture. What is the ABI for packages that use ctypes or 
> cffi?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120725/bdb4be4e/attachment.html>

From steve at pearwood.info  Thu Jul 26 03:57:25 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 26 Jul 2012 11:57:25 +1000
Subject: [Python-ideas] compatibility triples for binary distributions
In-Reply-To: <92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com>
References: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com>
	<92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com>
Message-ID: <5010A405.10704@pearwood.info>

Sorry for breaking threading, I actually wanted to reply to the post from 
Daniel Holth, but for some reason I don't have the original.

Miki Tebeka wrote:
> IMO  implementation names should be compatible with PEP-421<http://www.python.org/dev/peps/pep-0421/>. 
> 
> 
> Also instead of a cryptic long string, maybe some structured one?. For 
> example (JSON): {"implementation":"cpython","machine":"x86_64"}
> It's longer but more readable IMO (and easier to parse).

+1

See below.

> On Wednesday, July 25, 2012 7:55:34 AM UTC-7, Daniel Holth wrote:
[...]
>> The Python implementation is abbreviated. Each implementation has a
>> two-letter code:
>>
>>     py: Generic Python
>>     cp: CPython
>>     ip: IronPython
>>     pp: PyPy
>>     jy: Jython


Who would be responsible for registering and tracking these two-letter codes? 
What counts as "generic python" -- surely if anything is "generic", it is CPython?

I can see that you have missed out on a number of existing or experimental 
implementations, including:

Stackless
WPython
ActivePython
CLPython
PyMite
Python-iPod
TinyPy
Unladen Swallow
CapPython
HotPy
Skulpt
HoPe
Berp

among others. Some of these may be dead projects; others may be working but 
not maintained; some are actively maintained. Even if they only have a small 
number of users, they are still legitimate Python implementations. If there is 
need to distinguish implementations at all, surely there is a need to 
distinguish (say) TinyPy from CapPython?

I think that this demonstrates the folly of using a central registry of codes 
for implementations. There are more implementations than just the "big four", 
and although some of them have the life-expectancy of mayflies, some of them 
go on for many years.



-- 
Steven



From matt at whoosh.ca  Fri Jul 27 00:40:33 2012
From: matt at whoosh.ca (Matt Chaput)
Date: Thu, 26 Jul 2012 18:40:33 -0400
Subject: [Python-ideas] Marvelous movable memoryview
Message-ID: <5011C761.3060606@whoosh.ca>

By which I mean a memoryview that lets you change the start and end 
offsets of its view of the underlying object (not modifying the 
underlying object).

Let's say I want to slice a long string s into trigrams and write them 
to disk (or something).

   for i in xrange(0, len(s) - 3):
       x = s[i:i + 3]
       myfile.write(x)

At each step the slice copies the bytes of the string, even though all I 
do is write them to disk.

I could avoid copying with memoryviews...

   for i in xrange(0, len(s) - 3):
       x = memoryview(s)[i:i + 3]
       myfile.write(x)

...but this is actually much slower (3x slower in some quick tests). I'm 
guessing it's because of all the object creation (while string slicing 
probably uses fast paths).

Shouldn't I be able to do this?

   m = memoryview(s)
   for i in xrange(0, len(s) - 3):
       m.start = i
       m.end = i + 3
       myfile.write(m)


Cheers,

Matt



From dholth at gmail.com  Fri Jul 27 00:52:31 2012
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 26 Jul 2012 18:52:31 -0400
Subject: [Python-ideas] Package collections
Message-ID: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com>

See http://good-py.appspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120726/0c7ac2db/attachment.html>

From solipsis at pitrou.net  Fri Jul 27 01:25:56 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 27 Jul 2012 01:25:56 +0200
Subject: [Python-ideas] Marvelous movable memoryview
References: <5011C761.3060606@whoosh.ca>
Message-ID: <20120727012556.77a3a998@pitrou.net>

On Thu, 26 Jul 2012 18:40:33 -0400
Matt Chaput <matt at whoosh.ca> wrote:
> By which I mean a memoryview that lets you change the start and end 
> offsets of its view of the underlying object (not modifying the 
> underlying object).
> 
> Let's say I want to slice a long string s into trigrams and write them 
> to disk (or something).
> 
>    for i in xrange(0, len(s) - 3):
>        x = s[i:i + 3]
>        myfile.write(x)
> 
> At each step the slice copies the bytes of the string, even though all I 
> do is write them to disk.

To be honest, copying three bytes is a trivial operation compared to
all the rest (interpreting bytecode, instantiating and destructing
bytes objects, calling I/O routines, etc.).

> Shouldn't I be able to do this?
> 
>    m = memoryview(s)
>    for i in xrange(0, len(s) - 3):
>        m.start = i
>        m.end = i + 3
>        myfile.write(m)

You wouldn't win anything over the simple bytes slicing approach, IMO.
(even with PyPy or another interpreter)

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From solipsis at pitrou.net  Fri Jul 27 01:26:45 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 27 Jul 2012 01:26:45 +0200
Subject: [Python-ideas] Package collections
References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com>
Message-ID: <20120727012645.3e465fd7@pitrou.net>

On Thu, 26 Jul 2012 18:52:31 -0400
Daniel Holth <dholth at gmail.com> wrote:
> See http://good-py.appspot.com

Perhaps you sent your message too early, what specifically are you
proposing?

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From ncoghlan at gmail.com  Fri Jul 27 03:37:10 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Jul 2012 11:37:10 +1000
Subject: [Python-ideas] Marvelous movable memoryview
In-Reply-To: <5011C761.3060606@whoosh.ca>
References: <5011C761.3060606@whoosh.ca>
Message-ID: <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com>

On Fri, Jul 27, 2012 at 8:40 AM, Matt Chaput <matt at whoosh.ca> wrote:
> I could avoid copying with memoryviews...
>
>   for i in xrange(0, len(s) - 3):
>       x = memoryview(s)[i:i + 3]
>       myfile.write(x)
>
> ...but this is actually much slower (3x slower in some quick tests). I'm
> guessing it's because of all the object creation (while string slicing
> probably uses fast paths).

memorview objects are pretty big - they have to store a lot of
pointers and other objects that describe their view of the underlying
buffer. Bytes objects, on the other hand, have minimal overhead.

(These numbers are for 3.3, which uses a revamped memoryview implementation)

>>> import sys
>>> x = b"foo"
>>> sys.getsizeof(x)
36
>>> sys.getsizeof(memoryview(x))
184

There's also the execution speed overhead that comes from the
indirection when accessing the contents.

Thus, using views instead of copying really only starts to pay off
once you're talking about comparatively large chunks of data:

>>> x *= 1000
>>> sys.getsizeof(x)
3033
>>> sys.getsizeof(memoryview(x))
184

> Shouldn't I be able to do this?
>
>   m = memoryview(s)
>   for i in xrange(0, len(s) - 3):
>       m.start = i
>       m.end = i + 3
>       myfile.write(m)

No, because making memorviews mutable would be a huge increase in
complexity (and they're complex enough already - only with Stefan
Krah's work in 3.3 have we finally worked most of the kinks out of the
implementation).

What you *can* do with a memoryview, though, is slice it, and the
resulting object will be a memoryview that references a subset of the
original object. This can be done with full slicing flexibility in
3.3, or in a more limited fashion in earlier versions.

For example, processing a data sequence in chunks in 3.3 without
copying and without inadvertently keeping a potentially large data
object alive (and/or locked into immutability) by hanging on to a
buffer reference:

  chunk_len = 512 # For small chunks, copying is likely faster. Measure it!
  with memorview(data) as m:
      for offset in range(0, len(m), chunk_len):
          with m[offset:offset+chunk_len] as x:
              process_chunk(x)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Fri Jul 27 03:50:03 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Jul 2012 11:50:03 +1000
Subject: [Python-ideas] Package collections
In-Reply-To: <20120727012645.3e465fd7@pitrou.net>
References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com>
	<20120727012645.3e465fd7@pitrou.net>
Message-ID: <CADiSq7cVSsQK+SihQH-FOmkdHcfOkZPtXj+gxGZYwWHGsFZMww@mail.gmail.com>

On Fri, Jul 27, 2012 at 9:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 26 Jul 2012 18:52:31 -0400
> Daniel Holth <dholth at gmail.com> wrote:
>> See http://good-py.appspot.com
>
> Perhaps you sent your message too early, what specifically are you
> proposing?

I don't think it's a specific proposal yet, I believe it is just a
"here's a follow up for those that were reading the previous thread".

A little more prose in the OP would have helped make that clear, though...

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From solipsis at pitrou.net  Fri Jul 27 07:41:08 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 27 Jul 2012 07:41:08 +0200
Subject: [Python-ideas] Package collections
References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com>
	<20120727012645.3e465fd7@pitrou.net>
	<CADiSq7cVSsQK+SihQH-FOmkdHcfOkZPtXj+gxGZYwWHGsFZMww@mail.gmail.com>
Message-ID: <20120727074108.3b68ce86@pitrou.net>

On Fri, 27 Jul 2012 11:50:03 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Fri, Jul 27, 2012 at 9:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Thu, 26 Jul 2012 18:52:31 -0400
> > Daniel Holth <dholth at gmail.com> wrote:
> >> See http://good-py.appspot.com
> >
> > Perhaps you sent your message too early, what specifically are you
> > proposing?
> 
> I don't think it's a specific proposal yet, I believe it is just a
> "here's a follow up for those that were reading the previous thread".

Ah, right, I hadn't seen Daniel's message was a followup to another
thread. It came unthreaded here.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From lists at cheimes.de  Fri Jul 27 14:26:19 2012
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 27 Jul 2012 14:26:19 +0200
Subject: [Python-ideas] Marvelous movable memoryview
In-Reply-To: <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com>
References: <5011C761.3060606@whoosh.ca>
	<CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com>
Message-ID: <juu1db$d82$1@dough.gmane.org>

Am 27.07.2012 03:37, schrieb Nick Coghlan:

> Thus, using views instead of copying really only starts to pay off
> once you're talking about comparatively large chunks of data:
>
>>>> x *= 1000
>>>> sys.getsizeof(x)
> 3033
>>>> sys.getsizeof(memoryview(x))
> 184

It can pay off A LOT. I've recently added PEP 3118 buffer support to my
smc.freeimage library (Cython wrapper of FreeImage and LCMS).

smc.freeimage needs 0.006 sec for the loop as it returns a writable
buffer of the internal pixel data (zero copy). PIL runs 1.165 sec as it
uses the old buffer interface and copies its data every time.

Benchmark code:
https://bitbucket.org/tiran/smc.freeimage/src/b97e13e9f04c/contrib/benchmark.py

RW_COUNT = 300
tiff = Image(TIFF)
start = time()
for i in xrange(RW_COUNT):
    arr = asarray(tiff)
    # change last BGR -> RGB
    arr = arr[..., ::-1]
    bytescale(arr, 64, 192)
end = time() - start

tiff = pil_open(TIFF)
tiff.load()
start = time()
for i in xrange(RW_COUNT):
    arr = asarray(tiff)
    bytescale(arr, 64, 192)
end = time() - start


> No, because making memorviews mutable would be a huge increase in
> complexity (and they're complex enough already - only with Stefan
> Krah's work in 3.3 have we finally worked most of the kinks out of the
> implementation).

+1 for Nick's -1

The new buffer interface and memoryview's are already very complex. I
haven't found a library yet that supports all edge cases. Even NumPy
doesn't support multidimensional and non-contiguous buffer with
suboffsets. I wanted to use a sub-offset of 3 on the last dimension with
a negative stride on the last dimension to convert BGR to RGB inside the
buffer code. It didn't work because NumPy ignore the information.

Christian

	



From sturla at molden.no  Sat Jul 28 23:09:13 2012
From: sturla at molden.no (Sturla Molden)
Date: Sat, 28 Jul 2012 23:09:13 +0200
Subject: [Python-ideas] Marvelous movable memoryview
In-Reply-To: <juu1db$d82$1@dough.gmane.org>
References: <5011C761.3060606@whoosh.ca>
	<CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com>
	<juu1db$d82$1@dough.gmane.org>
Message-ID: <501454F9.2040802@molden.no>

Den 27.07.2012 14:26, skrev Christian Heimes:
> The new buffer interface and memoryview's are already very complex. I 
> haven't found a library yet that supports all edge cases. Even NumPy 
> doesn't support multidimensional and non-contiguous buffer with 
> suboffsets. I wanted to use a sub-offset of 3 on the last dimension 
> with a negative stride on the last dimension to convert BGR to RGB 
> inside the buffer code. It didn't work because NumPy ignore the 
> information.

For those who write C extensions with Cython, it might be useful to know 
that it can compile "Python-like" code that uses PEP 3118 buffers to 
very efficient C code:

     http://docs.cython.org/src/userguide/memoryviews.html

On a simple test posted to the cython-user list,

     https://github.com/jakevdp/memview_benchmarks

I only got 2.2 % increased run-time compared to plain C pointer 
arithmetics. That involved creating two million PEP 3118 buffer objects 
and computing one million dot products with length 1000.

This takes most of the complexity of using PEP 3118 buffers in C 
extensions away, and allows us to write C extensions with a syntax that 
are very similar to Python with NumPy arrays.

(On the other hand, the old "NumPy array syntax" in Cython is 
considerably slower, particularly when slicing the arrays.)


Sturla


From x at jwp.io  Sun Jul 29 02:47:31 2012
From: x at jwp.io (James William Pye)
Date: Sat, 28 Jul 2012 17:47:31 -0700
Subject: [Python-ideas] Python Environment Initialization Scripts
Message-ID: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>

I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist.
Some quick searches on -ideas and -dev didn't reveal anything either.

Has the idea of [python] environment initialization been pitched/rejected before or does it already exist?

Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization.

Without modifying libpython, I can achieve what I'm trying to do by:

 * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()")
 * Overriding the python3 executable (not desirable; consider embedding)
 * Provide an alternate executable. (not desirable; again embedding and lack of availability in scripts pointing directly to python3) 
 * LD_PRELOAD overrides? (portability problems abound, I imagine. mangling seems preferable)
 * Perform installation in the dependent (duplication of installation in each dependent)

[1] https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/CCHe8_WHRt8

From ericsnowcurrently at gmail.com  Sun Jul 29 02:56:53 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 28 Jul 2012 18:56:53 -0600
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
Message-ID: <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>

On Sat, Jul 28, 2012 at 6:47 PM, James William Pye <x at jwp.io> wrote:
> I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist.
> Some quick searches on -ideas and -dev didn't reveal anything either.
>
> Has the idea of [python] environment initialization been pitched/rejected before or does it already exist?
>
> Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization.
>
> Without modifying libpython, I can achieve what I'm trying to do by:
>
>  * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()")

No need to modify site.py.  Either create a sitecustomize.py in your
site-packages.  Here are some references:

http://docs.python.org/library/site.html
http://www.python.org/dev/peps/pep-0370

Nothing new is needed.

As to metapath hooks, one alternative is to make a helper function
that puts the hooks in place (or otherwise manages your hooks).  Then
you can call that API in each script that needs those hooks.

-eric


From x at jwp.io  Sun Jul 29 03:54:43 2012
From: x at jwp.io (James William Pye)
Date: Sat, 28 Jul 2012 18:54:43 -0700
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
Message-ID: <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>


On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> No need to modify site.py.  Either create a sitecustomize.py in your
> site-packages.

I had recently forgotten about that. Thanks. =)

However, from the documentation:

 "After these path manipulations, an attempt is made to import a module named site customize,"
 "which can perform arbitrary site-specific customizations. It is typically created by a system
 "administrator in the site-packages directory."

The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py.

In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon
`pip install metapathawesomeness` which may already be in use by the system administrator.
That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way.

> As to metapath hooks, one alternative is to make a helper function
> that puts the hooks in place (or otherwise manages your hooks).  Then
> you can call that API in each script that needs those hooks.

Yeah, I referred to that in my list of alternatives.
My gripe with it is the duplication across the set of dependent scripts/packages.

However, it would seem to be the most appropriate solution ATM. =\

From ericsnowcurrently at gmail.com  Sun Jul 29 05:17:47 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 28 Jul 2012 21:17:47 -0600
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
	<95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
Message-ID: <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>

On Sat, Jul 28, 2012 at 7:54 PM, James William Pye <x at jwp.io> wrote:
>
> On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> No need to modify site.py.  Either create a sitecustomize.py in your
>> site-packages.
>
> I had recently forgotten about that. Thanks. =)
>
> However, from the documentation:
>
>  "After these path manipulations, an attempt is made to import a module named site customize,"
>  "which can perform arbitrary site-specific customizations. It is typically created by a system
>  "administrator in the site-packages directory."
>
> The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py.
>
> In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon
> `pip install metapathawesomeness` which may already be in use by the system administrator.
> That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way.

That's where PEP 370 comes in.  Stick your sitecustomize.py into
~/.local/lib/pythonX.Y/site-packages.  Then it only applies when
python is run as you.

-eric


From x at jwp.io  Sun Jul 29 07:22:19 2012
From: x at jwp.io (James William Pye)
Date: Sat, 28 Jul 2012 22:22:19 -0700
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
	<95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
	<CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>
Message-ID: <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io>

On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> That's where PEP 370 comes in.  Stick your sitecustomize.py into
> ~/.local/lib/pythonX.Y/site-packages.  Then it only applies when
> python is run as you.

But it's not desirable for it to be limited one user...

The situation is that the meta_path hook is provided in a package that is depended on
by other packages that contain modules that are importable by the hook.

It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the
system administrator's sitecustomize or the user's sitecustomize. No?

Now, that's not to say that I'm unwilling to be inappropriate. >8)

From ncoghlan at gmail.com  Sun Jul 29 08:57:01 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 29 Jul 2012 16:57:01 +1000
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
	<95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
	<CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>
	<C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io>
Message-ID: <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com>

On Sun, Jul 29, 2012 at 3:22 PM, James William Pye <x at jwp.io> wrote:
> On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> That's where PEP 370 comes in.  Stick your sitecustomize.py into
>> ~/.local/lib/pythonX.Y/site-packages.  Then it only applies when
>> python is run as you.
>
> But it's not desirable for it to be limited one user...
>
> The situation is that the meta_path hook is provided in a package that is depended on
> by other packages that contain modules that are importable by the hook.
>
> It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the
> system administrator's sitecustomize or the user's sitecustomize. No?
>
> Now, that's not to say that I'm unwilling to be inappropriate. >8)

The short answer is that, no, a coherent proposal for application
specific preinitialisation hasn't been put forward as a PEP. The
slightly longer answer is that there are related problems in terms of
initialising debuggers, warnings, logging, faulthandler, coverage,
profiling, etc, during development where it would be incredibly
convenient to be able to execute a short snippet of Python code prior
to execution.

See http://bugs.python.org/issue14803 (I'm nervous about the security
implications of a PYTHONRUNFIRST environment variable that is enabled
by default, but the idea of a -C option doesn't pose any such
concerns)

In general though, messing with process global state (including the
import system) implicitly is just *not cool*. It leads to applications
that have a dependency on a particular tool being present, but don't
actually make sure that it is available. Thus you end up with manual
instructions that say "before running this script, make sure
infrastructure X is in place to automagically take care of things I
should be doing myself".

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From x at jwp.io  Sun Jul 29 11:44:55 2012
From: x at jwp.io (James William Pye)
Date: Sun, 29 Jul 2012 02:44:55 -0700
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
	<95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
	<CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>
	<C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io>
	<CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com>
Message-ID: <C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io>

On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The short answer is that, no, a coherent proposal for application
> specific preinitialisation hasn't been put forward as a PEP.

=)

> The slightly longer answer is that there are related problems in terms of
> initialising debuggers, warnings, logging, faulthandler, coverage,
> profiling, etc, during development where it would be incredibly
> convenient to be able to execute a short snippet of Python code prior
> to execution.

Yeah, that's been painful.

> See http://bugs.python.org/issue14803 (I'm nervous about the security
> implications of a PYTHONRUNFIRST environment variable that is enabled
> by default,

Rightly so.

> but the idea of a -C option doesn't pose any such
> concerns)

Cool. I implemented something like that for "-m postgresql.bin.pg_python"[1]
Never ended up using it much.

But I could see myself using it with the Python executable: alias py='python3 -C ?'
Tucking that into a sh source file that sets up my dev context.

> In general though, messing with process global state (including the
> import system) implicitly is just *not cool*. It leads to applications
> that have a dependency on a particular tool being present, but don't
> actually make sure that it is available. Thus you end up with manual
> instructions that say "before running this script, make sure
> infrastructure X is in place to automagically take care of things I
> should be doing myself".


Even in the case of some designated mechanism where there is some
attempt to expose/record changes to process global state so that it can
be easily referenced by a dependency, is there a win over making an explicit
attempt to validate that the state change occurred by the dependency?

I think in this particular case involving meta_path, there would be a
small win with regards to being able to perform the check at package
installation time as opposed to every time the package is imported,
given a "designated mechanism" of sorts.
Essentially, the package being installed requests assurance that a state
change *will* occur upon Python startup.

[1] https://github.com/jwp/py-postgresql/blob/v1.1/postgresql/python/command.py

From shibturn at gmail.com  Mon Jul 30 18:44:09 2012
From: shibturn at gmail.com (Richard Oudkerk)
Date: Mon, 30 Jul 2012 17:44:09 +0100
Subject: [Python-ideas] Better support for finalization with weakrefs
Message-ID: <jv6dl2$v8r$1@dough.gmane.org>

I would like to see better support for the use of weakrefs callbacks
for object finalization.

The current issues with weakref callbacks include:

1. They are rather low level, and working out how to use them
    correctly requires a bit of head scratching.  One must find
    somewhere to store the weakref till after the referent is dead, and
    without accidentally keeping the referent alive.  Then one must
    ensure that the callback frees the weakref (without leaving any
    remnant ref-cycles).

    When it is an option, using a __del__ method is far less hassle.

2. Callbacks invoked during interpreter shutdown are troublesome.  For
    instance they can throw exceptions because module globals have been
    replaced by None.

3. Sometimes you want the callback to be called at program exit even
    if the referent is still alive.

4. Exceptions thrown in callbacks do not produce a traceback.  This
    can make debugging rather awkward.  (Maybe this is because printing
    tracebacks is problematic during interpreter shutdown?)

(Note that problems 2-4 also apply to using __del__ methods.)

If possible I would like to see the weakref module provide a finalize
class to address these issues.  Trivial usage might be

     >>> class Kenny: pass
     ...
     >>> kenny = Kenny()
     >>> finalize(kenny, print, "you killed kenny!")
     <finalize.finalize object at 0xffed4f2c>
     >>> del kenny
     you killed kenny!

Prototype at https://gist.github.com/3208245

Cheers,

Richard



From solipsis at pitrou.net  Mon Jul 30 19:27:34 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 30 Jul 2012 19:27:34 +0200
Subject: [Python-ideas] Better support for finalization with weakrefs
References: <jv6dl2$v8r$1@dough.gmane.org>
Message-ID: <20120730192734.73454970@pitrou.net>


Hi,

On Mon, 30 Jul 2012 17:44:09 +0100
Richard Oudkerk <shibturn at gmail.com>
wrote:
> 
> 2. Callbacks invoked during interpreter shutdown are troublesome.  For
>     instance they can throw exceptions because module globals have been
>     replaced by None.

A solution was proposed a long time ago at
http://bugs.python.org/issue812369

> 4. Exceptions thrown in callbacks do not produce a traceback.  This
>     can make debugging rather awkward.  (Maybe this is because printing
>     tracebacks is problematic during interpreter shutdown?)

No, it's just because PyErr_WriteUnraisable() doesn't print the
traceback. We could just fix that.

> (Note that problems 2-4 also apply to using __del__ methods.)
> 
> If possible I would like to see the weakref module provide a finalize
> class to address these issues.  Trivial usage might be
> 
>      >>> class Kenny: pass
>      ...
>      >>> kenny = Kenny()
>      >>> finalize(kenny, print, "you killed kenny!")
>      <finalize.finalize object at 0xffed4f2c>
>      >>> del kenny
>      you killed kenny!

Looks good!

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net




From ericsnowcurrently at gmail.com  Tue Jul 31 07:49:06 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 30 Jul 2012 23:49:06 -0600
Subject: [Python-ideas] only raise ImportError out of imports
Message-ID: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com>

Currently, if module 'eggs' has bad syntax and you import it from
module 'spam', then you will get a SyntaxError in module spam:

---------------------------------------------------------
$ cat > eggs.py << EOF
a +
EOF
$ cat > spam.py << EOF
import eggs
EOF
$ python -c 'import spam'
Traceback (most recent call last):
  File "spam.py", line 2, in <module>
    import eggs
  File "/tmp/eggs.py", line 1
    a +
      ^
SyntaxError: invalid syntax
---------------------------------------------------------

My proposal is that, instead, an error during import always results in
an ImportError.  So in the above example, the error would have been an
ImportError rather than a SyntaxError.  However, rather than ignore
the cause of the import error, we would use exception chaining to save
the SyntaxError.  To get this effect currently would look something
like this:

---------------------------------------------------------
$ cat > spam.py << EOF
try:
    import eggs
except SyntaxError as e:
    raise ImportError("failed to import eggs") from e
EOF
$ python -c 'import spam'
Traceback (most recent call last):
  File "spam.py", line 2, in <module>
    import eggs
  File "/tmp/eggs.py", line 1
    a +
      ^
SyntaxError: invalid syntax

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "spam.py", line 4, in <module>
    raise ImportError("failed to import eggs") from e
ImportError: failed to import eggs
---------------------------------------------------------

With exception chaining the SyntaxError is stored on the ImportError:

---------------------------------------------------------
$ cat > ham.py << EOF
try:
    import spam
except ImportError as e:
    print("cause: {}".format(e.__cause__))
    print("context: {}".format(e.__context__))
EOF
$ python ham.py
cause: invalid syntax (eggs.py, line 1)
context: invalid syntax (eggs.py, line 1)
---------------------------------------------------------

The main benefit of this change would be to isolate errors during
import to the module where they happen.  The main challenge would be
with backward compatibility, so I won't hold my breath to see this in
Python 3.

While all feedback is welcome, I'd particularly like to hear from
those that actually use non-ImportError exceptions that come out of
the import statement.  Thanks!

-eric


From ncoghlan at gmail.com  Tue Jul 31 08:00:44 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Jul 2012 16:00:44 +1000
Subject: [Python-ideas] only raise ImportError out of imports
In-Reply-To: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com>
References: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com>
Message-ID: <CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com>

On Tue, Jul 31, 2012 at 3:49 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> The main benefit of this change would be to isolate errors during
> import to the module where they happen.  The main challenge would be
> with backward compatibility, so I won't hold my breath to see this in
> Python 3.
>
> While all feedback is welcome, I'd particularly like to hear from
> those that actually use non-ImportError exceptions that come out of
> the import statement.  Thanks!

The problem specifically with doing this for SyntaxError is that you
would break every interactive interpreter out there (including the
default one). They rely on seeing the SyntaxError directly in order to
highlight the offending piece of syntax.

Also dig into pkgutil - that broke when I started tinkering with this
kind of thing while migrating it over to using importlib.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ericsnowcurrently at gmail.com  Tue Jul 31 08:14:30 2012
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 31 Jul 2012 00:14:30 -0600
Subject: [Python-ideas] only raise ImportError out of imports
In-Reply-To: <CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com>
References: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com>
	<CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com>
Message-ID: <CALFfu7Dh0weJ88uSxkGG5BM=eBRAbZG94Vn1HKTsgsJirvSUVA@mail.gmail.com>

On Tue, Jul 31, 2012 at 12:00 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Tue, Jul 31, 2012 at 3:49 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> The main benefit of this change would be to isolate errors during
>> import to the module where they happen.  The main challenge would be
>> with backward compatibility, so I won't hold my breath to see this in
>> Python 3.
>>
>> While all feedback is welcome, I'd particularly like to hear from
>> those that actually use non-ImportError exceptions that come out of
>> the import statement.  Thanks!
>
> The problem specifically with doing this for SyntaxError is that you
> would break every interactive interpreter out there (including the
> default one). They rely on seeing the SyntaxError directly in order to
> highlight the offending piece of syntax.
>
> Also dig into pkgutil - that broke when I started tinkering with this
> kind of thing while migrating it over to using importlib.

Cool.  I'll have a look.

-eric


From ironfroggy at gmail.com  Tue Jul 31 15:28:53 2012
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 31 Jul 2012 09:28:53 -0400
Subject: [Python-ideas] Better support for finalization with weakrefs
In-Reply-To: <jv6dl2$v8r$1@dough.gmane.org>
References: <jv6dl2$v8r$1@dough.gmane.org>
Message-ID: <CAGaVwhTQQmWKhdgUdt0VESSY=nhnj-E8rVcUkM_8BeCW-RU-hQ@mail.gmail.com>

On Mon, Jul 30, 2012 at 12:44 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> I would like to see better support for the use of weakrefs callbacks
> for object finalization.
>
> The current issues with weakref callbacks include:
>
> 1. They are rather low level, and working out how to use them
>    correctly requires a bit of head scratching.  One must find
>    somewhere to store the weakref till after the referent is dead, and
>    without accidentally keeping the referent alive.  Then one must
>    ensure that the callback frees the weakref (without leaving any
>    remnant ref-cycles).
>
>    When it is an option, using a __del__ method is far less hassle.
>
> 2. Callbacks invoked during interpreter shutdown are troublesome.  For
>    instance they can throw exceptions because module globals have been
>    replaced by None.
>
> 3. Sometimes you want the callback to be called at program exit even
>    if the referent is still alive.
>
> 4. Exceptions thrown in callbacks do not produce a traceback.  This
>    can make debugging rather awkward.  (Maybe this is because printing
>    tracebacks is problematic during interpreter shutdown?)
>
> (Note that problems 2-4 also apply to using __del__ methods.)
>
> If possible I would like to see the weakref module provide a finalize
> class to address these issues.  Trivial usage might be
>
>     >>> class Kenny: pass
>     ...
>     >>> kenny = Kenny()
>     >>> finalize(kenny, print, "you killed kenny!")
>     <finalize.finalize object at 0xffed4f2c>
>     >>> del kenny
>     you killed kenny!
>
> Prototype at https://gist.github.com/3208245

I like how simple it is, but it might be too simple. That is,
shouldn't we have a way to unregister the callback?

> Cheers,
>
> Richard
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From x at jwp.io  Tue Jul 31 23:59:24 2012
From: x at jwp.io (James William Pye)
Date: Tue, 31 Jul 2012 14:59:24 -0700
Subject: [Python-ideas] Python Environment Initialization Scripts
In-Reply-To: <C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io>
References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io>
	<CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com>
	<95A84799-151A-4AA9-B1FC-68346309C602@jwp.io>
	<CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com>
	<C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io>
	<CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com>
	<C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io>
Message-ID: <B5E2A6BA-128E-465F-9330-B76DA8BDDC9C@jwp.io>

On Jul 29, 2012, at 2:44 AM, James William Pye <x at jwp.io> wrote:

> On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> The short answer is that, no, a coherent proposal for application
>> specific preinitialisation hasn't been put forward as a PEP.

sigh. *.pth files in site-packages/ can do it.

They can run imports.. I think this is the second time I've been surprised by that? =\