From tjreedy at udel.edu Sun Jul 1 00:01:08 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 30 Jun 2012 18:01:08 -0400 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CAMgkT__4EUv-Qwi8mu3bg3iwunszOcJ-_ZY4GsV81_UJriTB1A@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <CAEjd=rqy2+M34yPSnJLY834D6=Dk3050+GjPpvoE850dfjk2xQ@mail.gmail.com> <CAMgkT__4EUv-Qwi8mu3bg3iwunszOcJ-_ZY4GsV81_UJriTB1A@mail.gmail.com> Message-ID: <jsnsvd$bto$1@dough.gmane.org> On 6/30/2012 11:54 AM, Christopher Reay wrote: > The only hope for a large archive like this one is to wait long enough > to make sure you dont re hash the really regular ideas. > > ... ponders... Do I have time to read the archives? If you have time to post and discuss an idea, you have time to search gmane.comp.python.ideas at search.gmane.org and perhaps find out why not to bother posting. ? Do people mind adminiing the repetitive ideas? yes. -- Terry Jan Reedy From tjreedy at udel.edu Sun Jul 1 00:57:58 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 30 Jun 2012 18:57:58 -0400 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <CAEBZo3M2gcM5MmXHVXr+Y4ObeXL=_jScZBOivJq88ZY7mAMPxg@mail.gmail.com> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <CAEBZo3M2gcM5MmXHVXr+Y4ObeXL=_jScZBOivJq88ZY7mAMPxg@mail.gmail.com> Message-ID: <4FEF8476.1050609@udel.edu> On 6/29/2012 5:36 PM, Mike Graham wrote: > On Fri, Jun 29, 2012 at 4:32 PM, Georg Brandl <g.brandl at gmx.net> wrote: >> so far there were no negative votes > > As far as I know, Raymond Hettinger is the itertools maintainer and he > has repeatedly objected to this idea in the past (e.g. > http://bugs.python.org/issue6021 ). Hopefully we can get his input > again. See my other post on this thread. I think people should really spend a few minutes researching before posting repeated ideas. In this case, read the itertools doc to see if such a function exists. It does, in the recipes, as 'grouper'. Has anyone ever before proposed that the recipe be added as a function? Search the tracker for 'itertools grouper'. Besides #6021, there are also http://bugs.python.org/issue1643 'Add group() to itertools' Raymond: "Sorry, I'm not interested in adding this to the module. Discussions to- date on the subject seem to show more interest in playing with grouper variants than in actual use cases. While the recipe given in the docs is somewhat opaque, it runs at C-speed (zero trips around the eval- loop) and it is encapsulated in a re-usable function. Writing this in C does nothing to improve the situation. Also, when people like to play with variants, there is no general agreement on useful requirements (like fill-in behavior or raising an exception on uneven length inputs). Trying to write option to meet all needs (n=2, step=1) makes the code more difficult to learn and use -- see several variants in Alex's Python Cookbook. Another issue is that we have to be very selective about adding tools to the module. Each addition makes the overall toolset harder to use -- it is better to have a good set of basic building blocks." http://bugs.python.org/issue13095 "Support for splitting lists/tuples into chunks" Raymond: "These have been rejected before. There is always a trade-off in adding tools such as this -- it can take more time to learn and remember them than to write a trivial piece of code to do it yourself. Another issue is that people tend to disagree on how to handle an odd sized left-over group -- different use cases require different handling. We're trying to keep the core toolset reasonably small so that python remains simple and learnable. That raises the threshold for adding new tools." There are now 18 itertools, up from the original. Grouper, or any generic function, may not be the best for what one wants with a list. I proposed in my other post that we *do* need a new doc section or how-to on this topic. (I am working on an outline.) -- Terry Jan Reedy From steve at pearwood.info Sun Jul 1 02:24:47 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 Jul 2012 10:24:47 +1000 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CAGaVwhSHx_pmhEc=ncc6yFPqo_Up3sAMzZ9DJvTF-KgFof1CVw@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CAGaVwhSHx_pmhEc=ncc6yFPqo_Up3sAMzZ9DJvTF-KgFof1CVw@mail.gmail.com> Message-ID: <4FEF98CF.5050804@pearwood.info> Calvin Spealman wrote: > On Sat, Jun 30, 2012 at 9:59 AM, <fiatjaf at yahoo.com.br> wrote: >> the idea is to make an variable assignment at the same time that the >> existence of that variable -- which is being returned by a function -- is >> made. >> >> suppose we are returning a variable from the method 'get' from the 'request' >> object and them making some stuff with it, but that stuff we will only do if >> it exists, if not, we'll just pass, instead of writing: >> >> variable = self.request.get('variable') >> if variable: >> print variable >> >> we could write >> >> if self.request.get('variable') as variable: >> print variable >> >> seems stupid (or not?), but with lots of variables to process, this >> pre-assignment could be very unpleasant -- especially if, as the in the >> example case, very little use will be made of the tested variable. >> >> also, the "as" expression already exists and is very pythonic. > > This is probably the best solution to the problem What problem? I'm not convinced that what fiatjaf at yahoo.com.br describes is an actual problem that needs solving. So what if you have to bind a reference to a name before using it in an if statement? That's not a problem. "Solving" it doesn't make programming any easier, or give you more power. There is absolutely no difference in programming power or expressiveness between: if func(a, b) as x: process(x) and x = func(a, b) if x: process(x) (In fact, I would argue that the second is *slightly* more readable, as the name binding comes first, not last.) At best, this proposal merely adds a trivial bit of syntactic sugar to the language. Contrast that with "import as", which truly does add expressiveness to the language. Working around the lack of "import as" if Python didn't have it would not be so easy. The obvious solution is also the wrong solution: # import long_name as ln import long_name ln = long_name del long_name That's wrong, because it has the side-effect of replacing, then deleting, any existing binding to long_name. "import as" does not do that. Good suggestions for syntactic sugar should add power or expressiveness to the language, not merely save a line of code or a few keystrokes. As given, this idea becomes one more special case for people to learn, with no corresponding benefit. -1 on the idea as given. By the way, to the Original Poster, please configure your mail client to display a name that we can refer to you by, or sign your posts. It doesn't have to be your real name, just something that you would like to be known as. It is annoying to have to refer to you by your email address. -- Steven From mwm at mired.org Sun Jul 1 02:48:52 2012 From: mwm at mired.org (Mike Meyer) Date: Sat, 30 Jun 2012 20:48:52 -0400 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> Message-ID: <20120630204852.4b898a0a@bhuda.mired.org> On Sun, 1 Jul 2012 01:06:39 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > This proposal has been considered and rejected many times. It's not > general enough - it *only* works for those cases where the value to be > retained *and* the interesting condition are the same. > > Consider the simple case of a value that may be either None (not > interesting) or a number (interesting). Since the interesting values > include "0", which evaluates as False along with None, this limited > form of embedded assignment syntax would not help. What Nick failed to point out is that the existing uses of "as" all bind *specific classes of objects*, not general ones. > Embedded assignment in C isn't that limited., but nobody has yet > volunteered to take the radical step of proposing "(X as Y)" as a > general embedded assignment syntax. I suggest anyone consider such an > idea do a *lot* of research in the python-ideas archives first, though > (as the idea has seen plenty of discussion). It is not as obviously > flawed as the if-and-while statement only variant, but it would still > involve being rather persuasive to make such a significant change to > the language. Hasn't the BDFL rejected a general embedded assignment in any case? As a final note, reading checking the archives for your ideas will also give you an idea of what kinds of things are and aren't accepted (i.e. - what's "Pythonic") as well as process - including the kinds of questions you'll be asked here - that an idea goes through before being accepted. <mike -- Mike Meyer <mwm at mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From steve at pearwood.info Sun Jul 1 02:59:23 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 Jul 2012 10:59:23 +1000 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsn7um$2j1$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> Message-ID: <4FEFA0EB.4030407@pearwood.info> Serhiy Storchaka wrote: > As shown in issue #15016 [1], there is a use cases when it is useful to > determine that string can be encoded in ASCII or Latin1. In working with > Tk or Windows console applications can be useful to determine that > string can be encoded in UCS2. C API provides interface for this, but at > Python level it is not available. > > I propose to add to strings class new methods: isascii(), islatin1() and > isbmp() (in addition to such methods as isalpha() or isdigit()). The > implementation will be trivial. > > Pro: The current trick with trying to encode has O(n) complexity and has > overhead of exception raising/catching. Are you suggesting that isascii and friends would be *better* than O(n)? How can that work -- wouldn't it have to scan the string and look at each character? Why just ASCII, Latin1 and BMP (whatever that is, googling has not come up with anything relevant)? It seems to me that adding these three tests will open the doors to a steady stream of requests for new methods is<insert encoding name here>. I suggest that a better API would be a method that takes the name of an encoding (perhaps defaulting to 'ascii') and returns True|False: string.encodable(encoding='ascii') -> True|False Return True if string can be encoded using the named encoding, otherwise False. One last pedantic issue: strings aren't ASCII or Latin1, etc., but Unicode. There is enough confusion between Unicode text strings and bytes without adding methods whose names blur the distinction slightly. -- Steven From greg.ewing at canterbury.ac.nz Sun Jul 1 01:22:26 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 01 Jul 2012 11:22:26 +1200 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsnbel$pua$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> <CADiSq7e_hGXhjVuHjAUMV4WsdOMVQMMH5duhtja5pGG73tBV2A@mail.gmail.com> <20120630184316.5de2ce5e@pitrou.net> <jsnbel$pua$1@dough.gmane.org> Message-ID: <4FEF8A32.50607@canterbury.ac.nz> Serhiy Storchaka wrote: > Several enchantments have already been rejected for this reason. ^^^^^^^^^^^^ Yeah, programming does seem to be a black art sometimes... -- Greg From tjreedy at udel.edu Sun Jul 1 04:05:21 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 30 Jun 2012 22:05:21 -0400 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <4FEFA0EB.4030407@pearwood.info> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> Message-ID: <jsob9b$qo7$1@dough.gmane.org> On 6/30/2012 8:59 PM, Steven D'Aprano wrote: > Why just ASCII, Latin1 and BMP (whatever that is, googling has not come > up with anything relevant)? BMP = Unicode Basic MultilingualPlane, the first 2**16 codepoints http://unicode.org/roadmaps/bmp/ I presume the proposed isbmp would exclude surrogates in 16 bit implementations, but that was not clearly defined. > It seems to me that adding these three tests The temptation for these three tests is the the info is already available (at least for 2 of them) as an internal implementation-specific C-level attribute in 3.3(+). No O(n) scan needed. Someone could make a CPython3.3-specific module available on PyPI. > will open the doors to a steady stream of requests for new methods > is<insert encoding name here>. > I suggest that a better API would be a method that takes the name of an > encoding (perhaps defaulting to 'ascii') and returns True|False: > > string.encodable(encoding='ascii') -> True|False > > Return True if string can be encoded using the named encoding, otherwise > False. But then one might as well try the encoding and check for exception. The point of the proposal is to avoid things like try: body = text.encode('ascii') header = 'ascii' #abbreviating here except UnicodeEncodeError: try: body = text.encode('latin1') header = 'latin1' except UnicodeEncodeError: body = text.encode('utf-8') header = 'utf-8' > One last pedantic issue: strings aren't ASCII or Latin1, etc., but > Unicode. There is enough confusion between Unicode text strings and > bytes without adding methods whose names blur the distinction slightly. yes! -- Terry Jan Reedy From steve at pearwood.info Sun Jul 1 05:21:32 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 01 Jul 2012 13:21:32 +1000 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsob9b$qo7$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> Message-ID: <4FEFC23C.3040805@pearwood.info> Terry Reedy wrote: > On 6/30/2012 8:59 PM, Steven D'Aprano wrote: >> I suggest that a better API would be a method that takes the name of an >> encoding (perhaps defaulting to 'ascii') and returns True|False: >> >> string.encodable(encoding='ascii') -> True|False >> >> Return True if string can be encoded using the named encoding, otherwise >> False. > > But then one might as well try the encoding and check for exception. The > point of the proposal is to avoid things like > > try: > body = text.encode('ascii') > header = 'ascii' #abbreviating here > except UnicodeEncodeError: > try: > body = text.encode('latin1') > header = 'latin1' > except UnicodeEncodeError: > body = text.encode('utf-8') > header = 'utf-8' Right. And re-written with the hypothetical encodable method, you have the usual advantage of LBYL that it is slightly more concise: body = header = None for encoding in ('ascii', 'latin1', 'utf-8'): if text.encodable(encoding): body = text.encode(encoding) header = encoding instead of: body = header = None for encoding in ('ascii', 'latin1', 'utf-8'): try: body = text.encode(encoding) header = encoding except UnicodeEncodeError: pass As for as expressibility goes, it is not much of an advantage. But: - if there are optimizations that apply to some encodings but not others, the encodable method can take advantage of them without it being a promise of the language; - it only adds a single string method (and presumably a single bytes method, decodable) rather than a plethora of methods; So, I don't care much either way for a LBYL test, but if there is a good use case for such a test, better for it to be a single method taking the encoding name rather than a multitude of tests, or exposing an implementation-specific value that the coder then has to interpret themselves. -1 on isascii, islatin1, isbmp -1 on exposing max_code_point +0.5 on encodable -- Steven From storchaka at gmail.com Sun Jul 1 07:31:05 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 01 Jul 2012 08:31:05 +0300 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <4FEF8A32.50607@canterbury.ac.nz> References: <jsn7um$2j1$1@dough.gmane.org> <CADiSq7e_hGXhjVuHjAUMV4WsdOMVQMMH5duhtja5pGG73tBV2A@mail.gmail.com> <20120630184316.5de2ce5e@pitrou.net> <jsnbel$pua$1@dough.gmane.org> <4FEF8A32.50607@canterbury.ac.nz> Message-ID: <json9m$r67$1@dough.gmane.org> On 01.07.12 02:22, Greg Ewing wrote: > Serhiy Storchaka wrote: >> Several enchantments have already been rejected for this reason. > ^^^^^^^^^^^^ > > Yeah, programming does seem to be a black art sometimes... Sorry, I mean enhancements. But yes, there is no good programming without magic. From stefan_ml at behnel.de Sun Jul 1 07:44:56 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Jul 2012 07:44:56 +0200 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <4FEF6B10.9040409@udel.edu> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> Message-ID: <jsoo4p$via$1@dough.gmane.org> Terry Reedy, 30.06.2012 23:09: > For the doc, I think it would be helpful here and in most module > subchapters if there were a subchapter table of contents at the top (under > 9.1 in this case). Even though just 2 lines here (currently, but see > below), it would let people know that there *is* a recipes section. After > the appropriate tables, mention that there are example uses in the recipe > section. Possibly add similar tables in the recipe section. +1, the recipes are interesting enough to be highly visible, both as usage examples and to solve actual problems. > Another addition could be a new subsection on grouping (chunking) that > would discuss post-processing of grouper (as discussed above), as well as > other recipes, including ones specific to strings and sequences. It would > essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or > Chunking Sequences and Iterables". The synonyms will help external > searching. A toc would let people who have found this doc know to look for > this at the bottom. If it really is such an important use case for so many people, I agree that it's worth special casing it in the docs. It's not a trivial algorithmic step from a sequential iterable to a grouped iterable. Stefan From tjreedy at udel.edu Sun Jul 1 07:48:14 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Jul 2012 01:48:14 -0400 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <4FEFC23C.3040805@pearwood.info> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> Message-ID: <jsooav$11k$1@dough.gmane.org> On 6/30/2012 11:21 PM, Steven D'Aprano wrote: > Terry Reedy wrote: >> On 6/30/2012 8:59 PM, Steven D'Aprano wrote: > >>> I suggest that a better API would be a method that takes the name of an >>> encoding (perhaps defaulting to 'ascii') and returns True|False: >>> >>> string.encodable(encoding='ascii') -> True|False >>> >>> Return True if string can be encoded using the named encoding, otherwise >>> False. >> >> But then one might as well try the encoding and check for exception. >> The point of the proposal is to avoid things like >> >> try: >> body = text.encode('ascii') >> header = 'ascii' #abbreviating here >> except UnicodeEncodeError: >> try: >> body = text.encode('latin1') >> header = 'latin1' >> except UnicodeEncodeError: >> body = text.encode('utf-8') >> header = 'utf-8' > > Right. And re-written with the hypothetical encodable method, you have > the usual advantage of LBYL that it is slightly more concise: > > body = header = None > for encoding in ('ascii', 'latin1', 'utf-8'): > if text.encodable(encoding): > body = text.encode(encoding) > header = encoding But you are doing about half the work twice. > instead of: > > body = header = None > for encoding in ('ascii', 'latin1', 'utf-8'): > try: > body = text.encode(encoding) > header = encoding > except UnicodeEncodeError: > pass > As for as expressibility goes, it is not much of an advantage. But: > > - if there are optimizations that apply to some encodings but not others, > the encodable method can take advantage of them without it being a > promise of the language; It would be an optimization limited to a couple of encodings with CPython. Using it for cross-version code would be something like the trap of depending on the CPython optimization of repeated string concatenation. > - it only adds a single string method (and presumably a single bytes > method, decodable) rather than a plethora of methods; Decodable would always require a scan of the bytes. Might as well just decode and look for UnicodeDecodeError. > So, I don't care much either way for a LBYL test, but if there is a good > use case for such a test, My claim is that there is only a good use case if it is O(1), which would only be a few cases on CPython. > better for it to be a single method taking the > encoding name rather than a multitude of tests, or exposing an > implementation-specific value that the coder then has to interpret > themselves. > > -1 on isascii, islatin1, isbmp I do not see much of any use for isbmp. Maybe I missed something in the original post. > -1 on exposing max_code_point Jython and IronPython are stuck with the underlying platform implementations, which I believe are like the current semi-utf-16 narrow builds. So it would have to be a CPython-only attribute for now. (PyPy might consider adopting the new Unicode implementation someday too.) > +0.5 on encodable encodable would indirectly expose max_code_point since it would only be really useful and likely used when max_code_point was available and applicable. In other words, s.encodable('latin1') is equivalent to s.max_code_point == 255. if isbmp *is* useful, I don't think it can be duplicated with .encodable. Python seems not to have a ucs-2 codec. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Jul 1 08:27:25 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Jul 2012 16:27:25 +1000 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsooav$11k$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> <jsooav$11k$1@dough.gmane.org> Message-ID: <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com> On Sun, Jul 1, 2012 at 3:48 PM, Terry Reedy <tjreedy at udel.edu> wrote: > encodable would indirectly expose max_code_point since it would only be > really useful and likely used when max_code_point was available and > applicable. In other words, s.encodable('latin1') is equivalent to > s.max_code_point == 255. > > if isbmp *is* useful, I don't think it can be duplicated with .encodable. > Python seems not to have a ucs-2 codec. Rewinding back to the reasons the question is being asked, the reason this information is useful at the Python level is the same reason it is useful at the C level: it matters for finding the most efficient means of representing the text as bytes (which can then have further implications for the kind of quoting used, etc). The interesting breakpoints can actually be expressed in terms of the number of bits in the highest code point: 7 - encode as ASCII (or latin-1 or utf-8) 8 - encode as latin-1 8+ - encode as utf-8 Specifically, it's a payload microoptimisation for the latin-1 case - the latin-1 string will be shorter than the corresponding utf-8 string (how much shorter depends on the number of non-ASCII characters). I believe it also makes an additional difference in the email case by changing the kind of quoting that is used to something with lower overhead that can't handle utf-8. The "try it and see" approach suffers a potentially high speed penalty if the non-latin-1 characters appear late in the string: try: # Likely no need to try ASCII, since there's no efficiency gain over latin-1 payload = message.encode("latin-1") except UnicodeEncodeError: payload = message.encode("utf-8") Using max() and ord() to check in advance doesn't help, since that *locks in* the O(n) penalty. The reason I think a max_code_point() method is a good potential solution is that it can be advertised as O(n) worst case, but potentially O(1) if the implementation caches the answer internally. Another alternative would be a __max__ and __min__ protocol that allowed efficient answers for the max() and min() builtins. The latter would have the advantage of allowing other containers (like range objects) to provide efficient implementations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From raymond.hettinger at gmail.com Sun Jul 1 09:07:56 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 1 Jul 2012 00:07:56 -0700 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <jsoo4p$via$1@dough.gmane.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <jsoo4p$via$1@dough.gmane.org> Message-ID: <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com> On Jun 30, 2012, at 10:44 PM, Stefan Behnel wrote: >> >> Another addition could be a new subsection on grouping (chunking) that >> would discuss post-processing of grouper (as discussed above), as well as >> other recipes, including ones specific to strings and sequences. It would >> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or >> Chunking Sequences and Iterables". The synonyms will help external >> searching. A toc would let people who have found this doc know to look for >> this at the bottom. > > If it really is such an important use case for so many people, I agree that > it's worth special casing it in the docs. It's not a trivial algorithmic > step from a sequential iterable to a grouped iterable. I'm not too keen on adding a section like this to the itertools docs. Instead, I would be open adding "further reading" section with external links to interesting iterator writeups in blogs, cookbooks, stack overflow answers, wikis, etc. If one of you wants to craft an elegant blog post on "Grouping, Blocking, or Chunking Sequences and Iterables", I would be happy to link to it. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120701/0532fc7f/attachment.html> From solipsis at pitrou.net Sun Jul 1 18:22:33 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Jul 2012 18:22:33 +0200 Subject: [Python-ideas] Happy leap second References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com> Message-ID: <20120701182233.589552b5@pitrou.net> On Sat, 30 Jun 2012 10:23:45 -0400 Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > Even though many have hoped that the authorities would stop fiddling > with our clocks, today a leap second will be inserted in UTC. > Systems using Olson/IANA timezone database have a way to deal with > this without adjusting their clocks, but few systems are configured > that way: > > $ TZ=right/UTC date -d @1341100824 > Sat Jun 30 23:59:60 UTC 2012 If I believe the Internet, many server systems got broken by the leap second (*). Even one of our buildbots seems affected: - before: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/419 - after: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/420 (weird errors with timeouts, and a crash) (*) https://linuxfr.org/users/nono/journaux/leap-second Regards Antoine. From guido at python.org Sun Jul 1 21:11:25 2012 From: guido at python.org (Guido van Rossum) Date: Sun, 1 Jul 2012 21:11:25 +0200 Subject: [Python-ideas] Happy leap second In-Reply-To: <20120701182233.589552b5@pitrou.net> References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com> <20120701182233.589552b5@pitrou.net> Message-ID: <CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com> So would "supporting" the leap second in Python have made any difference? On Sun, Jul 1, 2012 at 6:22 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sat, 30 Jun 2012 10:23:45 -0400 > Alexander Belopolsky > <alexander.belopolsky at gmail.com> wrote: > >> Even though many have hoped that the authorities would stop fiddling >> with our clocks, today a leap second will be inserted in UTC. >> Systems using Olson/IANA timezone database have a way to deal with >> this without adjusting their clocks, but few systems are configured >> that way: >> >> $ TZ=right/UTC date -d @1341100824 >> Sat Jun 30 23:59:60 UTC 2012 > > If I believe the Internet, many server systems got broken by the leap > second (*). > Even one of our buildbots seems affected: > - before: > http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/419 > - after: > http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/420 > (weird errors with timeouts, and a crash) > > > (*) https://linuxfr.org/users/nono/journaux/leap-second > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Sun Jul 1 21:43:35 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Jul 2012 21:43:35 +0200 Subject: [Python-ideas] Happy leap second References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com> <20120701182233.589552b5@pitrou.net> <CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com> Message-ID: <20120701214335.59220b05@pitrou.net> On Sun, 1 Jul 2012 21:11:25 +0200 Guido van Rossum <guido at python.org> wrote: > So would "supporting" the leap second in Python have made any difference? No. These look like system (kernel?) bugs. Regards Antoine. From ncoghlan at gmail.com Mon Jul 2 00:14:03 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Jul 2012 08:14:03 +1000 Subject: [Python-ideas] Happy leap second In-Reply-To: <20120701214335.59220b05@pitrou.net> References: <CAP7h-xYJpZWcOQXEEy6RUfZHiYxgNY9BkE=jHQ4Mgo3OTWEBRQ@mail.gmail.com> <20120701182233.589552b5@pitrou.net> <CAP7+vJLWWrWWS=T9UPrB9p2E_UDfBzJac118MjK0M5+L7M5oTA@mail.gmail.com> <20120701214335.59220b05@pitrou.net> Message-ID: <CADiSq7eME-+uy==QLGQofZ5mno=b9j_pEf8_xOPv7z9P-EWJmg@mail.gmail.com> It was a kernel race condition (and subsequent live lock) after ntpd notified the clock subsystem of the coming leap second ( http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second) The joys of fine grained locking :) Cheers, Nick. -- Sent from my phone, thus the relative brevity :) On Jul 2, 2012 5:45 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > On Sun, 1 Jul 2012 21:11:25 +0200 > Guido van Rossum <guido at python.org> wrote: > > So would "supporting" the leap second in Python have made any difference? > > No. These look like system (kernel?) bugs. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120702/1c9f243a/attachment.html> From steve at pearwood.info Mon Jul 2 10:38:19 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Jul 2012 18:38:19 +1000 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsooav$11k$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> <jsooav$11k$1@dough.gmane.org> Message-ID: <20120702083819.GB27130@ando> On Sun, Jul 01, 2012 at 01:48:14AM -0400, Terry Reedy wrote: > >As for as expressibility goes, it is not much of an advantage. But: > > > >- if there are optimizations that apply to some encodings but not others, > > the encodable method can take advantage of them without it being a > > promise of the language; > > It would be an optimization limited to a couple of encodings with > CPython. Using it for cross-version code would be something like the > trap of depending on the CPython optimization of repeated string > concatenation. I'd hardly call it a trap. It's not like string concatenation which is expected to be O(N**2) on CPython but occasionally falls back to O(N**2). It would be O(N) expected on all platforms, but occasionally does better. Perhaps an anti-trap -- sometimes it does better than expected, rather than worse. > >- it only adds a single string method (and presumably a single bytes > > method, decodable) rather than a plethora of methods; > > Decodable would always require a scan of the bytes. Might as well just > decode and look for UnicodeDecodeError. *shrug* Perhaps so. bytes.decodable() would only be a LBYL convenience method. > >So, I don't care much either way for a LBYL test, but if there is a good > >use case for such a test, > > My claim is that there is only a good use case if it is O(1), which > would only be a few cases on CPython. *shrug* Again, I'm not exactly championing this proposal. I can see that an encodable method would be useful, but not that much more useful than trying to encode and catching the exception. A naive O(N) version of encodable() is trivial to implement. -- Steven From steve at pearwood.info Mon Jul 2 10:52:24 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Jul 2012 18:52:24 +1000 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> <jsooav$11k$1@dough.gmane.org> <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com> Message-ID: <20120702085224.GC27130@ando> On Sun, Jul 01, 2012 at 04:27:25PM +1000, Nick Coghlan wrote: > Rewinding back to the reasons the question is being asked, the reason > this information is useful at the Python level is the same reason it > is useful at the C level: it matters for finding the most efficient > means of > representing the text as bytes (which can then have further > implications for the kind of quoting used, etc). The interesting > breakpoints can actually be expressed in terms of the number of bits > in the highest code point: > 7 - encode as ASCII (or latin-1 or utf-8) > 8 - encode as latin-1 > 8+ - encode as utf-8 I'm of two minds here. On the one hand, I question the wisdom of encouraging the use of anything but UTF-8. It's unfortunate enough that there are still cases where people have to use older encodings, without encouraging people to use Latin1 or ASCII in order to save a handful of bytes in a 20K email. On the other hand, there are use-cases for non-UTF-8 encodings, and people will want to check whether or not a string is encodable in various encodings. Why make that harder/slower/less convenient than it need be? > Specifically, it's a payload microoptimisation for the latin-1 case - > the latin-1 string will be shorter than the corresponding utf-8 string Just to be clear here, you're referring to byte strings, yes? > (how much shorter depends on the number of non-ASCII characters). I > believe it also makes an additional difference in the email case by > changing the kind of quoting that is used to something with lower > overhead that can't handle utf-8. > > The "try it and see" approach suffers a potentially high speed penalty > if the non-latin-1 characters appear late in the string: > > try: > # Likely no need to try ASCII, since there's no efficiency > gain over latin-1 > payload = message.encode("latin-1") > except UnicodeEncodeError: > payload = message.encode("utf-8") > > Using max() and ord() to check in advance doesn't help, since that > *locks in* the O(n) penalty. > > The reason I think a max_code_point() method is a good potential > solution is that it can be advertised as O(n) worst case, but > potentially O(1) if the implementation caches the answer internally. The downside is that the caller is then responsible for interpreting that value (i.e. mapping a max code point to an encoding). The other downside is that doesn't do anything to help those who are stuck with legacy encodings. Although maybe that doesn't matter, since they will just do the "try it and see" approach. > Another alternative would be a __max__ and __min__ protocol that > allowed efficient answers for the max() and min() builtins. The latter > would have the advantage of allowing other containers (like range > objects) to provide efficient implementations. +1 on that, although I think that should be a separate issue. -- Steven From ironfroggy at gmail.com Mon Jul 2 12:45:43 2012 From: ironfroggy at gmail.com (Calvin Spealman) Date: Mon, 2 Jul 2012 06:45:43 -0400 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jsn7um$2j1$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> Message-ID: <CAGaVwhRAtaOmdaArg-yES+CiDAdOPgh+eHrV9XK9hwwNXag9yQ@mail.gmail.com> On Sat, Jun 30, 2012 at 12:03 PM, Serhiy Storchaka <storchaka at gmail.com> wrote: > As shown in issue #15016 [1], there is a use cases when it is useful to > determine that string can be encoded in ASCII or Latin1. In working with Tk > or Windows console applications can be useful to determine that string can > be encoded in UCS2. C API provides interface for this, but at Python level > it is not available. > > I propose to add to strings class new methods: isascii(), islatin1() and > isbmp() (in addition to such methods as isalpha() or isdigit()). The > implementation will be trivial. > > Pro: The current trick with trying to encode has O(n) complexity and has > overhead of exception raising/catching. > > Contra: In most cases after determining characters range we still need to > encode a string with the appropriate encoding. New methods will complicate > already overloaded strings class. > > Objections? -1 It doesn't make sense to special case them, instead of a simpler canencode() method added. It could save memory, but I don't see it saving time. > [1] http://bugs.python.org/issue15016 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From storchaka at gmail.com Mon Jul 2 13:32:50 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 02 Jul 2012 14:32:50 +0300 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <20120702085224.GC27130@ando> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> <jsooav$11k$1@dough.gmane.org> <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com> <20120702085224.GC27130@ando> Message-ID: <jss0rs$6a9$1@dough.gmane.org> On 02.07.12 11:52, Steven D'Aprano wrote: >> Another alternative would be a __max__ and __min__ protocol that >> allowed efficient answers for the max() and min() builtins. The latter >> would have the advantage of allowing other containers (like range >> objects) to provide efficient implementations. > > +1 on that, although I think that should be a separate issue. This is issue #15226. From tjreedy at udel.edu Mon Jul 2 18:35:58 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 02 Jul 2012 12:35:58 -0400 Subject: [Python-ideas] isascii()/islatin1()/isbmp() In-Reply-To: <jss0rs$6a9$1@dough.gmane.org> References: <jsn7um$2j1$1@dough.gmane.org> <4FEFA0EB.4030407@pearwood.info> <jsob9b$qo7$1@dough.gmane.org> <4FEFC23C.3040805@pearwood.info> <jsooav$11k$1@dough.gmane.org> <CADiSq7fYRAd_gDuTnWNZ3gcHZM4fwGcJ6_5gkXbSLegBzZ_8UQ@mail.gmail.com> <20120702085224.GC27130@ando> <jss0rs$6a9$1@dough.gmane.org> Message-ID: <jssilh$3md$1@dough.gmane.org> On 7/2/2012 7:32 AM, Serhiy Storchaka wrote: > On 02.07.12 11:52, Steven D'Aprano wrote: >>> Another alternative would be a __max__ and __min__ protocol that >>> allowed efficient answers for the max() and min() builtins. The latter >>> would have the advantage of allowing other containers (like range >>> objects) to provide efficient implementations. >> >> +1 on that, although I think that should be a separate issue. > > This is issue #15226. http://bugs.python.org/issue15226 was about exposing the max codepoint, which the OP thought was readily available in C, but is not. (It is at creation, but it is then replaced by 1 of 4 values.) -- Terry Jan Reedy From raymond.hettinger at gmail.com Tue Jul 3 06:23:29 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 2 Jul 2012 21:23:29 -0700 Subject: [Python-ideas] [Python-Dev] itertools.chunks(iterable, size, fill=None) In-Reply-To: <jspe70$4gq$1@dough.gmane.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <jsoo4p$via$1@dough.gmane.org> <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com> <jspe70$4gq$1@dough.gmane.org> Message-ID: <877C3C8D-1DE3-4DB0-812A-B352E8608364@gmail.com> On Jul 1, 2012, at 5:01 AM, Stefan Behnel wrote: > To address the main problem of users not finding what they need, what about > simply extending the docstring of the grouper() Here's a small change to the docstring: http://hg.python.org/cpython/rev/d32f21d87363 FWIW, if you're interested in load balancing applications, George Sakkis's itertools recipe for roundrobin() may be of interest. Another interesting iterator technique that is not well known is the two-argument form of iter() which is a marvel for transforming callables into iterators: for block in iter(partial(f.read, 1024), ''): ... for diceroll in iter(partial(randrange(1, 7), 4): ... Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120702/8550f12c/attachment.html> From techtonik at gmail.com Wed Jul 4 11:57:20 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 4 Jul 2012 12:57:20 +0300 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <jsl3cb$knn$1@dough.gmane.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> Message-ID: <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote: > On 26.06.2012 10:03, anatoly techtonik wrote: >> >> Now that Python 3 is all about iterators (which is a user killer >> feature for Python according to StackOverflow - >> http://stackoverflow.com/questions/tagged/python) would it be nice to >> introduce more first class functions to work with them? One function >> to be exact to split string into chunks. >> >> itertools.chunks(iterable, size, fill=None) >> >> Which is the 33th most voted Python question on SO - >> >> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464 >> >> P.S. CC'ing to python-dev@ to notify about the thread in python-ideas. >> > > Anatoly, so far there were no negative votes -- would you care to go > another step and propose a patch? Was about to say "no problem", but in fact - there is. Sorry from whining from my side and thanks for nudging. The only thought that a simple task of copy/pasting relevant code from http://docs.python.org/library/itertools.html?highlight=itertools#recipes will require a few hours waiting of download (still not everybody has a high-speed internet) makes me switch to other less time consuming tasks before getting around to it. These tasks become more important in a few hours, and basically I've passed through this many times before. It then becomes quite hard to switch back. I absolutely don't mind someone else being credited for the idea, because ideas usually worthless without implementation. It will be interesting to design how the process could work in a separate thread. For now the best thing I can do (I don't risk even to mention anything with 3.3) is to copy/paste code from the docs here: from itertools import izip_longest def chunks(iterable, size, fill=None): """Split an iterable into blocks of fixed-length""" # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx args = [iter(iterable)] * size return izip_longest(fillvalue=fill, *args) BTW, this doesn't work as expected (at least for strings). Expected is: chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx' got: chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x') Needs more round tuits definitely. From anacrolix at gmail.com Wed Jul 4 18:19:10 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 5 Jul 2012 00:19:10 +0800 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <20120630204852.4b898a0a@bhuda.mired.org> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> Message-ID: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> There's no point doing this in Python since scoping is function-level and not lexical. Just move the assignment to the line preceding the if statement. On Sun, Jul 1, 2012 at 8:48 AM, Mike Meyer <mwm at mired.org> wrote: > On Sun, 1 Jul 2012 01:06:39 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> This proposal has been considered and rejected many times. It's not >> general enough - it *only* works for those cases where the value to be >> retained *and* the interesting condition are the same. >> >> Consider the simple case of a value that may be either None (not >> interesting) or a number (interesting). Since the interesting values >> include "0", which evaluates as False along with None, this limited >> form of embedded assignment syntax would not help. > > What Nick failed to point out is that the existing uses of "as" all > bind *specific classes of objects*, not general ones. > >> Embedded assignment in C isn't that limited., but nobody has yet >> volunteered to take the radical step of proposing "(X as Y)" as a >> general embedded assignment syntax. I suggest anyone consider such an >> idea do a *lot* of research in the python-ideas archives first, though >> (as the idea has seen plenty of discussion). It is not as obviously >> flawed as the if-and-while statement only variant, but it would still >> involve being rather persuasive to make such a significant change to >> the language. > > Hasn't the BDFL rejected a general embedded assignment in any case? > > As a final note, reading checking the archives for your ideas will > also give you an idea of what kinds of things are and aren't accepted > (i.e. - what's "Pythonic") as well as process - including the kinds of > questions you'll be asked here - that an idea goes through before > being accepted. > > <mike > -- > Mike Meyer <mwm at mired.org> http://www.mired.org/ > Independent Software developer/SCM consultant, email for more information. > > O< ascii ribbon campaign - stop html mail - www.asciiribbon.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From jeanpierreda at gmail.com Wed Jul 4 18:35:57 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jul 2012 12:35:57 -0400 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> Message-ID: <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com> On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote: > There's no point doing this in Python since scoping is function-level > and not lexical. Just move the assignment to the line preceding the if > statement. I'm not sure what distinction you want to draw here between "lexical scoping" and "function-level scoping". Is your issue that scopes cannot be inferred during the lexing phase? Or is it something else? Generally, AIUI, because lexical scoping is only really interesting in contrast to static scoping, any variable whose scope can be determined statically based on where it's defined is considered "lexically scoped". -- Devin From masklinn at masklinn.net Wed Jul 4 19:25:01 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 4 Jul 2012 19:25:01 +0200 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> Message-ID: <2CDC2D81-0A20-4B93-B8BA-7BAD8C5D5C21@masklinn.net> On 2012-07-04, at 18:19 , Matt Joiner wrote: > There's no point doing this in Python since scoping is function-level > and not lexical. Just move the assignment to the line preceding the if > statement. Technically, scoping is *both* lexical and function-level (as opposed to block-based), although writing to a lexical scope is slightly more complex than in other languages with mutable lexical scopes. From masklinn at masklinn.net Wed Jul 4 19:30:12 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 4 Jul 2012 19:30:12 +0200 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com> Message-ID: <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net> On 2012-07-04, at 18:35 , Devin Jeanpierre wrote: > On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote: >> There's no point doing this in Python since scoping is function-level >> and not lexical. Just move the assignment to the line preceding the if >> statement. > > I'm not sure what distinction you want to draw here between "lexical > scoping" and "function-level scoping". Is your issue that scopes > cannot be inferred during the lexing phase? Or is it something else? Guessing he misused "lexical scoping" for block-based scoping (as in e.g. C) or explicit scoping constructs (e.g. `let`-type constructs). > Generally, AIUI, because lexical scoping is only really interesting in > contrast to static scoping And I see you miswrite as well, "lexical scoping" and "static scoping" are the same thing, and opposed to "dynamic scoping" (where name resolution traverses the call stack). From techtonik at gmail.com Thu Jul 5 15:36:24 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Jul 2012 16:36:24 +0300 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <4FEF6B10.9040409@udel.edu> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> Message-ID: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> Before anything else I must apologize for significant lags in my replies. I can not read all of them to hold in my head, so I reply one by one as it goes trying not to miss a single point out there. It would be much easier to do this in unified interface for threaded discussions, but for now there is no capabilities for that neither in Mailman nor in GMail. And when it turns out that the amount of text is too big, and I spend a lot of time trying to squeeze it down and then it becomes pointless to send at all. Now back on the topic: On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy <tjreedy at udel.edu> wrote: > On 6/29/2012 4:32 PM, Georg Brandl wrote: >> >> On 26.06.2012 10:03, anatoly techtonik wrote: >>> >>> Now that Python 3 is all about iterators (which is a user killer >>> feature for Python according to StackOverflow - >>> http://stackoverflow.com/questions/tagged/python) would it be nice to >>> introduce more first class functions to work with them? One function >>> to be exact to split string into chunks. > > Nothing special about strings. It seemed so, but it just appeared that grouper recipe didn't work for me. >>> itertools.chunks(iterable, size, fill=None) > > This is a renaming of itertools.grouper in 9.1.2. Itertools Recipes. You > should have mentioned this. I think of 'blocks' rather than 'chunks', but I > notice several SO questions with 'chunk(s)' in the title. I guess `block` gives too low signal/noize ration in search results. That's why it probably also called chunks in other languages, where `block` stand for something else (I speak of Ruby blocks). >>> Which is the 33th most voted Python question on SO - >>> >>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464 > > I am curious how you get that number. I do note that there are about 15 > other Python SO questions that seem to be variations on the theme. There > might be more if 'blocks' and 'groups' were searched for. It's easy: 1. Go http://stackoverflow.com/ 2. Search [python] 3. Click `votes` tab 4. Choose `30 per page` at the bottom 5. Jump to the second page, there it is 4th from the top: http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30 As for duplicates - feel free to mark them as such. SO allows everybody to do this (unlike Roundup). >> Anatoly, so far there were no negative votes -- would you care to go >> another step and propose a patch? > > That is because Raymond H. is not reading either list right now ;-) > Hence the Cc:. Also because I did not yet respond to a vague, very > incomplete idea. > > From Raymond's first message on http://bugs.python.org/issue6021 , add > grouper: > > "This has been rejected before. I quite often see such arguments and I can't stand to repeat that these are not arguments. It is good to know, but when people use that as a reason to close tickets - that's just disgusting. To the Raymond's honor he cares to explain. > * It is not a fundamental itertool primitive. The recipes section in > the docs shows a clean, fast implementation derived from zip_longest(). What is the definition of 'fundamental primitive'? To me the fact that top answer for chunking strings on SO has 2+ times more votes than itertools versions is a clear 5 sigma indicator that something is wrong with this Standard model without chunks boson. > * There is some debate on a correct API for odd lengths. Some people > want an exception, some want fill-in values, some want truncation, and > some want a partially filled-in tuple. The alone is reason enough not > to set one behavior in stone. use case 3.1: odd lengths exception (CHOOSE ONE) 1. I see that no itertools function throws exceptions, check manually: len(iterable) / float(size) == len(iterable) // float(size) 2. Explicitly - itertools.chunks(iterable, size, fill=None) + itertools.chunks(iterable, size, fill=None, exception=False) use case 3.2. fill in value. it is here (SOLVED) use case 3.3: truncation no itertools support truncation, do manually chunks(iter, size)[:len(iter)//size) use case 4: partially filled-in tuple What should be there? >>> chunks('ABCDEFG', 3, 'x') >>> | More replies and workarounds to some of the raised points are below. > * There is an issue with having too many itertools. The module taken as > a whole becomes more difficult to use as new tools are added." There can be only two reasons to that: * chosen basis is bad (many functions that are rarely used or easily emulated) * basis is good, but insufficient, because iterators universe is more complicated than we think > This is not to say that the question should not be re-considered. Given the > StackOverflow experience in addition to that of the tracker and python-list > (and maybe python-ideas), a special exception might be made in relation to > points 1 and 3. --[offtopic about Python enhancements / proposals feedback]-- Yes, without SO I probably wouldn't trigger this at all. Because tracker doesn't help with raising importance - there are no votes, no feature proposals, no "stars". And what I "like" the most is that very "nice" resolution status - "committed/rejected" - which doesn't say anything at all. Python list? I try not to disrupt the frequency there. Python ideas? Too low participation level for gathering signals. There are many people that read, support, but don't want to reply (don't want to stand out or just lazy). There are many outside who don't want to be subscribed at all. There are 2000+ people spending time on Python conferences all over the world each year we see only a couple of reactions for every Python idea here. Quite often there are mistakes and omissions that would be nice to correct and you can't. So StackOverflow really helps here, but it is a Q&A tool, which is still much better than ML that are solely for chatting, brainstorming and all the crazy reading / writing stuff. They don't help to develop ideas collaboratively. Quite often I am just lost in amount of text to handle. --[/offtopic]-- > It regard to point 2: many 'proposals', including Anatoly's, neglect this > detail. But the function has to do *something* when seqlen % grouplen != 0. > So an 'idea' is not really a concrete programmable proposal until > 'something' is specified. > > Exception -- not possible for an itertool until the end of the iteration > (see below). To raise immediately for sequences, one could wrap grouper. > > def exactgrouper(sequence, k): # untested > if len(sequence) % k: > raise ValueError('Sequence length {} must be a multiple of group length > {}'.format(len(sequence), k) > else: > return itertools.grouper(sequence, k) Right. Iterator is not a sequence, because it doesn't know the length of its sequence. The method should not belong to itertools at all then. Python 3 is definitely become more complicated. I'd prefer to keep separated from iterator stuff, but it seems more harder with every iteration. > Of course, sequences can also be directly sequentially sliced (but should > the result be an iterable or sequence of blocks?). But we do not have a > seqtools module and I do not think there should be another method added to > the seq protocol. I'd expect strings chunked into strings and lists into lists. Don't want to know anything about protocols. > Fill -- grouper always does this, with a default of None. > > Truncate, Remainder -- grouper (zip_longest) cannot directly do this and no > recipes are given in the itertools docs. (More could be, see below.) > > Discussions on python-list gives various implementations either for > sequences or iterables. For the latter, one approach is "it = > iter(iterable)" followed by repeated islice of the first n items. Another is > to use a sentinal for the 'fill' to detect a final incomplete block (tuple > for grouper). > > def grouper_x(n, iterable): # untested > sentinal = object() > for g in grouper(n, iterable, sentinal): > if g[-1] != sentinal: > yield g > else: > # pass to truncate > # yield g[:g.index(sentinal) for remainer > # raise ValueError for delayed exception We need a simple function to split a sequence into chunks(). Now we face with the problem to apply that technique to a sequence of infinite length when a last element of infinite sequence is encountered. You might be thinking now that this is a reduction to absurdity. But I'd say it is an exit from the trap. Mathematically this problem can't be solved. I am not ignoring your solution - I think it's quite feasible, but isn't it an overcomplication? I mean 160 people out of 149 who upvoted the question are pretty happy with an answer that just outputs the last chunk as-is: http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python chunks('ABCDEFG', 3) --> 'ABC' 'DEF' 'G' And it is quite nice solution to me, because you're free to do anything you'd like if you expect you data to be odd: for chunk in chunks('ABCDEFG', size): if len(chunk) < size: raise Tail You can make a helper iterator out of it too. > --- > The above discussion of point 2 touches on point 4, which Raymond neglected > in the particular message above but which has come up before: What are the > allowed input and output types? An idea is not a programmable proposal until > the domain, range, and mapping are specified. Domain? Mapping? I am not ignoring existing knowledge and experience. I just don't want to complicate and don't see appropriate `import usecase` in current context, so I won't try to guess what this means. in string -> out list of strings in list -> out list of lists > Possible inputs are a specific sequence (string, for instance), any > sequence, any iterable. Possible outputs are a sequence or iterator of > sequence or iterator. The various python-list and stackoverflow posts > questions asks for various combinations. zip_longest and hence grouper takes > any iterable and returns an iterator of tuples. (An iterator of maps might > be more useful as a building block.) This is not what one usually wants with > string input, for instance, nor with range input. To illustrate: Allright. Got it. Sequences have a length and can be sliced with [i:j], iterator can't be sliced (and hence no chunks can be made). So this function doesn't belong to itertools - it is a missing string or sequence method. We can't have a chunk with an iterator, because iterator over a string decomposes it into a group of pieces with no reverse function. We can have a group and then join the group into something. But this requires the knowledge of appropriate join() function for the iterator, and probably not efficient. As there are no such function (must be that Mapping you referenced above) - the recomposition into chunks is impossible. > import itertools as it > > def grouper(n, iterable, fillvalue=None): > "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx" > args = [iter(iterable)] * n > return it.zip_longest(*args, fillvalue=fillvalue) > > print(*(grouper(3, 'ABCDEFG', 'x'))) # probably not wanted > print(*(''.join(g) for g in grouper(3, 'ABCDEFG', 'x'))) > # > ('A', 'B', 'C') ('D', 'E', 'F') ('G', 'x', 'x') > ABC DEF Gxx > > -- > What to do? One could easily write 20 different functions. So more thought > is needed before adding anything. -1 on the idea as is. I've learned a new English type of argument - "straw man" (I used to call this "hijacking"). This -1 doesn't belong to original idea. It belongs to proposal of itertools.chunks() with a long list of above points and completely different user stories (i.e. not "split string into chunks"). I hope you still +1 with 160 people on SO that think Python needs an easy way to chunk sequences. > For the doc, I think it would be helpful here and in most module subchapters > if there were a subchapter table of contents at the top (under 9.1 in this > case). Even though just 2 lines here (currently, but see below), it would > let people know that there *is* a recipes section. After the appropriate > tables, mention that there are example uses in the recipe section. Possibly > add similar tables in the recipe section. Unfortunately, it appeared that grouper() is not chunks(). It doesn't delivers list of list of chars given string as an input instead of list of chunks. > Another addition could be a new subsection on grouping (chunking) that would > discuss post-processing of grouper (as discussed above), as well as other > recipes, including ones specific to strings and sequences. It would > essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or > Chunking Sequences and Iterables". The synonyms will help external > searching. A toc would let people who have found this doc know to look for > this at the bottom. This makes matters pretty ugly. In ideal language there should be less docs, not more. From techtonik at gmail.com Thu Jul 5 15:47:53 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Jul 2012 16:47:53 +0300 Subject: [Python-ideas] [Python-Dev] itertools.chunks(iterable, size, fill=None) In-Reply-To: <jspe70$4gq$1@dough.gmane.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <jsoo4p$via$1@dough.gmane.org> <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com> <jspe70$4gq$1@dough.gmane.org> Message-ID: <CAPkN8xJrKgAUaLw3siTpzAjUB=eg1S08oWu-Z=7ThHqgU0eM+w@mail.gmail.com> On Sun, Jul 1, 2012 at 3:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > > To address the main problem of users not finding what they need, what about > simply extending the docstring of the grouper() function with a sentence > like this: > > "This functionality is also called 'chunking' or 'blocking' and can be used > for load distribution and sharding." > > That would make it easy for users to find what they are looking for when > they search the page for "chunk". I find that a much more common and less > ambiguous name than "grouping", which reminds me more of "group by". In appeared that "chunking" and "grouping" are different kind of tasks. You can chunk a sequence (sting) by slicing it into smaller sequences, but you can not chunk in iterable - you can only group it. There is an data loss about the structure that occurs when a sequence (string) becomes an iterator: chunks ABCDE -> AB CD E group ABCDE -> A B C D E -> (A B) (C D) (D E) From techtonik at gmail.com Thu Jul 5 16:33:19 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Jul 2012 17:33:19 +0300 Subject: [Python-ideas] [Python-Dev] itertools.chunks(iterable, size, fill=None) In-Reply-To: <jt225u$iih$1@dough.gmane.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com> <jt225u$iih$1@dough.gmane.org> Message-ID: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com> On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 7/4/2012 5:57 AM, anatoly techtonik wrote: >> >> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote: > > >>> Anatoly, so far there were no negative votes -- would you care to go >>> another step and propose a patch? >> >> >> Was about to say "no problem", > > > Did you read that there *are* strong negative votes? And that this idea has > been rejected before? I summarized the objections in my two responses and > pointed to the tracker issues. One of the objections is that there are 4 > different things one might want if the sequence length is not an even > multiple of the chunk size. Your original 'idea' did not specify. I actually meant that there is a problem to propose a patch in the sense of getting checkout, working on a diff, sending it by attaching to bug tracker as developer guide says. >> For now the best thing I can do (I don't risk even to mention anything >> with 3.3) is to copy/paste code from the docs here: >> >> from itertools import izip_longest >> def chunks(iterable, size, fill=None): >> """Split an iterable into blocks of fixed-length""" >> # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx >> args = [iter(iterable)] * size >> return izip_longest(fillvalue=fill, *args) > > > Python ideas is about Python 3 ideas. Please post Python 3 code. > > This is actually a one liner > > return zip_longest(*[iter(iterable)]*size, fillvalue=file) > > We don't generally add such to the stdlib. Can you figure out from the code what this stuff does? It doesn't give chunks of strings. >> BTW, this doesn't work as expected (at least for strings). Expected is: >> chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx' >> got: >> chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x') > > > One of the problems with idea of 'add a chunker' is that there are at least > a dozen variants that different people want. That's not the problem. People always want something extra. The problem that we don't have a real wish distribution. If 1000 people want chunks and 1 wants groups with exception - we still account these as equal variants. Therefore my idea is deliberately limited to "string to chunks" user story, and SO implementation proposal. > I discussed the problem of > return types issue in my responses. I showed how to get the 'expected' > response above using grouper, but also suggested that it is the wrong basis > for splitting strings. Repeated slicing make more sense for concrete > sequence types. > > def seqchunk_odd(s, size): > # include odd size left over > for i in range(0, len(s), size): > yield s[i:i+size] > > print(list(seqchunk_odd('ABCDEFG', 3))) > # > ['ABC', 'DEF', 'G'] Right. That's the top answer on SO that people think should be in stdlib. Great we are talking about the same thing actually. > def seqchunk_even(s, size): > # only include even chunks > for i in range(0, size*(len(s)//size), size): > yield s[i:i+size] > > print(list(seqchunk_even('ABCDEFG', 3))) > # > ['ABC', 'DEF'] This is deducible from seqchunk_odd(s, size) > def strchunk_fill(s, size, fill): > # fill odd chunks > q, r = divmod(len(s), size) > even = size * q > for i in range(0, even, size): > yield s[i:i+size] > if size != even: > yield s[even:] + fill * (size - r) > > print(list(strchunk_fill('ABCDEFG', 3, 'x'))) > # > ['ABC', 'DEF', 'Gxx'] Also deducible from seqchunk_odd(s, size) > Because the 'fill' value is necessarily a sequence for strings, > strchunk_fill would only work for lists and tuples if the fill value were > either required to be given as a tuple or list of length 1 or if it were > internally converted inside the function. Skipping that for now. > > Having written the fill version based on the even version, it is easy to > select among the three behaviors by modifying the fill version. > > def strchunk(s, size, fill=NotImplemented): > # fill odd chunks > q, r = divmod(len(s), size) > even = size * q > for i in range(0, even, size): > yield s[i:i+size] > if size != even and fill is not NotImplemented: > yield s[even:] + fill * (size - r) > > print(*strchunk('ABCDEFG', 3)) > print(*strchunk('ABCDEFG', 3, '')) > print(*strchunk('ABCDEFG', 3, 'x')) > # > ABC DEF > ABC DEF G > ABC DEF Gxx I now don't even think that fill value is needed as argument. if len(chunk) < size: chunk.extend( [fill] * ( size - len(chunk)) ) > I already described how something similar could be done by checking each > grouper output tuple for a fill value, but that requires that the fill value > be a sentinal that could not otherwise appear in the tuple. One could modify > grouper to fill with a private object() and check the last item of each > group for that sentinal and act accordingly (delete, truncate, or replace). > A generic api needs some thought, though. I just need to chunk strings and sequences. Generic API is too complex without counting all usecases and iterating over them. > An issue I did not previously mention is that people sometimes want > overlapping chunks rather than contiguous disjoint chunks. The slice > approach trivially adapts to that. > > def seqlap(s, size): > for i in range(len(s)-size+1): > yield s[i:i+size] > > print(*seqlap('ABCDEFG', 3)) > # > ABC BCD CDE DEF EFG > > A sliding window for a generic iterable requires a deque or ring buffer > approach that is quite different from the zip-longest -- grouper approach. That's why I'd like to drastically reduce the scope of proposal. itertools doesn't seem to be the best place anymore. How about sequence method? string.chunks(size) -> ABC DEF G list.chunks(size) -> [A,B,C], [C,D,E],[G] If somebody needs a keyword argument - this can come later without breaking compatibility. From steve at pearwood.info Thu Jul 5 17:57:17 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 Jul 2012 01:57:17 +1000 Subject: [Python-ideas] [Python-Dev] itertools.chunks(iterable, size, fill=None) In-Reply-To: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com> <jt225u$iih$1@dough.gmane.org> <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com> Message-ID: <4FF5B95D.3060208@pearwood.info> anatoly techtonik wrote: > On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote: >> A sliding window for a generic iterable requires a deque or ring buffer >> approach that is quite different from the zip-longest -- grouper approach. > > That's why I'd like to drastically reduce the scope of proposal. > itertools doesn't seem to be the best place anymore. How about > sequence method? > > string.chunks(size) -> ABC DEF G > list.chunks(size) -> [A,B,C], [C,D,E],[G] -1 This is a fairly trivial problem to solve, and there are many variations on it. Many people will not find the default behaviour helpful, and will need to write their own. Why complicate the API for all sequence types with this? I don't believe that we should enshrine one variation as a built-in method, without any evidence that it is the most useful or common variation. Even if there is one variation far more useful than the others, that doesn't necessarily mean we ought to make it a builtin method unless it is a fundamental sequence operation, has wide applicability, and is genuinely hard to write. I don't believe chunking meets *any* of those criteria, let alone all three. Not every six line function needs to be a builtin. I believe that splitting a sequence (or a string) into fixed-size chunks is more of a programming exercise problem than a genuinely useful tool. That does not mean that there is never any real use-cases for splitting into fixed-size chunks, only that this is the function that *seems* more useful in theory than it turns out in practice. Compare this with more useful sequence/iteration tools, like (say) zip. You can hardly write a hundred lines of code without using zip at least once. But I bet you can write tens of thousands of lines of code without needing to split sequences into fixed chunks like this. Besides, the name "chunks" is more general than how you are using it. For example, I consider chunking to be splitting a sequence up at a various delimiters or separators, not at fixed character positions. E.g. "the third word of item two of the fourth line" is a chunk. This fits more with the non-programming use of the term chunk or chunking, and has precedence in Apple's Hypertalk language, which literally allowed you to talk about words, items and lines of text, each of which are described as chunks. This might be a good candidate for a utility module made up of assorted useful functions, but not for the string and sequence APIs. -- Steven From steve at pearwood.info Thu Jul 5 18:09:44 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 06 Jul 2012 02:09:44 +1000 Subject: [Python-ideas] [Python-Dev] itertools.chunks(iterable, size, fill=None) In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> Message-ID: <4FF5BC48.9080302@pearwood.info> anatoly techtonik wrote: >>>> Which is the 33th most voted Python question on SO - >>>> >>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464 >> I am curious how you get that number. I do note that there are about 15 >> other Python SO questions that seem to be variations on the theme. There >> might be more if 'blocks' and 'groups' were searched for. > > It's easy: > 1. Go http://stackoverflow.com/ > 2. Search [python] > 3. Click `votes` tab > 4. Choose `30 per page` at the bottom > 5. Jump to the second page, there it is 4th from the top: > http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30 Yes. I don't think this is particularly significant. Have a look at some of the questions with roughly the same number of votes: #26 "How can I remove (chomp) a newline in Python?" 176 votes #33 "How do you split a list into evenly sized chunks in Python?" 149 votes #36 "Accessing the index in Python for loops" 144 votes Being 33rd most voted question doesn't really mean much. By the way, why is this discussion going to both python-dev and python-ideas? -- Steven From techtonik at gmail.com Thu Jul 5 21:41:29 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Jul 2012 22:41:29 +0300 Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev] itertools.chunks(iterable, size, fill=None)) Message-ID: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > anatoly techtonik, 05.07.2012 15:36: >> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote: >>> From Raymond's first message on http://bugs.python.org/issue6021 , add >>> grouper: >>> >>> "This has been rejected before. >> >> I quite often see such arguments and I can't stand to repeat that >> these are not arguments. It is good to know, but when people use that >> as a reason to close tickets - that's just disgusting. > > The *real* problem is that people keep bringing up topics (and even spell > them out in the bug tracker) without searching for existing discussions > and/or tickets first. That's why those who do such a search (or who know > what they are talking about anyway) close these tickets with the remark > "this has been rejected before", instead of repeating an entire heap of > arguments all over again to feed a discussion that would only lead to the > same result as it did before, often several times before. Make the bloody FAQ and summarize this stuff? Why waste each others time? If people don't enjoy repeating themselves over and over - there is a bloody wiki. What should happen to people to start extracting gems of knowledge from piles of dusty sheets called list "archives" for others to admire. No, it is easier to say "it was already discussed many times", "why don't you Google yourself", "so far you're only complaining", etc. If people can't find anything - why everybody thinks they are ignorant and lazy. Even if it so, why nobody thinks that maybe that bloody Xapian index is dead again for a bloody amount of moons nobody knows why and how many exactly? Why nobody thinks that lazy coders can also help with development? Maybe that laziness is the primary reason some major groups actually prefer Python to Java, C++ and other more interesting languages (such as PHP) when it comes to typing? Make it easy and the patches will follow. Answers like "this was discussed before" don't make it easy to understand, and leaving users rereading old 19xx archives that people don't reread themselves will likely make users bounce and never (NEVER!) come up with some proposal again. An "organic" way to keep traffic low. Miscommunication is a bad experience for users, bad experience for developers, everybody is annoyed and as a result such nice language as Python loses points on TIOBE (and convenient chunk() functions to munch-munch on the sequence data). Wheew. :-F From techtonik at gmail.com Thu Jul 5 21:55:26 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Jul 2012 22:55:26 +0300 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping Message-ID: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> This one is practical. I am looking at NaCl SDK download page: https://developers.google.com/native-client/sdk/download "you need Python installed", "download SDK update utility" What makes me sad that update utility is a Python script in a zip file - nacl_sdk.zip which includes shell script and a .bat file for launching this Python script. This makes me kind of sad. You have Python installed. Why can't you just crossplatformly do: mkdir nacl cd nacl python -m urllib get http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py python update_sdk.py From amauryfa at gmail.com Thu Jul 5 22:24:27 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2012 22:24:27 +0200 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> Message-ID: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> 2012/7/5 anatoly techtonik <techtonik at gmail.com>: > This makes me kind of sad. You have Python installed. Why can't you > just crossplatformly do: > > mkdir nacl > cd nacl > python -m urllib get > http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py > python update_sdk.py I'm sure there is already a way with standard python tools. Something along these lines: python -c "from urllib.request import urlretrieve; urlretrieve('URL', 'update_sdk.zip')" python -m update_sdk.zip The second command will work if the zip file has a __main__.py. Do you think we need other tools? -- Amaury Forgeot d'Arc From simon.sapin at kozea.fr Thu Jul 5 22:15:10 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Thu, 05 Jul 2012 22:15:10 +0200 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> Message-ID: <4FF5F5CE.8020701@kozea.fr> Le 05/07/2012 21:55, anatoly techtonik a ?crit : > This one is practical. I am looking at NaCl SDK download page: > https://developers.google.com/native-client/sdk/download > > "you need Python installed", "download SDK update utility" > > What makes me sad that update utility is a Python script in a zip file > - nacl_sdk.zip > which includes shell script and a .bat file for launching this Python script. > > This makes me kind of sad. You have Python installed. Why can't you > just crossplatformly do: > > mkdir nacl > cd nacl > python -m urllib get > http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py > python update_sdk.py Hi, "Sadness" drama aside, what?s your point? Is this a proposal to add a command-line API for urlretrieve()? In that case, +1. (For what my vote is worth.) The use cases are limited (just use the Python API) but they do exist, as you showed. Regards, -- Simon Sapin From ned at nedbatchelder.com Thu Jul 5 23:07:14 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 05 Jul 2012 17:07:14 -0400 Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev] itertools.chunks(iterable, size, fill=None)) In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> Message-ID: <4FF60202.1040909@nedbatchelder.com> On 7/5/2012 3:41 PM, anatoly techtonik wrote: > Make the bloody FAQ and summarize this stuff? Why waste each others > time? If people don't enjoy repeating themselves over and over - there > is a bloody wiki. What should happen to people to start extracting > gems of knowledge from piles of dusty sheets called list "archives" > for others to admire. > > No, it is easier to say "it was already discussed many times", "why > don't you Google yourself", "so far you're only complaining", etc. If > people can't find anything - why everybody thinks they are ignorant > and lazy. Even if it so, why nobody thinks that maybe that bloody > Xapian index is dead again for a bloody amount of moons nobody knows > why and how many exactly? Why nobody thinks that lazy coders can also > help with development? Maybe that laziness is the primary reason some > major groups actually prefer Python to Java, C++ and other more > interesting languages (such as PHP) when it comes to typing? Make it > easy and the patches will follow. Answers like "this was discussed > before" don't make it easy to understand, and leaving users rereading > old 19xx archives that people don't reread themselves will likely make > users bounce and never (NEVER!) come up with some proposal again. An > "organic" way to keep traffic low. > > Miscommunication is a bad experience for users, bad experience for > developers, everybody is annoyed and as a result such nice language as > Python loses points on TIOBE (and convenient chunk() functions to > munch-munch on the sequence data). Anatoly, miscommunication is a bad experience, yes. This is an excellent example of it. I honestly have no idea what you are advocating in this message, partly because sarcasm doesn't work well. You seem really upset, but I for one don't know what it is you want. --Ned. > Wheew. :-F > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From wuwei23 at gmail.com Fri Jul 6 07:05:04 2012 From: wuwei23 at gmail.com (alex23) Date: Thu, 5 Jul 2012 22:05:04 -0700 (PDT) Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev] itertools.chunks(iterable, size, fill=None)) In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> Message-ID: <05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com> On Jul 6, 5:41?am, anatoly techtonik <techto... at gmail.com> wrote: > Why nobody thinks that lazy coders can also > help with development? Maybe that laziness is the primary reason some > major groups actually prefer Python to Java, C++ and other more > interesting languages (such as PHP) when it comes to typing? You make this criticism fairly regularly and yet you yourself constantly make suggestions and refuse to provide implementations. I have a hard time reconciling the implicit anger in this statement: > Why waste each others time? If people don't enjoy repeating themselves over > and over - there is a bloody wiki. ...with ones you have previously made defending your _own_ unwillingness to examine pre-existing discussion: > It's too boring to live in a world of existing knowledge and expertise You yourself feel you have no responsibility in understanding a situation before demanding that it be raised for discussion. I'm not sure _what_ others could put into place that would _make_ you perform this action. > Answers like "this was discussed > before" don't make it easy to understand, and leaving users rereading > old 19xx archives that people don't reread themselves will likely make > users bounce and never (NEVER!) come up with some proposal again. Once again you're putting the burden on _others_ to somehow come up with mechanisms to force people to do the due diligence they should be capable of doing as a programmer. Frankly, I'm of the opinion that if someone cannot be bothered putting the effort into understanding the pre-existing content for a proposal they're making, then yes, they simply should not come up with that proposal. I'm not entirely sure why you feel people should be entitled to request far more than they're willing to contribute. From wuwei23 at gmail.com Fri Jul 6 07:08:09 2012 From: wuwei23 at gmail.com (alex23) Date: Thu, 5 Jul 2012 22:08:09 -0700 (PDT) Subject: [Python-ideas] Bloody FAQ (Was: [Python-Dev] itertools.chunks(iterable, size, fill=None)) In-Reply-To: <05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com> References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com> <05347f1b-17b9-48e5-a99b-c744b4c8e258@nw7g2000pbb.googlegroups.com> Message-ID: <55df23be-bfb5-40f7-aa51-6b7e89c129e3@m2g2000pbv.googlegroups.com> On Jul 6, 3:05?pm, alex23 <wuwe... at gmail.com> wrote: > the pre-existing content That should actually be "context". From stephen at xemacs.org Fri Jul 6 07:26:57 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 06 Jul 2012 14:26:57 +0900 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> Message-ID: <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> Annoying cross-post trimmed. anatoly techtonik writes: > Before anything else I must apologize for significant lags in my > replies. No, that's nothing you need to apologize for. Taking time to formulate a decent reply is common courtesy, and commendable. We're really *not* in a hurry here; Python is going to be around for a few more decades at least. What you need to apologize for is the major faux pas of cross-posting. Please cut it out, and in particular trim your own address list when replying, or use Reply-To (and maybe Mail-Followup-To) to redirect others' replies to an appropriate list. IMO, the right list for this discussion is python-ideas, but at the very least choose one. and again he writes: > Make the bloody FAQ and summarize this stuff? Why waste each others > time? If people don't enjoy repeating themselves over and over - > there is a bloody wiki. What should happen to people to start > extracting gems of knowledge from piles of dusty sheets called list > "archives" for others to admire. This is a terrible idea for python-dev and python-ideas. While it is frustrating to get a "been there, done that, rejected with extreme prejudice" reply, and there's no question that searching the archives is a hit-and-mostly-miss kind of thing because of the difficulty of choosing good search terms, it's really not that costly to come back with "I'm sorry, I couldn't find the thread". Rather than spend effort on writing a FAQ that would rather quickly turn into a monstrosity hardly more easy to search than the archives themselves, and almost never be read, we should devote any effort to improving the capability for searching archives (and the wiki and the issue tracker). The problem with trying to put everything into the FAQ is that it's a terribly unrewarding thing to do. Most of the stuff you will write will be ignored and profit nobody. It really needs to be selected, by somebody with taste and knowledge of user needs. Presumably O'Reilly or somebody has a "Python Hacks" book -- go buy it, which will encourage the author to keep up the good work. If they don't, why don't you propose it to them, and write it yourself? Then you can measure how good an idea it is by your royalties. Or if you don't feel like writing it yourself, maybe you can convince O'Reilly to come up with a big enough advance to interest somebody like Terry Reedy or Steven d'Aprano or even Raymond Hettinger or David Beazley. Regards, From mwm at mired.org Fri Jul 6 08:00:30 2012 From: mwm at mired.org (Mike Meyer) Date: Fri, 06 Jul 2012 02:00:30 -0400 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com> "Stephen J. Turnbull" <stephen at xemacs.org> wrote: >anatoly techtonik writes: > > Make the bloody FAQ and summarize this stuff? Why waste each others > > time? If people don't enjoy repeating themselves over and over - > > there is a bloody wiki. What should happen to people to start > > extracting gems of knowledge from piles of dusty sheets called list > > "archives" for others to admire. > >This is a terrible idea for python-dev and python-ideas. While it is >frustrating to get a "been there, done that, rejected with extreme >prejudice" reply, and there's no question that searching the archives >is a hit-and-mostly-miss kind of thing because of the difficulty of >choosing good search terms, it's really not that costly to come back >with "I'm sorry, I couldn't find the thread". Rather than spend >effort on writing a FAQ that would rather quickly turn into a >monstrosity hardly more easy to search than the archives themselves, >and almost never be read, we should devote any effort to improving the >capability for searching archives (and the wiki and the issue >tracker). Python-dev i'm not familiar enough with to formulate an opinion, but I think it could work well for python-ideas. While better search tools always help, a mail list is mostly unorganized, and a wiki need a purpose (possibly provided by a person or small group of people who take on the responsibility of maintaining the thing), or it becomes an even less organized mess. A FAQ, on the other hand, *has* a purpose. It also has a natural source of new material in the list. Python.org already has multiple FAQs with an integrated search engine (that are hopefully integrated into the python.org search engine), and last time I looked (admitadly long ago), a tool for maintaining FAQs. While I don't think I'm quaified to write answer for python-ideas faqs, I believe I can decide whether or not something is appropriate as a python-ideas FAQ entry. If the tool for maintinaing them isn't a memory error and is still in use, I'd certainly be willing to do that. But that's the easy job. The trick is getting the list members to play along. If they provide what is most people consider the definitive answer a couple of times, then the next time the question comes up, write it up as a FAQ entry and submit it as such as well to the list. The time after that, just provide a pointer to the FAQ entry. This worked very well for the FreeBSD FAQ, which is a fairly large document. They didn't even have a spiffy tool for maintaining it, but had to take submissions and add or fix markup and then check them in to the document tree. -- Sent from my Android tablet. Please excuse my swyping. From stephen at xemacs.org Fri Jul 6 11:22:03 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 06 Jul 2012 18:22:03 +0900 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com> Message-ID: <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp> Mike Meyer writes: > Python-dev i'm not familiar enough with to formulate an opinion, > but I think it could work well for python-ideas. You're missing the point. Improving the FAQ for python-*list* is a great idea, but Mr. techtonik proposes a FAQ for python-*ideas*. IOW, if the question is actually on-topic, the answers are reasons why something isn't going to ever be in Python. True, often the explanation of why a proposed feature is inappropriate is of the form "this three-line function does the job", but if the question is really a FAQ, it presumably has been asked multiple times on python-list, and should be picked up there. > This worked very well for the FreeBSD FAQ, which is a fairly large > document. No, it didn't. The FreeBSD FAQ is quite obviously not oriented to telling developers what isn't going to go into the next version of FreeBSD and why not. If people want to move ahead with this discussion, it really ought to move to python-list IMO. From mwm at mired.org Fri Jul 6 17:38:42 2012 From: mwm at mired.org (Mike Meyer) Date: Fri, 6 Jul 2012 11:38:42 -0400 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com> <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20120706113842.28e7a3ae@bhuda.mired.org> On Fri, 06 Jul 2012 18:22:03 +0900 "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > Mike Meyer writes: > > Python-dev i'm not familiar enough with to formulate an opinion, > > but I think it could work well for python-ideas. > You're missing the point. Improving the FAQ for python-*list* is a > great idea, but Mr. techtonik proposes a FAQ for python-*ideas*. IOW, > if the question is actually on-topic, the answers are reasons why > something isn't going to ever be in Python. That point I got. > True, often the explanation of why a proposed feature is inappropriate > is of the form "this three-line function does the job", but if the > question is really a FAQ, it presumably has been asked multiple times > on python-list, and should be picked up there. That I didn't, and it's fair enough. But the onus to fix this *still* falls on the people who are answering the same question over and over. They need to start submitting the answers to the FAQ (there is a history & design FAQ that would seem to be an appropriate place for them) rather than just to the list. > > This worked very well for the FreeBSD FAQ, which is a fairly large > > document. > No, it didn't. The FreeBSD FAQ is quite obviously not oriented to > telling developers what isn't going to go into the next version of > FreeBSD and why not. If you're claiming that the FreeBSD FAQ technique didn't work well for it, then I disagree. That the FreeBSD FAQ has a different orientation is true, but not clearly relevant. > If people want to move ahead with this discussion, it really ought to > move to python-list IMO. Not in mine. The problem is repeated questions on -ideas. My proposed solution has to be implemented by -ideas readers. They need to start submitting frequently typed answers to the appropriate FAQ, not just to the list. <mike -- Mike Meyer <mwm at mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From jae+python at jaerhard.com Fri Jul 6 20:14:04 2012 From: jae+python at jaerhard.com (=?iso-8859-1?B?SvxyZ2VuIEEu?= Erhard) Date: Fri, 6 Jul 2012 20:14:04 +0200 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? Message-ID: <20120706181404.GB3347@jaerhard.com> I just now ran into (since learning about the recipes) having use for a recipe (then I found out that I can't use it anyway, but that's a different thing) So... I consider it very strange and annoying that there's this code. Ready to use. But you have to copy-and-paste it into your code *somewhere* instead of easily importing it. As to the "it makes using Python harder to learn", I beg to differ: an addition to the *language* (like, say, metaclasses) *can* make it more complicated. But additions to the stdlib? What about "batteries included"? Not a motto anymore (I heard rumors, so maybe that's the case) Putting these in some official(!) iterutils (or name it what you want) package would be a solution for so many people. Yes, not everyone needs grouper's current functionality. But for the many who do, it'd be there already. And someone thought those recipes are useful, didn't you? So why not make them more easily available? And then there's Alex Martelli at http://stackoverflow.com/a/1641242/238406, and I agree with him wholeheartedly. Bye, J From stephen at xemacs.org Fri Jul 6 20:53:31 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 07 Jul 2012 03:53:31 +0900 Subject: [Python-ideas] itertools.chunks(iterable, size, fill=None) In-Reply-To: <20120706113842.28e7a3ae@bhuda.mired.org> References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com> <jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu> <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com> <87fw95793y.fsf@uwakimon.sk.tsukuba.ac.jp> <1a7c74ab-084e-4c87-ade1-27d8c7f0b788@email.android.com> <8762a16y84.fsf@uwakimon.sk.tsukuba.ac.jp> <20120706113842.28e7a3ae@bhuda.mired.org> Message-ID: <87txxk67ro.fsf@uwakimon.sk.tsukuba.ac.jp> Mike Meyer writes: > That I didn't, and it's fair enough. But the onus to fix this *still* > falls on the people who are answering the same question over and > over. They need to start submitting the answers to the FAQ (there is a > history & design FAQ that would seem to be an appropriate place for > them) rather than just to the list. I disagree that it's appropriate. The History & Design FAQ isn't intended to be encyclopedic, and I don't see that making it so would be useful. The point is that if "This has been rejected before" is sufficient answer, the proponent withdraws, everybody happy. And if not, they have to go to the archives anyway. They need to answer the defects brought up in the list discussion, point by point, or they're not going to get a hearing. > If you're claiming that the FreeBSD FAQ technique didn't work well > for it, then I disagree. No, I'm claiming that AFAICS the FreeBSD FAQ didn't even try to fix the problem that you and anatoly percieve. > The problem is repeated questions on -ideas. Indeed!<wink/> But I think the solution is to encourage asking those questions on python-list first.[1] The experts there are both willing to answer such questions, and have a better sense of FAQ-iness (relatively few users interested in the question will propose a general solution). And they have an incentive to document answers if they notice a FAQ. If no suitable idiom appears, then work up a change proposal and post to -ideas. Footnotes: [1] Yes, I'm being mildly ironic; people typically won't know it's a FAQ until they ask. Nevertheless, asking on python-list before posting to either python-ideas or python-dev should be encouraged ("unless you're Dutch"<wink/>). From g.brandl at gmx.net Fri Jul 6 21:30:41 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 06 Jul 2012 21:30:41 +0200 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> Message-ID: <jt7ebl$duo$2@dough.gmane.org> On 05.07.2012 22:24, Amaury Forgeot d'Arc wrote: > 2012/7/5 anatoly techtonik <techtonik at gmail.com>: >> This makes me kind of sad. You have Python installed. Why can't you >> just crossplatformly do: >> >> mkdir nacl >> cd nacl >> python -m urllib get >> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py >> python update_sdk.py > > I'm sure there is already a way with standard python tools. Something > along these lines: > > python -c "from urllib.request import urlretrieve; urlretrieve('URL', > 'update_sdk.zip')" > python -m update_sdk.zip > > The second command will work if the zip file has a __main__.py. > Do you think we need other tools? The "python -m urllib" (don't think "get" is required) interface certainly looks nice and is similar in style with many of the other __main__ stuff we add to stdlib modules. Georg From anacrolix at gmail.com Sat Jul 7 18:03:15 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Sun, 8 Jul 2012 00:03:15 +0800 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com> <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net> Message-ID: <CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com> Sorry yes I meant block scoping. I assimilated terms from further up in the thread I think. I mean that there can be no temporary/block/localized scope for a name to within the if statement, so there's no reason not to put it on the line above. FWIW I've grown fond of function level scoping, it really reduces confusion in closures and encourages smaller functions to avoid name leaks. Go has something like the OP asks for: if x := SomeFunc(); x > 3 { // x scoped only within this block } Haskell also lets you name expressions like: someFunc y at x:xs = ... But in Python I still think this is best: x = some_func() if x > 3: # no scoping :) On Thu, Jul 5, 2012 at 1:30 AM, Masklinn <masklinn at masklinn.net> wrote: > On 2012-07-04, at 18:35 , Devin Jeanpierre wrote: > >> On Wed, Jul 4, 2012 at 12:19 PM, Matt Joiner <anacrolix at gmail.com> wrote: >>> There's no point doing this in Python since scoping is function-level >>> and not lexical. Just move the assignment to the line preceding the if >>> statement. >> >> I'm not sure what distinction you want to draw here between "lexical >> scoping" and "function-level scoping". Is your issue that scopes >> cannot be inferred during the lexing phase? Or is it something else? > > Guessing he misused "lexical scoping" for block-based scoping (as in > e.g. C) or explicit scoping constructs (e.g. `let`-type constructs). > >> Generally, AIUI, because lexical scoping is only really interesting in >> contrast to static scoping > > And I see you miswrite as well, "lexical scoping" and "static scoping" > are the same thing, and opposed to "dynamic scoping" (where name > resolution traverses the call stack). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From masklinn at masklinn.net Sat Jul 7 18:15:35 2012 From: masklinn at masklinn.net (Masklinn) Date: Sat, 7 Jul 2012 18:15:35 +0200 Subject: [Python-ideas] the optional "as" statement inside "if" statements In-Reply-To: <CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com> References: <CAEjd=rqxgSPeRoSs7=+Wfo3OzYJNdYcpMJLV3W2xMSNHXqZe4g@mail.gmail.com> <CADiSq7f-PXV9tUmSCC0ycnbvWb_hV1fd68VNSa=wtWd-9F3QMg@mail.gmail.com> <20120630204852.4b898a0a@bhuda.mired.org> <CAB4yi1P7N4-yo5fLaSQaQ0-7wWtQ263ydMg-BECGAcQH7674Rg@mail.gmail.com> <CABicbJL-J4Dk6-Wm1cCpM2YnUUWA-g=JqfgH7TennNmDqtKrhA@mail.gmail.com> <09066A71-B626-4A35-8366-7FD3C9ED415C@masklinn.net> <CAB4yi1MEny88R_ajmqbsj7aXnX_mUH+MpF72F3AZUMB8K1Z-hg@mail.gmail.com> Message-ID: <24E42BCE-2875-4DFB-9E60-1B8A1CF9A7AA@masklinn.net> On 2012-07-07, at 18:03 , Matt Joiner wrote: > > Haskell also lets you name expressions like: > > someFunc y at x:xs = ... as-patterns have very little relation with what was described in the thread, as they're about keeping the "packed" version of a pattern-matched expression. Guards are probably closer: case someFunc val of x | x > 3 -> ? From christopherreay at gmail.com Sat Jul 7 20:07:02 2012 From: christopherreay at gmail.com (Christopher Reay) Date: Sat, 7 Jul 2012 19:07:02 +0100 Subject: [Python-ideas] Explanation on how to search the archives Message-ID: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> I have heard two suggestions on how to search the achives one is: If you have time to post and discuss an idea, you have time to search gmane.comp.python.ideas at search.gmane.org and perhaps find out why not to bother posting. And the Other: It's more a matter of working out how to point Google (or the search engine of your choice) at the archives in a useful way. In this case: https://www.google.com/search?q=inurl%3Apython-ideas%20site%3Amail.python.org%20embedded%20assignment This is great information. Is there a page suggesting how to search archives effectively? Whilst much of that skill is generic, and clear concise explanation of what is expected from the polite user of the mailing list (and other Python mailing lists) could help. A link to that page could be put in the footer of the python mailing list emails.. and voila, a very easy thing to reference and get righteous about people not having read. What do you think? Christopher -- Be prepared to have your predictions come true -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120707/2bc25a93/attachment.html> From steve at pearwood.info Sat Jul 7 21:01:59 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 Jul 2012 05:01:59 +1000 Subject: [Python-ideas] Explanation on how to search the archives In-Reply-To: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> Message-ID: <4FF887A7.5070101@pearwood.info> Christopher Reay wrote: > Is there a page suggesting how to search archives effectively? [...] > What do you think? I think that is an excellent idea for some person or persons to write up on their blogs or other websites. There's no need to wait for permission, or for The One True Official Page -- anyone can write up a page and link to it. There is even a python.org wiki, sadly under-utilized in my opinion. Anyone with an account who wants to start a page there can. Not every concept vaguely related to programming in Python falls under the responsibility of python.org. Teaching people how to search the Internet and interact with mailing lists without being annoying is not the job of the Python devs, but it's a good way for people to help out. -- Steven From mikegraham at gmail.com Sun Jul 8 22:22:43 2012 From: mikegraham at gmail.com (Mike Graham) Date: Sun, 8 Jul 2012 16:22:43 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators Message-ID: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> A common stumbling block for new users is writing decorators that take arguments. To create a decorator like @timesn(n) def f(y): ... We write code like def timesn(n) def decorator(f): def inner(y): return n * f(y) return inner return decorator which confuses many users and can be a handful to type. I wonder if it would be clearer for people to write def timesn(n)(f): def inner(y): return n * f(y) return inner which is more concise and looks a lot more like a non-parametric decorator someone might have written already. The syntax is mostly self-explaining and could potentially be useful in other contexts. There exist tools like the decorator library to try to simplify this already, but in my experience they mostly serve to confuse people using decorators for the first time more. One thing I didn't specify was whether `n` was nonlocal or not and the behavior of something that keeps and reuses timesn(some_specific_n) multiple times. Does anyone think a feature like this may be useful? Regards, Mike From jeanpierreda at gmail.com Sun Jul 8 22:41:02 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 8 Jul 2012 16:41:02 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <CABicbJJ7dWNC1sAx=rqqBmHVmG-VbfXNh7H05z+zFete41iBTg@mail.gmail.com> On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote: > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. Ooh, +1 Semantically the function and the decorator-params are both "arguments" to the decorator, and this confuses a lot of people into writing silly things like def decorator(arg1, arg2, f): ... In actuality, these two are equivalent, and this new def syntax reflects that: @decorator(arg): def foo(...): ... def foo(...): ... foo = decorator(arg)(foo) As opposed to the usual definition syntax, which is not really intuitive. > One thing I didn't specify was whether `n` was nonlocal or not and the > behavior of something that keeps and reuses timesn(some_specific_n) > multiple times. For the purposes of decorators I don't think it matters. I would guess that keeping and reusing the first arg is more useful for non-decorator purposes, and making them a nonlocal is more consistent with the rest of Python, if we reuse the argument. Plus it'd align better semantically with the old way of defining parametric decorators. -- Devin From solipsis at pitrou.net Sun Jul 8 23:52:46 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 8 Jul 2012 23:52:46 +0200 Subject: [Python-ideas] Syntax for defining parametric decorators References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <20120708235246.7eaa54e8@pitrou.net> On Sun, 8 Jul 2012 16:22:43 -0400 Mike Graham <mikegraham at gmail.com> wrote: > > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner But then, why not: def timesn(n)(f)(y): return n * f(y) ? Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From ncoghlan at gmail.com Mon Jul 9 00:54:07 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Jul 2012 08:54:07 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> Step one is to stop referring to decorator factories as decorators. Beyond that, no, this is too limited - it only helps when there's no extra code in the outer scope which, in my experience, is the exception rather than the rule. Closures are an advanced programming concept - there are limits to how simple they are ever going to be. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/1fb07743/attachment.html> From grosser.meister.morti at gmx.net Mon Jul 9 01:13:40 2012 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Mon, 09 Jul 2012 01:13:40 +0200 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <4FFA1424.30009@gmx.net> On 07/08/2012 10:22 PM, Mike Graham wrote: > A common stumbling block for new users is writing decorators that take > arguments. To create a decorator like > > @timesn(n) > def f(y): > ... > > We write code like > > def timesn(n) > def decorator(f): > def inner(y): > return n * f(y) > return inner > return decorator > > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner > Why not write this?: def timesn(n)(f)(y): return n * f(y) This would be a currified function. One could implement something like that like this: def curry(f): for i in range(f.func_code.co_argcount-1): f = (lambda f: lambda *args: partial(f,*args))(f) return f @curry def timesn(n,f,y): return n * f(y) > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. > > There exist tools like the decorator library to try to simplify this > already, but in my experience they mostly serve to confuse people > using decorators for the first time more. > > One thing I didn't specify was whether `n` was nonlocal or not and the > behavior of something that keeps and reuses timesn(some_specific_n) > multiple times. > > Does anyone think a feature like this may be useful? > > Regards, > Mike From jeanpierreda at gmail.com Mon Jul 9 01:47:07 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 8 Jul 2012 19:47:07 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <4FFA1424.30009@gmx.net> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <4FFA1424.30009@gmx.net> Message-ID: <CABicbJJBzqE_oBOhdb8R0S2N-uBBw7AkuuaeRQVdCgpce-rmUg@mail.gmail.com> On Sun, Jul 8, 2012 at 7:13 PM, Mathias Panzenb?ck <grosser.meister.morti at gmx.net> wrote: > This would be a currified function. One could implement something like that > like this: > > def curry(f): > for i in range(f.func_code.co_argcount-1): > f = (lambda f: lambda *args: partial(f,*args))(f) > return f > > @curry > def timesn(n,f,y): > return n * f(y) But then how do you do: def linearcomposition(m, c=0)(f)(x): return m * f(x) + c Python isn't Haskell, so we can't really currify every argument and expect things to work out, esp. because of variadic functions (defaults or even *args and **kwargs). -- Devin From steve at pearwood.info Mon Jul 9 02:52:15 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 Jul 2012 10:52:15 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <4FFA2B3F.5010106@pearwood.info> Mike Graham wrote: > A common stumbling block for new users is writing decorators that take > arguments. To create a decorator like I question that assertion. Have you spent much time on the tutor at python.org mailing list? I have, and I can say that decorators is not something that many new users there are concerned about *at all*, let alone decorators which take arguments. I believe that decorator factories, and closures in general, are a moderately advanced technique, and not a stumbling block for most new users. I can't say I remember the last time I've seen anyone ask for help writing decorator factories on either the tutor or python-list mailing list. So I don't believe that making new syntax is neither needed or desirable. -1 on the idea. Some more issues/problems with the idea: > We write code like > > def timesn(n) > def decorator(f): > def inner(y): > return n * f(y) > return inner > return decorator In my experience, most decorator factories (or factory-factories in general, not just for generators but any time you need to wrap a factory function in a closure) require at least one pre-processing step, and occasionally one or two post-processing steps as well: def timesn(n) pre_process() def decorator(f): pre_process() @functools.wraps(f) def inner(y): return n * f(y) post_process() return inner post_process() return decorator With your suggested syntax, you lose a scope, and it is unclear to me what this would do: def timesn(n)(f): pre_process() @functools.wraps(f) def inner(f): return n * f(y) post_process() return inner > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner > > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. Concise, yes, but I disagree that it is self-explaining. I'm having a lot of difficulty in thinking of how I would explain it to even a moderately experienced user except by expanding it out to the explicit nested function form. I'm not sure I can even explain it to myself. What will happen when the user invariably writes something like this? def ordinary(a)(b): return a + b My guess is that they will get a syntax error, but it will be a syntax error in the wrong place: instead of clearly flagging the error (a)(b) as invalid syntax, as you get now py> def ordinary(a)(b): File "<stdin>", line 1 def ordinary(a)(b): ^ SyntaxError: invalid syntax Instead, the user will get a less useful syntax error on the following line, where the compiler sees that the function def is not followed immediately by another function def. py> def ordinary(a)(b): ... return a + b File "<stdin>", line 1 return a + b ^ SyntaxError: invalid syntax So whatever clarity you (allegedly) gain when writing decorator factories, you lose when making an error. > There exist tools like the decorator library to try to simplify this > already, but in my experience they mostly serve to confuse people > using decorators for the first time more. In my opinion, that's because the decorator and decorator factory concept are already as simple as they can possibly be. Anything you do to them adds complexity, not reduces it. I mean, it's a simple concept: a decorator is a kind of function which returns a function. If you put that inside a third function, you have a function which returns a function which returns a function. Any syntax to disguise that simplicity is only going to make things more complex, not less. I haven't given any thought to how this will effect class decorators, but I suspect it will run into the same problems. -- Steven From mikegraham at gmail.com Mon Jul 9 03:11:00 2012 From: mikegraham at gmail.com (Mike Graham) Date: Sun, 8 Jul 2012 21:11:00 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> Message-ID: <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > Beyond that, no, this is too limited - it only helps when there's no extra > code in the outer scope which, in my experience, is the exception rather > than the rule. I'm not sure I think that's the case. functools.wraps, functools.lru_cache, and reprlib.recursive_repr are the only stdlib decorator factories I could think of, and all of them could use this syntax. So could the ones I could quickly find in my own code. Mike From mikegraham at gmail.com Mon Jul 9 03:27:51 2012 From: mikegraham at gmail.com (Mike Graham) Date: Sun, 8 Jul 2012 21:27:51 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <4FFA2B3F.5010106@pearwood.info> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <4FFA2B3F.5010106@pearwood.info> Message-ID: <CAEBZo3Pq64GA2=1qxsZSg5-JRa7dG-hQh6Rz6WvLwKEQA_SrdQ@mail.gmail.com> On Sun, Jul 8, 2012 at 8:52 PM, Steven D'Aprano <steve at pearwood.info> wrote: > Mike Graham wrote: >> >> A common stumbling block for new users is writing decorators that take >> arguments. To create a decorator like > > I question that assertion. Have you spent much time on the tutor at python.org > mailing list? I have, and I can say that decorators is not something that > many new users there are concerned about *at all*, let alone decorators > which take arguments. I haven't, but I have spend considerable time helping people learn Python. I'm a gold-badge Python answerer on Stack Overflow, an op/regular on the #python IRC channel on freenode, and have taught people in the meatosphere. Decorators is certainly a topic that interests some people as they're still getting to know Python, and my most recent Stack Overflow question was from someone confused about some code using decorators and as recently as this morning I saw someone on #python say they "wanted to learn about decorators" in the abstract. In any event, I didn't mean _people completely brand new to Python_ when I talked about people who were new. > In my experience, most decorator factories (or factory-factories in general, > not just for generators but any time you need to wrap a factory function in > a closure) require at least one pre-processing step, and occasionally one or > two post-processing steps as well: In surveying actual decorator factories in the stdlib and my own code, I didn't notice it to be the case there was any preprocessing inside the outermost function. Maybe I just found the wrong ones. > Concise, yes, but I disagree that it is self-explaining. > > I'm having a lot of difficulty in thinking of how I would explain it to even > a moderately experienced user except by expanding it out to the explicit > nested function form. I'm not sure I can even explain it to myself. > > What will happen when the user invariably writes something like this? > > def ordinary(a)(b): > return a + b > > > My guess is that they will get a syntax error, but it will be a syntax error > in the wrong place: instead of clearly flagging the error (a)(b) as invalid > syntax, as you get now > > py> def ordinary(a)(b): > File "<stdin>", line 1 > def ordinary(a)(b): > ^ > SyntaxError: invalid syntax > > > Instead, the user will get a less useful syntax error on the following line, > where the compiler sees that the function def is not followed immediately by > another function def. > > > py> def ordinary(a)(b): > ... return a + b > File "<stdin>", line 1 > return a + b > ^ > SyntaxError: invalid syntax > > > So whatever clarity you (allegedly) gain when writing decorator factories, > you lose when making an error. I'm glad to get a different perspective here. It was evidently much more self-explanatory to the folks I discussed this with before posting. I wouldn't have expected this to require the def to be immediately followed by another def or for that to be the only (or most-common) case. I would expect that to be proper syntax under this idea, and for ordinary(3)(4) to be 7. I don't know if I said anything to suggest that the syntax should be quite so specialcased, but I hope not. Mike From jeanpierreda at gmail.com Mon Jul 9 03:45:11 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 8 Jul 2012 21:45:11 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <4FFA2B3F.5010106@pearwood.info> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <4FFA2B3F.5010106@pearwood.info> Message-ID: <CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com> On Sun, Jul 8, 2012 at 8:52 PM, Steven D'Aprano <steve at pearwood.info> wrote: > Concise, yes, but I disagree that it is self-explaining. I think the idea is that just as def foo(a): ... can be called as foo(x) to execute the body, def foo(a)(b): ... can be called as foo(x)(y) to execute the body. > I'm having a lot of difficulty in thinking of how I would explain it to even > a moderately experienced user except by expanding it out to the explicit > nested function form. I'm not sure I can even explain it to myself. What's wrong with explaining it by expansion? Also, I doubt it's as hard as you claim. Something similar -- curried functions -- have been a staple of functional programming languages since before Python ever existed. This should also help explain it to lots of new users with wider backgrounds. > What will happen when the user invariably writes something like this? > > def ordinary(a)(b): > return a + b Surely it'd define a curried function s.t. ordinary(1)(2) == 3? -- Devin From ncoghlan at gmail.com Mon Jul 9 03:46:16 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Jul 2012 11:46:16 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> Message-ID: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> On Mon, Jul 9, 2012 at 11:11 AM, Mike Graham <mikegraham at gmail.com> wrote: > On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> Beyond that, no, this is too limited - it only helps when there's no extra >> code in the outer scope which, in my experience, is the exception rather >> than the rule. > > I'm not sure I think that's the case. - functools.wraps Not really, as it uses functools.partial, not a nested def (sure you could rewrite it to use a nested def instead, but why?) > functools.lru_cache No, as it has pre- and post-processing steps around the nested function definition. - reprlib.recursive_repr Yes, this one would qualify. However, the simplest possible complete definition of decorators is as follows: - a decorator expression (the bit after "@") on a def statement must produce a function decorator - a decorator expression on a class statement must produce a class decorator - a function decorator accepts a function and produces another object. This is typically another function (or even the original function for annotation and registration decorators), but may also be a different kind of object such as a descriptor (e.g. property, classmethod, staticmethod) or context manager (e.g. contextlib.contextmanager) - a class decorator is similar, but accepts a class rather than a function - a decorator factory is any callable that returns a decorator - function decorators are often written as functions that return a nested function (1 level of lexical nesting) - function decorator factories are often written as functions that return a function that returns a functions (2 levels of lexical nesting) Creating a dedicated syntax for the special case of function decorator factories written to use lexical nesting makes the language as a whole *more* complicated rather than less. The only way to actually make them *simpler* would be to eliminate one or more of the bullet points from the above list, and that can't be done while retaining the current functionality. Yes, it means that to write efficient custom decorator factories you need to understand both the decoration process and lexical closures. That's *OK* - it just means writing custom decorator factories (as opposed to using those written by others) is a moderately advanced metaprogramming technique (it's still a lot simpler than writing a custom metaclass, though). Some concepts are just hard to get your head around, but that doesn't mean we should be creating dedicated syntax for a very specialised use case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Mon Jul 9 00:24:58 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 09 Jul 2012 10:24:58 +1200 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <4FFA08BA.8090504@canterbury.ac.nz> Mike Graham wrote: > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner +1 from me on this. Scheme has an analogous feature, btw, which is very elegant. I'd love to see this in Python. As has been pointed out, you can go further and collapse this into def timesn(n)(f)(y): return n * f(y) This means it would also be useful for non-parametric decorators as well, e.g. def fivetimes(f)(x): return 5 * f(x) -- Greg From jeanpierreda at gmail.com Mon Jul 9 03:55:09 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 8 Jul 2012 21:55:09 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> Message-ID: <CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com> On Sun, Jul 8, 2012 at 9:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > However, the simplest possible complete definition of decorators is as follows: > > - a decorator expression (the bit after "@") on a def statement must > produce a function decorator > - a decorator expression on a class statement must produce a class decorator > - a function decorator accepts a function and produces another object. > This is typically another function (or even the original function for > annotation and registration decorators), but may also be a different > kind of object such as a descriptor (e.g. property, classmethod, > staticmethod) or context manager (e.g. contextlib.contextmanager) > - a class decorator is similar, but accepts a class rather than a function > - a decorator factory is any callable that returns a decorator > - function decorators are often written as functions that return a > nested function (1 level of lexical nesting) > - function decorator factories are often written as functions that > return a function that returns a functions (2 levels of lexical > nesting) Mmm, this distinction of "kinds of decorators" is silly. The only requirement for something to be a decorator is that it takes one argument. The type of the argument could be anything. def dumb_decorator(x): return x + 2 @dumb_decorator @apply def foo(): return 5 > Creating a dedicated syntax for the special case of function decorator > factories written to use lexical nesting makes the language as a whole > *more* complicated rather than less. The only way to actually make > them *simpler* would be to eliminate one or more of the bullet points > from the above list, and that can't be done while retaining the > current functionality. Nothing about the OP's suggestion was specific to "function decorator factories". Observe a class decorator factory: def stupid_augmented_subclass_decorator(**kwargs)(cls): class MyClass(cls): pass MyClass.__dict__.update(**kwargs) return MyClass -- Devin From ncoghlan at gmail.com Mon Jul 9 04:00:41 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Jul 2012 12:00:41 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <4FFA2B3F.5010106@pearwood.info> <CABicbJKt5ZzfXuBqmhVXmJwiR=_OSuR4Si6hue3iHzwKCnqx4Q@mail.gmail.com> Message-ID: <CADiSq7f3JCVQmgyshroiAJX91NWzeXgQz-8OJwPf95WzATT6Lg@mail.gmail.com> On Mon, Jul 9, 2012 at 11:45 AM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: >> I'm having a lot of difficulty in thinking of how I would explain it to even >> a moderately experienced user except by expanding it out to the explicit >> nested function form. I'm not sure I can even explain it to myself. > > What's wrong with explaining it by expansion? > > Also, I doubt it's as hard as you claim. Something similar -- curried > functions -- have been a staple of functional programming languages > since before Python ever existed. This should also help explain it to > lots of new users with wider backgrounds. OK, I think it would be *much* better to approach the problem from that angle. def incremental(x)(f)(y): return x + f(y) As equivalent to: # implied names are not legal identifiers - the compiler gets to do that # because of its privileged role in naming things def incremental(x): def <incremental:1>(f): def <incremental:2>(y): return x +f(y) return <incremental:1> I'm still not convinced of the general applicability (since it breaks down as soon as you want to decorate or otherwise pre- or post-process any of the steps, thus forcing people to learn the full "callable-returning-a-callable" idiom anyway), but it's probably worth writing up as a PEP in the 3.4 timeframe. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 9 04:02:48 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Jul 2012 12:02:48 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> <CABicbJJPYi_GoWJ0Bxe3afuXTDVQzf+OV2aYUNC1oob4p7+c3A@mail.gmail.com> Message-ID: <CADiSq7eAwYDeduayqhk-=M2cJ_+sZ4D6rtZECPoSY2=GLGO_rw@mail.gmail.com> On Mon, Jul 9, 2012 at 11:55 AM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > Nothing about the OP's suggestion was specific to "function decorator > factories". The sole motivating use case presented was "A common stumbling block for new users is writing decorators that take arguments." It's a terrible motivating use case for curried functions, because they only work for the most trivial of decorator factories. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From haoyi.sg at gmail.com Mon Jul 9 04:04:14 2012 From: haoyi.sg at gmail.com (Haoyi Li) Date: Sun, 8 Jul 2012 19:04:14 -0700 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> Message-ID: <CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com> Just another point of data, Scala has exactly what is being proposed: multiple parameter lists as synctactic sugar for nested return functions http://stackoverflow.com/questions/4697404/scala-currying-by-nested-functions-or-by-multiple-parameter-lists It also works with all the non-trivial edge-casey stuff python has (args, default/named args, varargs, kwargs, splats). My experience with this is that it actually works pretty well, the number of "function returning function" types which require pre- or post- processing is relatively small, and often this pre/post processing can simply be moved into the body of the function for no real cost. The cases where that cannot be done, it usually is because apart from simply returning a new function, I'm also enclosing over some mutable state in the outer function. In these cases I've found that this implicitly captured mutable state is better handled with an object-based decorator, making the stateful nature explicit. The lru_cache decorator seems like a good candidate for using explicit objects to hold the cache, rather cunningly enclosed mutable state. I agree that as a "decorator factory helper", having extra syntax is unwarranted. However, I think that in the general case, being able to define curried functions in one line, a.l.a. def func(a)(b)(c) = a * b * c rather than def func(a): def wrapper_two(b): def wrapper_one(c): return a * b * c return wrapper_one return wrapper_two would in fact be very nice. From the Zen of Python, I would say these are big wins for - Flat is better than nested - Readability Counts - Beautiful is better than ugly whether it's a win or lose for Explicit is better than Implicit is debatable: In the implementation domain, you're being less explicit about what's actually happening (function returning functions) but in the "how-to-use-me" domain you're being more explicit about what you actually want (call this function with multiple parameter lists). def f(a)(b)(c) looks much more similar to how you would use it: f(1)(2)(3) than that horrible triple-nested monstrosity above. -Haoyi On Sun, Jul 8, 2012 at 6:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Mon, Jul 9, 2012 at 11:11 AM, Mike Graham <mikegraham at gmail.com> wrote: > > On Sun, Jul 8, 2012 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > >> Beyond that, no, this is too limited - it only helps when there's no > extra > >> code in the outer scope which, in my experience, is the exception rather > >> than the rule. > > > > I'm not sure I think that's the case. > > - functools.wraps > > Not really, as it uses functools.partial, not a nested def (sure you > could rewrite it to use a nested def instead, but why?) > > > functools.lru_cache > > No, as it has pre- and post-processing steps around the nested > function definition. > > - reprlib.recursive_repr > > Yes, this one would qualify. > > However, the simplest possible complete definition of decorators is as > follows: > > - a decorator expression (the bit after "@") on a def statement must > produce a function decorator > - a decorator expression on a class statement must produce a class > decorator > - a function decorator accepts a function and produces another object. > This is typically another function (or even the original function for > annotation and registration decorators), but may also be a different > kind of object such as a descriptor (e.g. property, classmethod, > staticmethod) or context manager (e.g. contextlib.contextmanager) > - a class decorator is similar, but accepts a class rather than a function > - a decorator factory is any callable that returns a decorator > - function decorators are often written as functions that return a > nested function (1 level of lexical nesting) > - function decorator factories are often written as functions that > return a function that returns a functions (2 levels of lexical > nesting) > > Creating a dedicated syntax for the special case of function decorator > factories written to use lexical nesting makes the language as a whole > *more* complicated rather than less. The only way to actually make > them *simpler* would be to eliminate one or more of the bullet points > from the above list, and that can't be done while retaining the > current functionality. > > Yes, it means that to write efficient custom decorator factories you > need to understand both the decoration process and lexical closures. > That's *OK* - it just means writing custom decorator factories (as > opposed to using those written by others) is a moderately advanced > metaprogramming technique (it's still a lot simpler than writing a > custom metaclass, though). > > Some concepts are just hard to get your head around, but that doesn't > mean we should be creating dedicated syntax for a very specialised use > case. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120708/d1ce8f89/attachment.html> From wuwei23 at gmail.com Mon Jul 9 06:16:51 2012 From: wuwei23 at gmail.com (alex23) Date: Sun, 8 Jul 2012 21:16:51 -0700 (PDT) Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <20120706181404.GB3347@jaerhard.com> References: <20120706181404.GB3347@jaerhard.com> Message-ID: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> On Jul 7, 4:14?am, J?rgen A. Erhard <jae+pyt... at jaerhard.com> wrote: > So... I consider it very strange and annoying that there's this code. > Ready to use. ?But you have to copy-and-paste it into your code > *somewhere* instead of easily importing it. I find it strange and annoying when people expect the standard library to contain _everything_. What you gain in the convenience of readily imported code imposes a cost on the ongoing maintenance and support of the standard lib. Are you volunteering to write the PEP for its inclusion, provide the implementation and support it until the end of time? > As to the "it makes using Python harder to learn", I beg to > differ: an addition to the *language* (like, say, metaclasses) *can* > make it more complicated. ?But additions to the stdlib? ?What about > "batteries included"? ?Not a motto anymore (I heard rumors, so maybe > that's the case) Every single function added to the library makes the library broader and thus more difficult to retain in memory. It _is_ possible to present a case without snide allusions based on hearsay: http://www.python.org/about/ > Putting these in some official(!) iterutils (or name it what you want) > package would be a solution for so many people. ?Yes, not everyone > needs grouper's current functionality. ?But for the many who do, it'd > be there already. So in the extremely likely outcome that a piece of functionality has multiple implementations with different interpretations of edge cases, who gets to decide which one is "more standard" than the other? If such a decision has to be made, then it's not really "standard". > And someone thought those recipes are useful, > didn't you? ?So why not make them more easily available? Because not everything needs to be in the standard library. Because searching somewhere like Activestate's cookbook and copying the implementations you find useful really isn't an onerous task. Because if *you* feel there's an obvious lack in the stdlib then roll your own package that covers it and put it up on PyPI. If it becomes insanely popular, then you have a lot more leverage for insisting that something would be beneficial to be included other than gut feeling & convenience. From ncoghlan at gmail.com Mon Jul 9 06:58:06 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Jul 2012 14:58:06 +1000 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> Message-ID: <CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com> On Mon, Jul 9, 2012 at 2:16 PM, alex23 <wuwei23 at gmail.com> wrote: > So in the extremely likely outcome that a piece of functionality has > multiple implementations with different interpretations of edge cases, > who gets to decide which one is "more standard" than the other? If > such a decision has to be made, then it's not really "standard". This is often what lies at the heart of "not every 3 line function needs to be <a builtin>/<in the standard library>". It's very easy to hit a point of diminishing returns where the number of possible alternatives mean that attempting to create an abstraction layer ends up creating a UI that is *more complicated* than just writing your own utility function that does exactly what you want. Grouping is one such API (fixed length? Variable length with a delimiter? Pad last group? Drop last group? Error early? Accept arbitrary iterables? Accept sequences only? Support overlapping windows?). Recursive descent into arbitrary collections is another (Error on reference loops? Treat strings/bytes/bytearray as atomic types? Treat any implementor of the buffer interface as an atomic type? Support only hashable objects?). There are a few key reasons why things end up in the standard library: 1. They're closely coupled to the language definition and implementation and should be updated in sync with it (e.g. sys, imp, gc, importlib, dis, inspect) 2. We want to use them in other parts of the standard library or its test suite (e.g. collections, many of the unittest enhancements) 3. They solve a common problem that is otherwise prone to being handled with incomplete solutions that lead to incorrect code (e.g. ipaddress instead of regex based recipes for processing IP addresses) 4. It's an old module that would probably be left to PyPI these days, but was added in earlier times when stdlib inclusion criteria were less strict (but isn't broken so there's no harm in keeping it around). Sometimes a problem is sufficiently common, and has few enough variations, that it's worth creating and providing a standard version. That's the general aim of the itertools module. Other times, the core problem is difficult enough that it's worth providing a standard solution, even if it does mean including a vast array of configuration options (e.g. subprocess. Popen). However, sometimes, the correct answer to "Hey, this is a really common pattern" is not "We should provide an API that uses that pattern internally" but "we should document this pattern, so people know it's a common idiom and can tailor it to their specific use case and preferences". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Mon Jul 9 07:48:37 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 09 Jul 2012 14:48:37 +0900 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CADiSq7fZ8vwyvXOXhAAghe_D6z8aXzPJpLgBibJAkszyEDWyKg@mail.gmail.com> <CAEBZo3M_+ggmYCRsAUtNedWZLfd6ASE2Dc1JjsYDG6c-ru47wQ@mail.gmail.com> <CADiSq7dGhOqHkB-6UwBo9R+ntc885Z1nLOiaGuZyk=ampovSTQ@mail.gmail.com> <CALruUQJVTNo1bKWJ9U38--qcEO+2a_=k4s60NPOL28hMqsb2PA@mail.gmail.com> Message-ID: <87obnp5vt6.fsf@uwakimon.sk.tsukuba.ac.jp> Haoyi Li writes: > However, I think that in the general case, being able to define > curried functions in one line, a.l.a. [...] > def f(a)(b)(c) > > looks much more similar to how you would use it: > > f(1)(2)(3) I don't understand. I wouldn't use it that way! I would think if that were common usage, and the curried form only occasional, def f(a,b,c): pass f(1,2,3) would be more natural in Python (and use functools.partial for the occasional currying). Presumably if you were using this syntax, you'd be doing something more like def f(a)(b)(c): pass def g(h, x): h(x) g(f(1)(2), 3) which doesn't look very nice to me, nor do I find the def to be a particularly intuitive way of reminding me that the proper usage is a call of a function in curried form (specifically, I would not be reminded of whether the expected curried form is a 1st-, 2nd-, or 3rd-order function). I also suppose that you wouldn't be able to do def f(a)(b)(c): pass def g(x,y): pass h(f(1)) h(g) (ie, without LBYL or EAFP constructs in h, which would simply be throwing the complexity into the caller's backyard). Even though def bar(a)(b): pass def baz(x): pass quux(bar(1)) quux(baz) would work fine. So to me, there may be something in this syntax, but my initial impression is that it's trying to turn Python into something it's not. From haoyi.sg at gmail.com Mon Jul 9 11:07:49 2012 From: haoyi.sg at gmail.com (Haoyi Li) Date: Mon, 9 Jul 2012 02:07:49 -0700 Subject: [Python-ideas] Syntax for defining parametric decorators Message-ID: <2237588560696344575@unknownmsgid> I admit calling it all at once would be very strange, but i still think the "call me three times to get result" comes across waaaay more clearly than with nested defs. Multiple arguments in one list is about the same, it's true, until you really need the curried version, like in decorators, then you end up with the nasty tower-of-defs which sparked this discussion. Regarding "not knowing how many times it should be called", the idea is that you break it up when you have a meaningful, reusable unit. That's to say you wouldn't do a def f(a)(b)(c) Unless f f(1) f(1)(2) f(1)(2)(3) Are all meaningful, useful functions. Doesn't happen very often, granted, but when it does (e.g. for decorator factories and decorators) it'd be nice to have. I view curried defs like default parameters: Doesnt add anything that cant already be done (you can emulate default args with telescoping methods) but cuts boilerplate, makes things clearer and overall nice to have. Whether it's nice enough to justify extra syntax is of course open for debate. From: Stephen J. Turnbull Sent: 7/9/2012 1:48 AM To: Haoyi Li Cc: Nick Coghlan; Python-Ideas Subject: Re: [Python-ideas] Syntax for defining parametric decorators Haoyi Li writes: > However, I think that in the general case, being able to define > curried functions in one line, a.l.a. [...] > def f(a)(b)(c) > > looks much more similar to how you would use it: > > f(1)(2)(3) I don't understand. I wouldn't use it that way! I would think if that were common usage, and the curried form only occasional, def f(a,b,c): pass f(1,2,3) would be more natural in Python (and use functools.partial for the occasional currying). Presumably if you were using this syntax, you'd be doing something more like def f(a)(b)(c): pass def g(h, x): h(x) g(f(1)(2), 3) which doesn't look very nice to me, nor do I find the def to be a particularly intuitive way of reminding me that the proper usage is a call of a function in curried form (specifically, I would not be reminded of whether the expected curried form is a 1st-, 2nd-, or 3rd-order function). I also suppose that you wouldn't be able to do def f(a)(b)(c): pass def g(x,y): pass h(f(1)) h(g) (ie, without LBYL or EAFP constructs in h, which would simply be throwing the complexity into the caller's backyard). Even though def bar(a)(b): pass def baz(x): pass quux(bar(1)) quux(baz) would work fine. So to me, there may be something in this syntax, but my initial impression is that it's trying to turn Python into something it's not. From stefan_ml at behnel.de Mon Jul 9 11:47:48 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Jul 2012 11:47:48 +0200 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <jte9c4$2ej$1@dough.gmane.org> Mike Graham, 08.07.2012 22:22: > A common stumbling block for new users is writing decorators that take > arguments. To create a decorator like > > @timesn(n) > def f(y): > ... > > We write code like > > def timesn(n) > def decorator(f): > def inner(y): > return n * f(y) > return inner > return decorator > > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner > > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. >From an innocent look, I have no idea what the syntax is supposed to mean. Clearly doesn't hint at a factory for me. Stefan From mwm at mired.org Mon Jul 9 11:56:55 2012 From: mwm at mired.org (Mike Meyer) Date: Mon, 9 Jul 2012 05:56:55 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <jte9c4$2ej$1@dough.gmane.org> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> Message-ID: <20120709055655.49a978c7@bhuda.mired.org> On Mon, 09 Jul 2012 11:47:48 +0200 Stefan Behnel <stefan_ml at behnel.de> wrote: > Mike Graham, 08.07.2012 22:22: > > def timesn(n)(f): > > def inner(y): > > return n * f(y) > > return inner > > > > which is more concise and looks a lot more like a non-parametric > > decorator someone might have written already. The syntax is mostly > > self-explaining and could potentially be useful in other contexts. > > >From an innocent look, I have no idea what the syntax is supposed to mean. > Clearly doesn't hint at a factory for me. Does to me. timesn is clearly a function of one argument that is going to return a callable object that takes one argument. So: x5 = times(5) print x5(7) should print 35. I'd be interested in seeing an implementation of partial that used this mechanism. Should be straightforward, but I'm in the midst of a crunch. <mike -- Mike Meyer <mwm at mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From steve at pearwood.info Mon Jul 9 13:32:59 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 Jul 2012 21:32:59 +1000 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CADiSq7c-0YjgNQGP6DS06DAGRFt14bNo3Xb1-HZ_xHP9Ym_6hA@mail.gmail.com> Message-ID: <4FFAC16B.7030206@pearwood.info> Nick Coghlan wrote: > It's very easy to hit a point of diminishing returns where the number > of possible alternatives mean that attempting to create an abstraction > layer ends up creating a UI that is *more complicated* than just > writing your own utility function that does exactly what you want. [...] +1 on everything Nick says here. One additional comment: > However, sometimes, the correct answer to "Hey, this is a really > common pattern" is not "We should provide an API that uses that > pattern internally" but "we should document this pattern, so people > know it's a common idiom and can tailor it to their specific use case > and preferences". Well, maybe, but documenting software patterns are not necessarily "our" job (whoever "our" is -- the Python devs, or python.org, or the PSI, or whoever people think is responsible). Python has not been a one-man project for many, many years. There is a rich Python ecosystem now, not just Guido or the Python devs, but StackOverflow, comp.lang.python, dozens of books, more bloggers than you can poke a stick at, #python, etc etc etc. It's not "our" responsibility to teach every Python programmer every conceivable pattern. At some point we can just draw a line and say "Yeah, that's a useful pattern, but we're not going to document it. Somebody else can do it." Even if that line is somewhat arbitrary. -- Steven From ironfroggy at gmail.com Mon Jul 9 14:28:58 2012 From: ironfroggy at gmail.com (Calvin Spealman) Date: Mon, 9 Jul 2012 08:28:58 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com> On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote: > A common stumbling block for new users is writing decorators that take > arguments. To create a decorator like > > @timesn(n) > def f(y): > ... > > We write code like > > def timesn(n) > def decorator(f): > def inner(y): > return n * f(y) > return inner > return decorator > > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner > > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. > > There exist tools like the decorator library to try to simplify this > already, but in my experience they mostly serve to confuse people > using decorators for the first time more. > > One thing I didn't specify was whether `n` was nonlocal or not and the > behavior of something that keeps and reuses timesn(some_specific_n) > multiple times. > > Does anyone think a feature like this may be useful? While I'm a bit taken aback by the syntax, I recognize this is mostly because I'm not used to it. Objectively, it makes enough sense that I think it fits. I think even if the only use case is decorator factories, they are a useful enough thing that this is okay. We have a specific syntax to support using decorators, why not better support writing them? Still, I'd like to see any ideas people have for alternative uses for this syntax. > Regards, > Mike > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From jsbueno at python.org.br Mon Jul 9 15:14:26 2012 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 9 Jul 2012 10:14:26 -0300 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <CAGaVwhRfDnKpCLask7r9YiR1WG11Bn7pbWOUVHGwLFrfWcTSDQ@mail.gmail.com> Message-ID: <CAH0mxTROfkmJdtsX1eY-7wTPa_n3SPOq_O3n3bx_ZAax5=YpiQ@mail.gmail.com> On 9 July 2012 09:28, Calvin Spealman <ironfroggy at gmail.com> wrote: > On Sun, Jul 8, 2012 at 4:22 PM, Mike Graham <mikegraham at gmail.com> wrote: >> A common stumbling block for new users is writing decorators that take >> arguments. To create a decorator like >> >> @timesn(n) >> def f(y): >> ... -1 for the new syntax, since a decorator for trivial decorator factories (rather, parameterized decorators) can be written in a total of 5 lines: from functools import partial def parameterized(decorator): def part_decorator(*args, **kw): return partial(decorator, *args, **kw) return part_decorator >>> @parameterized ... def times(n, func): ... def new_func(*args, **kw): ... return n * func(*args, **kw) ... return new_func ... >>> >>> @times(3) ... def add(x ,y): ... return x + y ... >>> add(1,1) 6 ------- For more complex cases, requiring pre processing, post processing, and so on, the normal syntax can cut it. And, certainly, a more complex such decorator could be written, so that it accepts some arguments itself. (I would be +1 for such a decorator in functools) js -><- From aquavitae69 at gmail.com Mon Jul 9 17:18:23 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 9 Jul 2012 17:18:23 +0200 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> Message-ID: <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com> On Jul 9, 2012 6:17 AM, "alex23" <wuwei23 at gmail.com> wrote: > > On Jul 7, 4:14 am, J?rgen A. Erhard <jae+pyt... at jaerhard.com> wrote: > > So... I consider it very strange and annoying that there's this code. > > Ready to use. But you have to copy-and-paste it into your code > > *somewhere* instead of easily importing it. > > I find it strange and annoying when people expect the standard library > to contain _everything_. What you gain in the convenience of readily > imported code imposes a cost on the ongoing maintenance and support of > the standard lib. Are you volunteering to write the PEP for its > inclusion, provide the implementation and support it until the end of > time? While I agree that these recipes are not suitable for inclusion in stdlib (for reasons made elsewhere), I don't agree with your (somewhat aggressive) reasoning. The implementation has already been provided, and is being maintained in the form of documentation. Simply copying the recipes into a module is hardly an onerous task. And fairly often people mailing to this list *are* volunteering to write a PEP, provide an implementation, etc. > > > As to the "it makes using Python harder to learn", I beg to > > differ: an addition to the *language* (like, say, metaclasses) *can* > > make it more complicated. But additions to the stdlib? What about > > "batteries included"? Not a motto anymore (I heard rumors, so maybe > > that's the case) > > Every single function added to the library makes the library broader > and thus more difficult to retain in memory. > > It _is_ possible to present a case without snide allusions based on > hearsay: http://www.python.org/about/ > > > Putting these in some official(!) iterutils (or name it what you want) > > package would be a solution for so many people. Yes, not everyone > > needs grouper's current functionality. But for the many who do, it'd > > be there already. > > So in the extremely likely outcome that a piece of functionality has > multiple implementations with different interpretations of edge cases, > who gets to decide which one is "more standard" than the other? If > such a decision has to be made, then it's not really "standard". > > > And someone thought those recipes are useful, > > didn't you? So why not make them more easily available? > > Because not everything needs to be in the standard library. > > Because searching somewhere like Activestate's cookbook and copying > the implementations you find useful really isn't an onerous task. > > Because if *you* feel there's an obvious lack in the stdlib then roll > your own package that covers it and put it up on PyPI. If it becomes > insanely popular, then you have a lot more leverage for insisting that > something would be beneficial to be included other than gut feeling & > convenience. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/1fbf8dc5/attachment.html> From raymond.hettinger at gmail.com Mon Jul 9 18:18:54 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 9 Jul 2012 09:18:54 -0700 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com> Message-ID: <A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com> On Jul 9, 2012, at 8:18 AM, David Townshend wrote: > . Simply copying the recipes into a module is hardly an onerous task. In fact, it has already been done :-) http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120709/3765f429/attachment.html> From tjreedy at udel.edu Mon Jul 9 21:47:10 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Jul 2012 15:47:10 -0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CAEgL-fdV1+O7s7Rqv7gDRTJD0d+2Xay9nqeMbFUr-M2pSh-j7g@mail.gmail.com> <A68C315E-8786-4B9A-9530-79EF74A2ECDD@gmail.com> Message-ID: <jtfcge$8o1$2@dough.gmane.org> On 7/9/2012 12:18 PM, Raymond Hettinger wrote: > > On Jul 9, 2012, at 8:18 AM, David Townshend wrote: > >> . Simply copying the recipes into a module is hardly an onerous task. > > In fact, it has already been done :-) Multiple times. Possible addition to the doc in 9.1.2 after first sentence: These and similar recipes have been incorporated in various third-party modules. See ... > http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search -- Terry Jan Reedy From jeanpierreda at gmail.com Mon Jul 9 22:12:22 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 9 Jul 2012 16:12:22 -0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> Message-ID: <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> On Mon, Jul 9, 2012 at 12:16 AM, alex23 <wuwei23 at gmail.com> wrote: > I find it strange and annoying when people expect the standard library > to contain _everything_. What you gain in the convenience of readily > imported code imposes a cost on the ongoing maintenance and support of > the standard lib. Are you volunteering to write the PEP for its > inclusion, provide the implementation and support it until the end of > time? Are you saying the recipes are unsupported? -- Devin From wuwei23 at gmail.com Tue Jul 10 00:28:41 2012 From: wuwei23 at gmail.com (wu wei) Date: Tue, 10 Jul 2012 08:28:41 +1000 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> Message-ID: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda at gmail.com> wrote: > Are you saying the recipes are unsupported? > This is getting ridiculous. I'm not sure what is so controversial about not wanting to scrape up every piece of crap code off the web that some lazy coder can't be bothered to implement contextually and shove them into the _standard library_. Are _you_ seriously saying that every one of those random bits of code _is_ supported? That they will all be kept up to date as the language evolves? That they won't turn into another module that people complain is "stale"? Or another "why tkinter and not <x>" argument? Every inclusion to the library adds overhead to finding and using elements of the library. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120710/3814f28c/attachment.html> From phd at phdru.name Tue Jul 10 00:36:26 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jul 2012 02:36:26 +0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> Message-ID: <20120709223626.GB1279@iskra.aviel.ru> On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23 at gmail.com> wrote: > Every inclusion to the library adds overhead to finding and using elements > of the library. And supporting it. It's much harder to remove bits from stdlib than to add to it because stdlib have to maintain backward compatibility. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From alexandre.zani at gmail.com Tue Jul 10 00:42:52 2012 From: alexandre.zani at gmail.com (Alexandre Zani) Date: Mon, 9 Jul 2012 15:42:52 -0700 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <20120709223626.GB1279@iskra.aviel.ru> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> <20120709223626.GB1279@iskra.aviel.ru> Message-ID: <CAJVMpoqh=66p+uYEVi_4og4gFT6r0XFZJ1kzFgm_Ei+DRs-rOQ@mail.gmail.com> On Mon, Jul 9, 2012 at 3:36 PM, Oleg Broytman <phd at phdru.name> wrote: > On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23 at gmail.com> wrote: >> Every inclusion to the library adds overhead to finding and using elements >> of the library. > > And supporting it. It's much harder to remove bits from stdlib than > to add to it because stdlib have to maintain backward compatibility. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas It is a good point that those recipes because they are in the official docs are already requiring support and maintenance. However, as somebody above mentioned, there are many possible variations on those recipes and so it makes a lot more sense to tell people to implement their own version of grouper that will match their specific needs rather than create an infinitely configurable version of grouper in the stdlib. Furthermore, if the maintenance of those recipes ever becomes undesirable, it is a simple matter to strike them from the docs. The same cannot be said if they were inducted in the stdlib. Alexandre Zani From jeanpierreda at gmail.com Tue Jul 10 00:45:55 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 9 Jul 2012 18:45:55 -0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> Message-ID: <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com> On Mon, Jul 9, 2012 at 6:28 PM, wu wei <wuwei23 at gmail.com> wrote: > On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda at gmail.com> > wrote: >> >> Are you saying the recipes are unsupported? > > > This is getting ridiculous. I'm not sure what is so controversial about not > wanting to scrape up every piece of crap code off the web that some lazy This is code from the official documentation. > coder can't be bothered to implement contextually and shove them into the > _standard library_. Are _you_ seriously saying that every one of those > random bits of code _is_ supported? No, I am suggesting that example code that exists in the documentation is supported. I think you are confused. -- Devin From tjreedy at udel.edu Tue Jul 10 03:52:55 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Jul 2012 21:52:55 -0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com> Message-ID: <jtg1u7$9nc$1@dough.gmane.org> On 7/9/2012 6:45 PM, Devin Jeanpierre wrote: > No, I am suggesting that example code that exists in the documentation > is supported. Well, in the sense that we try to make sure that they work as advertised *with the example data*, yes. However, that is the end of the support. Examples are just examples, put there to illustrate a didactic point. They are often simplified to serve the specific purpose. They are not intended to be production code, and they are not supported as that. Error handling is eliminated for clarity.# The api has not necessarily been thought through very well and is subject to change.# The multiple examples are not particularly meant to work together. #Whereas, for example, the new in 3.3 ipaddress module, while based on existing 3rd party code, has had its api reexamined to make sure it well defined, documented, and something we will (likely) stick with. Nick has added a lot of error checking code with messages that try to say what is wrong rather than just 'you goofed'. And oh yes, the test suite has been expanded and reviewed with an aim to 100% coverage. -- Terry Jan Reedy From jeanpierreda at gmail.com Tue Jul 10 05:38:38 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 9 Jul 2012 23:38:38 -0400 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <jtg1u7$9nc$1@dough.gmane.org> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com> <jtg1u7$9nc$1@dough.gmane.org> Message-ID: <CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com> On Mon, Jul 9, 2012 at 9:52 PM, Terry Reedy <tjreedy at udel.edu> wrote: > Well, in the sense that we try to make sure that they work as advertised > *with the example data*, yes. However, that is the end of the support. > Examples are just examples, put there to illustrate a didactic point. They > are often simplified to serve the specific purpose. They are not intended to > be production code, and they are not supported as that. I've never assumed that (obviously, I guess), and that isn't stated in the docs. If that's the level of support of those recipes, this should be made clear inside the itertools documentation, because they _are_ used in peoples' code. For example, see the third party module linked earlier in the thread. On the contrary, the docs state the recipes are for creating an "extended toolset", implying that they can be used freely the same as the rest of the code. -- Devin From stephen at xemacs.org Tue Jul 10 06:39:50 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Jul 2012 13:39:50 +0900 Subject: [Python-ideas] itertools recipes: why not add them to the stdlib *somewhere*? In-Reply-To: <CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com> References: <20120706181404.GB3347@jaerhard.com> <206b7fef-3f0a-48c5-b994-a0e75cfcac0e@vb9g2000pbc.googlegroups.com> <CABicbJK-rXY95z8mVNRFLXOY9KsaNzwiUj3Rhg+LSf=3BjBEWg@mail.gmail.com> <CAM32KhqFg2Bjb9Jx8aiWx-ZtJMNxAf_uLDMGobjNi7Z-xTUP7A@mail.gmail.com> <CABicbJJG06V-ihfwBohVhRSGyHt-UMmFG7JygjOWGU3_jvaZWA@mail.gmail.com> <jtg1u7$9nc$1@dough.gmane.org> <CABicbJJozkqVj2BHi9q7v8aY9MdW50EfHFDCKarW1nLAFHmTPQ@mail.gmail.com> Message-ID: <87fw905iw9.fsf@uwakimon.sk.tsukuba.ac.jp> Devin Jeanpierre writes: > I've never assumed that (obviously, I guess), and that isn't stated in > the docs. If that's the level of support of those recipes, this should > be made clear inside the itertools documentation, It already is clear. The very name "recipe" indicates that it may not be to everyone's taste, that it's not carefully designed to handle corner cases, and that people should review the code for their own use. The (very sparse) documentation associated with the recipes, which is clearly oriented to advising you how you can (a) produce similar code for different functions yourself and (b) make it better further emphasizes what the name indicates. Of course, all that may not be obvious to non-native speakers, but that's a completely different issue from the starting point of this thread, which was that these recipes belong in the stdlib rather than in its documentation. I have nothing to add to Terry's post explaining why that is not appropriate for these recipes "as is." > because they _are_ used in peoples' code. Of course they are. That's what they're there for. From techtonik at gmail.com Tue Jul 10 08:22:11 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 10 Jul 2012 09:22:11 +0300 Subject: [Python-ideas] Explanation on how to search the archives In-Reply-To: <4FF887A7.5070101@pearwood.info> References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> <4FF887A7.5070101@pearwood.info> Message-ID: <CAPkN8xJg-wZ8A3_iranDCfmhpWYFu9xms0U9kGgg90Hi-7qZyw@mail.gmail.com> On Sat, Jul 7, 2012 at 10:01 PM, Steven D'Aprano <steve at pearwood.info> wrote: > > Teaching people how to search the Internet and > interact with mailing lists without being annoying is not the job of the > Python devs Aren't they primarily Python devs who are annoyed and complain that people don't search properly? I don't think your proposal to just write somewhere on the Internet, but not on python.org helps. Even wiki.python.org doesn't help if people don't have a chance to read it before posting. The idea about giving a convenient search tool is pretty valid. The true problem is that *.python.org is unable to do anything (not even a search form here - http://mail.python.org/mailman/listinfo/python-ideas). For usenet there were auto announcements, for Google Groups and forums there are sticky threads with advices. -- anatoly t. From ncoghlan at gmail.com Tue Jul 10 08:32:24 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Jul 2012 16:32:24 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <jte9c4$2ej$1@dough.gmane.org> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> Message-ID: <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> On Mon, Jul 9, 2012 at 7:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > >From an innocent look, I have no idea what the syntax is supposed to mean. > Clearly doesn't hint at a factory for me. I should also mention that I have a different proposal that affects the way one would write functions-that-returns-functions. I've been messing around with the idea of statement local namespaces for years (see PEP 3150) trying to find something that I consider better than the status quo, and PEP 403's statement local function and class definitions (http://www.python.org/dev/peps/pep-0403/) are the current incarnation. With those, the number of statements in a simple wrapping decorator factory doesn't change, but the return statements can be moved above their respective function definitions: def notify_on_call(callback, *cb_args, **cb_kwds): in return decorator def decorator(f): in return wrapped @functools.wraps(f) def wrapped(*args, **kwds): callback(*cb_args, cb_kwds) return f(*args, **kwds) Rather than the current out-of-order: def notify_on_call(callback, *cb_args, **cb_kwds): def decorator(f): @functools.wraps(f) def wrapped(*args, **kwds): callback(*cb_args, cb_kwds) return f(*args, **kwds) return wrapped return decorator (Note: I haven't updated the PEP in a while, so it currently still disallows combining the in statement with decorators - I think that's a mistake, and will be updating it some time post 3.3) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From techtonik at gmail.com Tue Jul 10 09:01:02 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 10 Jul 2012 10:01:02 +0300 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> Message-ID: <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com> On Thu, Jul 5, 2012 at 11:24 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote: > 2012/7/5 anatoly techtonik <techtonik at gmail.com>: >> This makes me kind of sad. You have Python installed. Why can't you >> just crossplatformly do: >> >> mkdir nacl >> cd nacl >> python -m urllib get >> http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/update_sdk.py >> python update_sdk.py > > I'm sure there is already a way with standard python tools. Something > along these lines: > > python -c "from urllib.request import urlretrieve; urlretrieve('URL', > 'update_sdk.zip')" > python -m update_sdk.zip > > The second command will work if the zip file has a __main__.py. > Do you think we need other tools? Wow! Nice, but still a hack. I doubt many people wouls accept magic files messing in the root dir of the package. The only way it will look good is: python -m update_sdk.zip sdk_update[.py] but I don't know if that's supported. -- anatoly t. From amauryfa at gmail.com Tue Jul 10 09:51:25 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 10 Jul 2012 09:51:25 +0200 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com> Message-ID: <CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com> 2012/7/10 anatoly techtonik <techtonik at gmail.com>: > Wow! Nice, but still a hack. I doubt many people wouls accept magic > files messing in the root dir of the package. The only way it will > look good is: > > python -m update_sdk.zip sdk_update[.py] > > but I don't know if that's supported. -m does work with zip files (with a __main__.py file) and command line arguments are passed. Or did you expect another kind of support? -- Amaury Forgeot d'Arc From christopherreay at gmail.com Tue Jul 10 09:58:31 2012 From: christopherreay at gmail.com (Christopher Reay) Date: Tue, 10 Jul 2012 08:58:31 +0100 Subject: [Python-ideas] Explanation on how to search the archives In-Reply-To: <4FF887A7.5070101@pearwood.info> References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> <4FF887A7.5070101@pearwood.info> Message-ID: <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com> Im thinking something like: Python-ideas mailing list Python-ideas at python.org http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas> How to Search the Archives: http://mail.python.org/dontReinventTheWheel Searching the archives is the *first* place you should go when exploring new ideas. They are full of (mostly!) excellent ideas, as well as showing how to explain your idea well, and helping you to not reinvent the wheel... or the tabloid. Please search for relevant keywords around your idea before posting to python-ideas. Thanks! As the footer of the emails. This is a small addition which provides some great meta information to "people with great ideas" and also allows a terse and simple response when a veteran detects a repeated idea. -- Be prepared to have your predictions come true -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120710/9d66f2b5/attachment.html> From ncoghlan at gmail.com Tue Jul 10 11:11:58 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Jul 2012 19:11:58 +1000 Subject: [Python-ideas] Python as a tool to download stuff for bootstrapping In-Reply-To: <CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com> References: <CAPkN8xJnvN8RuUPiYCuURDUnuPJCX3kiZbAhcm_9SNQLOhCtLw@mail.gmail.com> <CAGmFidaPwmA13k3cOCj+50AJCJRUe3GaVBBpiEfRDAK91oE2Nw@mail.gmail.com> <CAPkN8xKuLoK+OPyxwHnoLkPOY1Kbz7nAH7SV4NckVbkSVvEc-A@mail.gmail.com> <CAGmFida1RBGRTZw6hao0+U5m1pcWwZiCiS63K_a48ouLjgSqdQ@mail.gmail.com> Message-ID: <CADiSq7dKfrNfDr5uv__GQe8CvJvETrU8wiDCW3RW_PuC_xeWhA@mail.gmail.com> On Tue, Jul 10, 2012 at 5:51 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote: > 2012/7/10 anatoly techtonik <techtonik at gmail.com>: >> Wow! Nice, but still a hack. I doubt many people wouls accept magic >> files messing in the root dir of the package. The only way it will >> look good is: >> >> python -m update_sdk.zip sdk_update[.py] >> >> but I don't know if that's supported. > > -m does work with zip files (with a __main__.py file) and command line > arguments are passed. > Or did you expect another kind of support? -m doesn't handle zip files - you just execute them directly: python <script> = source files, bytecode files, directories, zip files (requires a top level __main__.py in the latter two cases) python -m <module> = modules, packages (requires a __main__ submodule for the latter case) The switch is only to tell the interpreter whether you're providing a filesystem path (default) or a Python module name (-m switch). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Jul 11 03:12:10 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 Jul 2012 11:12:10 +1000 Subject: [Python-ideas] Explanation on how to search the archives In-Reply-To: <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com> References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> <4FF887A7.5070101@pearwood.info> <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com> Message-ID: <4FFCD2EA.7090705@pearwood.info> Christopher Reay wrote: > Im thinking something like: [...] > How to Search the Archives: > http://mail.python.org/dontReinventTheWheel > Searching the archives is the *first* place you should go when exploring > new ideas. ... > As the footer of the emails. > > This is a small addition which provides some great meta information to > "people with great ideas" and also allows a terse and simple response when > a veteran detects a repeated idea. More bloat for every single email which will be utterly ignored. Case in point: the mail digest for the python-dev mailing list already states: When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..." and similar for python-list, tutor, etc. And this message is at the *top* of the digest, not the bottom, and a mere two lines, not a big wall of text[1]. Yet there is still a steady stream of posts like these: http://mail.python.org/pipermail/python-ideas/2012-June/015547.html http://mail.python.org/pipermail/python-dev/2012-July/120910.html http://mail.python.org/pipermail/tutor/2012-July/090213.html Don't believe me? How often do you read the "This email is Confidential blah blah blah yadda watermelon blah" footers at the bottom of oh-so-many corporate emails? Everybody, sender and receiver, carries the cost of additional bloat in every email, and the people who most need the reminder that Python is twenty years old and this "brilliant idea" most likely has been thought of a dozen times before are the ones who are least likely to read instructions or pay any attention to them even if they do. [1] Standards of what counts as big blocks of text have been steadily falling since the Internet became popular. I'm just waiting for people to start replying to twitter posts with "TL;DR", at which point I intend to turn my computer off and move to a cave in the desert. -- Steven From ben+python at benfinney.id.au Thu Jul 12 03:31:57 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 12 Jul 2012 11:31:57 +1000 Subject: [Python-ideas] Explanation on how to search the archives References: <CAMgkT_8qZ9pcKCSYONwwTJr3qsSQosV=2BS+B8KbnjjPiJ_tTw@mail.gmail.com> <4FF887A7.5070101@pearwood.info> <CAMgkT__L_KU=jRMob6tCjtHOK_FJT6aAAk87vMzgQdkpG6L1yg@mail.gmail.com> <4FFCD2EA.7090705@pearwood.info> Message-ID: <87y5mpojci.fsf@benfinney.id.au> Steven D'Aprano <steve at pearwood.info> writes: > [1] Standards of what counts as big blocks of text have been steadily > falling since the Internet became popular. I'm just waiting for people > to start replying to twitter posts with "TL;DR", at which point I > intend to turn my computer off and move to a cave in the desert. I used to read ?TL;DR?, but now I don't have the time to read five characters. -- \ ?Simplicity is prerequisite for reliability.? ?Edsger W. | `\ Dijkstra | _o__) | Ben Finney From steve at pearwood.info Thu Jul 12 03:37:00 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 12 Jul 2012 11:37:00 +1000 Subject: [Python-ideas] Moving return above statements [was Re: Syntax for defining parametric decorators] In-Reply-To: <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> Message-ID: <4FFE2A3C.1090905@pearwood.info> Nick Coghlan wrote: > On Mon, Jul 9, 2012 at 7:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: >> >From an innocent look, I have no idea what the syntax is supposed to mean. >> Clearly doesn't hint at a factory for me. > > I should also mention that I have a different proposal that affects > the way one would write functions-that-returns-functions. I've been > messing around with the idea of statement local namespaces for years > (see PEP 3150) trying to find something that I consider better than > the status quo, and PEP 403's statement local function and class > definitions (http://www.python.org/dev/peps/pep-0403/) are the current > incarnation. > > With those, the number of statements in a simple wrapping decorator > factory doesn't change, but the return statements can be moved above > their respective function definitions: > > def notify_on_call(callback, *cb_args, **cb_kwds): > in return decorator > def decorator(f): > in return wrapped > @functools.wraps(f) > def wrapped(*args, **kwds): > callback(*cb_args, cb_kwds) > return f(*args, **kwds) I *really* don't like the way that the "in return" statement reads like it creates a new block, but the following lines are not indented. If I were writing this as pseudo-code for a human reader, I would surely indent the following lines. Even if the part following the "in return" were limited to a single expression, I would prefer to indent it if it appears on another line. Using lambda as an example: # Best lambda x: expression # Acceptable lambda x:\ expression # Unacceptable lambda x:\ expression Looking back at "in return", here's a simple example which doesn't use nested functions. Compare: def function(arg): in return value value = process(arg) print("Value is", value) versus: def function(arg, verbose=False): in return value: value = process(arg) print("Value is", value) The lack of indentation (and trailing colon) makes the first extremely unpythonic -- everything else that creates a new block is indented. Consequently I hate the first one and am merely cold to the second. > Rather than the current out-of-order: > > def notify_on_call(callback, *cb_args, **cb_kwds): > def decorator(f): > @functools.wraps(f) > def wrapped(*args, **kwds): > callback(*cb_args, cb_kwds) > return f(*args, **kwds) > return wrapped > return decorator I would not describe that as "out-of-order". Seems to me that it is precisely in order: first you create the object (a function), then you return it. You can't return something before it exists. It seems to me that this proposal, and the older PEP 3150, are the ones which are out-of-order: you use things before they are defined. For what it's worth, I have slightly warmed to PEP 3150 and would give it a very tentative +0.125: def notify_on_call(callback, *cb_args, **cb_kwds): return decorator given: def decorator(f): @functools.wraps(f) def wrapped(*args, **kwds): callback(*cb_args, cb_kwds) return f(*args, **kwds) return wrapped I don't think the above is any improvement at all on the status quo, but 3150 would allow you to write maths expressions more mathematically: def func(x): return a**2 + 3*a given: a = 1/(sin(x*pi)) + (cos(x)-1)/2 which I'm not sure will help people used to reading code, but it should at least be familiar territory to mathematicians and maths geeks. (Aside: I actually prefer that bikeshed to be called "where" rather than "given".) -- Steven From tjreedy at udel.edu Thu Jul 12 04:19:01 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 11 Jul 2012 22:19:01 -0400 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> Message-ID: <jtlc79$195$1@dough.gmane.org> On 7/8/2012 4:22 PM, Mike Graham wrote: > A common stumbling block for new users is writing decorators that take > arguments. To create a decorator like > > @timesn(n) > def f(y): > ... > > We write code like > > def timesn(n) > def decorator(f): > def inner(y): > return n * f(y) > return inner > return decorator > > which confuses many users and can be a handful to type. I wonder if it > would be clearer for people to write > > def timesn(n)(f): > def inner(y): > return n * f(y) > return inner > > which is more concise and looks a lot more like a non-parametric > decorator someone might have written already. The syntax is mostly > self-explaining and could potentially be useful in other contexts. As others have noted, this is a proposal for introducing optional automatic function currying to python. Parametric decorators are an obvious use case, because they require double function nesting, but neither the interpreter not the function, such as timesn, would know or care how the function will be used. If you are really concerned specifically about decorator expressions, I think someone suggestion of a new decorator function in functools would be the better way to go. For people 'raised' with imperative languages without nested functions and currying, function currying is definately *not* 'self-explaining'. The difficultly is conceptual, not syntactical. I am 99% sure it would result in more confusion, not less. > There exist tools like the decorator library to try to simplify this > already, but in my experience they mostly serve to confuse people > using decorators for the first time more. > > One thing I didn't specify was whether `n` was nonlocal or not and the > behavior of something that keeps and reuses timesn(some_specific_n) > multiple times. > > Does anyone think a feature like this may be useful? Just about every proposal would be 'useful' to someone in some situations. Is it useful enough to justify the costs? My brief answer is that the syntactic sugar of automatic currying only applies to a few situations and does not add any new functionality. It would be another thing to learn, which most people would not get until they learned the more general explicit nesting. For some people, say from always-curried languages, it might be an attractive nuisance in that they would use it and incur the added overhead when it is pointless. -- Terry Jan Reedy From ncoghlan at gmail.com Thu Jul 12 05:49:14 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 Jul 2012 13:49:14 +1000 Subject: [Python-ideas] Moving return above statements [was Re: Syntax for defining parametric decorators] In-Reply-To: <4FFE2A3C.1090905@pearwood.info> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> <4FFE2A3C.1090905@pearwood.info> Message-ID: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info> wrote: > Looking back at "in return", here's a simple example which doesn't use > nested functions. Compare: > > > def function(arg): > in return value > value = process(arg) > print("Value is", value) Like a decorator line, the "in" clause is effectively part of the subsequent function/class definition - it's not a separate statement. That's why there's no ending colon. However, I'm now thinking a leading @ might be appropriate in order to make the decorator parallel crystal clear. > I would not describe that as "out-of-order". Seems to me that it is > precisely in order: first you create the object (a function), then you > return it. You can't return something before it exists. No, that's implementation order - it's not "pattern of thought" order. It's why English permits phrase like "this statement is true, given that this particular assumption is true". You can forward reference incidental details and fill them in later. Sorting is generally a better example: x = sorted(container) OK, this is sorted smallest to largest. Now, I want to sort according to the second item. I can do this out of order by introducing a key function before it's clear why I need it: def _key(v): return v[1] x = sorted(container, key=_key) Or, we can use the awkward itemgetter construct to get things back in the right order and restore the assignment and sorting operation to it's rightful place of prominence: from operator import itemgetter x = sorted(container, key=itemgetter(1)) Using a lambda expression achieves the same goal: x = sorted(container, key=lambda v: v[1]) Those are both ugly though, and are restricted to a single expression. More complicated sorting logic is forced back into using the out of order form. PEP 403 is designed to let you pull the sort key out into a trailing full-fledged function definition: @in x = sorted(container, key=f) def f(v): return v[1] The important part here is that the container is being sorted and the result assigned to 'x' - the exact sort key used isn't important to the overall flow of the algorithm. > It seems to me that this proposal, and the older PEP 3150, are the ones > which are out-of-order: you use things before they are defined. Yes, both PEP 403 and 3150 are out of order from a code *execution* point of view - but they're designed to match *patterns of thought* that work that way. It's especially prevalent in callback based programming - you end up with a lot of one-shot functions that aren't actually all that important in their own right, what really matters is the call you're handing them over to. Sometimes you're only defining a separate function because that's the only syntax Python has for passing a piece of code to a function, even though the code is only used once. Decorators eliminated this problem for the "f = deco(f)" and "C = deco(C)" cases by baking the ability to make such calls into function and class definitions. PEP 403 is solely about extending that same capability to arbitrary simple statements. > For what it's worth, I have slightly warmed to PEP 3150 and would give it a > very tentative +0.125: PEP 3150 died mainly due to "If the implementation is hard to explain, it's a bad idea", but also due to the fact that you regularly ended up with double-indents for callback based code. PEP 403 only has to a support a statement local reference to a single name, which is comparatively easy. > (Aside: I actually prefer that bikeshed to be called "where" rather than > "given".) PEP 3150 has an entire section devoted to the reasons "where" isn't a viable choice (not that that PEP stands any realistic chance of ever being resurrected). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From raymond.hettinger at gmail.com Thu Jul 12 05:19:20 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 11 Jul 2012 20:19:20 -0700 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <jtlc79$195$1@dough.gmane.org> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jtlc79$195$1@dough.gmane.org> Message-ID: <6343E5B9-66E7-44B4-B6AE-D16D1B064C56@gmail.com> On Jul 11, 2012, at 7:19 PM, Terry Reedy wrote: > For people 'raised' with imperative languages without nested functions and currying, function currying is definately *not* 'self-explaining'. The difficultly is conceptual, not syntactical. I am 99% sure it would result in more confusion, not less. FWIW, I've taught many Python classes this year and have found the current style easy to teach. I agree with Terry's assessment that introducing new syntactic trickery would cause more problems than it would solve. I also agree with Nick's sentiment that we would be better-off with a vocabulary that clearly distinguishes between between decorators and decorator factories. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120711/0fb7e1c9/attachment.html> From ubershmekel at gmail.com Thu Jul 12 07:23:22 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 12 Jul 2012 08:23:22 +0300 Subject: [Python-ideas] Moving return above statements [was Re: Syntax for defining parametric decorators] In-Reply-To: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> <4FFE2A3C.1090905@pearwood.info> <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> Message-ID: <CANSw7KxAfTFBrVdpQNdqamM0xLLtsCacj851OAA=ezVeh_i7EQ@mail.gmail.com> On Thu, Jul 12, 2012 at 6:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info> > wrote: > > Looking back at "in return", here's a simple example which doesn't use > > nested functions. Compare: > > > > > > def function(arg): > > in return value > > value = process(arg) > > print("Value is", value) > > Like a decorator line, the "in" clause is effectively part of the > subsequent function/class definition - it's not a separate statement. > That's why there's no ending colon. However, I'm now thinking a > leading @ might be appropriate in order to make the decorator parallel > crystal clear. > > > I would not describe that as "out-of-order". Seems to me that it is > > precisely in order: first you create the object (a function), then you > > return it. You can't return something before it exists. > > No, that's implementation order - it's not "pattern of thought" order. > It's why English permits phrase like "this statement is true, given > that this particular assumption is true". You can forward reference > incidental details and fill them in later. > > Sorting is generally a better example: > > x = sorted(container) > > OK, this is sorted smallest to largest. > > Now, I want to sort according to the second item. I can do this out of > order by introducing a key function before it's clear why I need it: > > def _key(v): > return v[1] > x = sorted(container, key=_key) > > Or, we can use the awkward itemgetter construct to get things back in > the right order and restore the assignment and sorting operation to > it's rightful place of prominence: > > from operator import itemgetter > x = sorted(container, key=itemgetter(1)) > > Using a lambda expression achieves the same goal: > > x = sorted(container, key=lambda v: v[1]) > > Those are both ugly though, and are restricted to a single expression. > More complicated sorting logic is forced back into using the out of > order form. > > PEP 403 is designed to let you pull the sort key out into a trailing > full-fledged function definition: > > @in x = sorted(container, key=f) > def f(v): > return v[1] > > The important part here is that the container is being sorted and the > result assigned to 'x' - the exact sort key used isn't important to > the overall flow of the algorithm. > > > > It seems to me that this proposal, and the older PEP 3150, are the ones > > which are out-of-order: you use things before they are defined. > > Yes, both PEP 403 and 3150 are out of order from a code *execution* > point of view - but they're designed to match *patterns of thought* > that work that way. It's especially prevalent in callback based > programming - you end up with a lot of one-shot functions that aren't > actually all that important in their own right, what really matters is > the call you're handing them over to. > > Sometimes you're only defining a separate function because that's the > only syntax Python has for passing a piece of code to a function, even > though the code is only used once. > > Decorators eliminated this problem for the "f = deco(f)" and "C = > deco(C)" cases by baking the ability to make such calls into function > and class definitions. PEP 403 is solely about extending that same > capability to arbitrary simple statements. > > > For what it's worth, I have slightly warmed to PEP 3150 and would give > it a > > very tentative +0.125: > > PEP 3150 died mainly due to "If the implementation is hard to explain, > it's a bad idea", but also due to the fact that you regularly ended up > with double-indents for callback based code. PEP 403 only has to a > support a statement local reference to a single name, which is > comparatively easy. > > > (Aside: I actually prefer that bikeshed to be called "where" rather than > > "given".) > > PEP 3150 has an entire section devoted to the reasons "where" isn't a > viable choice (not that that PEP stands any realistic chance of ever > being resurrected). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > The way I understand this proposal is similar to how MATLAB defines functions where the names of the variables that are returned are in/near the function declaration. It's useful on the occasions where you decide another variable is needed by the caller. Also, it's nice to have consistency in whatever's returned. The "return" keyword would be strange though. There are a few unobvious edge cases here. So I'd have to say -0.1 Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120712/5f940d97/attachment.html> From steve at pearwood.info Thu Jul 12 09:05:02 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 12 Jul 2012 17:05:02 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <jtlc79$195$1@dough.gmane.org> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jtlc79$195$1@dough.gmane.org> Message-ID: <20120712070502.GA29188@ando> On Wed, Jul 11, 2012 at 10:19:01PM -0400, Terry Reedy wrote: > For people 'raised' with imperative languages without nested functions > and currying, function currying is definately *not* 'self-explaining'. > The difficultly is conceptual, not syntactical. I am 99% sure it would > result in more confusion, not less. I was raised on Pascal, Fortran and Hypertalk, with some Forth-like languages (Forth and HP's RPL). None of which are functional, and I had never even heard of currying until *long* after starting with Python. But the idea of a function creating and returning a function was easy to understand once it was suggested to me. Once you have the right language to talk about a concept, some pretty complex concepts become simple. I suspect that most of the difficulty people have is because they don't clearly distinguish between a decorator and a decorator-factory. Once you make the conceptual leap to a function making a function (a decorator, which is a kind of function-factory), it is easy to think about a decorator-factory, a decorator-factory-factory, and so on. But if you don't cleanly distinguish them, in my opinion it becomes confusing and unclear whether you need one inner function or two. Of course, the book-keeping needed to make this all work is not necessarily trivial, but it's just book-keeping. New syntax won't make that easier. Singling out decorator-factories for special treatment is, in my opinion, a no-go. If this idea is to be sold, decorator-factories may be given as just one use-case for syntax for currying. But frankly, I don't think we need it, or should want it. Python is not Haskell, and while it's great to borrow ideas from Haskell (e.g. list comprehensions), it is notable that Python hasn't used the exact same syntax as Haskell. Compare: [toUpper c | c <- s] -- Haskell [c.upper() for c in s] # Python and tell me that "for c in s" isn't more readable than "| c <- s". I think syntax for currying def func(a)(b)(c)(d): ... f = func(1)(2)(3) f(4) is too Haskell-ish for my taste. I prefer the Python way: def func(a, b, c, d): ... f = functools.partial(func, 1, 2, 3) f(4) even though it is less concise. I prefer a stronger tool set in functools, including a way to use partial on arguments from the right. Perhaps there is a clean API for simplifying decorator factories. That could start as a published recipe on (say) ActiveState, and then move to functools if there was demand for it. But as usual, the bar to justify new syntax is rightly set much higher than the bar to justify new functionality. -- Steven From ncoghlan at gmail.com Thu Jul 12 09:17:06 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 Jul 2012 17:17:06 +1000 Subject: [Python-ideas] Syntax for defining parametric decorators In-Reply-To: <20120712070502.GA29188@ando> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jtlc79$195$1@dough.gmane.org> <20120712070502.GA29188@ando> Message-ID: <CADiSq7e63BcXKtt+JseEbiqgYf_jLRwbRDwb+Sx8LZugGpKc5A@mail.gmail.com> On Thu, Jul 12, 2012 at 5:05 PM, Steven D'Aprano <steve at pearwood.info> wrote: > I prefer a stronger tool set in functools, including a way to use > partial on arguments from the right. FWIW, my hope is that the provision of PEP 362 in Python 3.3 will encourage third parties to explore some richer options for manipulating callables now that it's much easier to get a clear definition of their signature. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Thu Jul 12 11:32:45 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 12 Jul 2012 18:32:45 +0900 Subject: [Python-ideas] Moving return above statements [was Re: Syntax for defining parametric decorators] In-Reply-To: <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> <4FFE2A3C.1090905@pearwood.info> <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> Message-ID: <87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > On Thu, Jul 12, 2012 at 11:37 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > Looking back at "in return", here's a simple example which doesn't use > > nested functions. Compare: > > > > > > def function(arg): > > in return value > > value = process(arg) > > print("Value is", value) > > Like a decorator line, the "in" clause is effectively part of the > subsequent function/class definition - it's not a separate statement. > That's why there's no ending colon. However, I'm now thinking a > leading @ might be appropriate in order to make the decorator parallel > crystal clear. Indeed I would prefer the leading "@", because the in clause (syntactically) "decorates" the following definition. But isn't Steven's example just plain invalid on two counts? That is, in the PEP (1) the "in" modifies a function or class definition, not an arbitrary statement, and (2) it does not have suite scope, it has statement scope. Another way of expressing (2) is that the print statement above is dead code. With respect to the idea itself, I'm a little concerned that in the case of the decorator, the idea is to deemphasize the decoration in favor of focusing on the decorated definition, while in the case of the "in" clause, the idea is to deemphasize the definition in favor of focusing on the "decorating" statement. Perhaps in practice this won't be a problem, though. For the "in" clause, the order of presentation will provide the emphasis on the "decorating" statement. OTOH, for the decorators, the order of presentation is no big deal compared to the avoidance of DRY violation. From ncoghlan at gmail.com Thu Jul 12 11:39:44 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 Jul 2012 19:39:44 +1000 Subject: [Python-ideas] Moving return above statements [was Re: Syntax for defining parametric decorators] In-Reply-To: <87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CAEBZo3O+hhsKo1qscPOziFN2ekPbC3sgSh2WnMFjEn9PGFCOpA@mail.gmail.com> <jte9c4$2ej$1@dough.gmane.org> <CADiSq7cPhNwru3_PUtuHbKzzd8tdAM8wD0bQ0Bnqc=+r5NEkcw@mail.gmail.com> <4FFE2A3C.1090905@pearwood.info> <CADiSq7cduOWurk6SGcfJnJxnd-1SbHG66+GWs9CoAZyjPnn+dA@mail.gmail.com> <87r4sh494y.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CADiSq7cRbuWLhkLMwCNQgCquRu1OzfAeMfOzrt5=6i6r-DoSiw@mail.gmail.com> On Thu, Jul 12, 2012 at 7:32 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > With respect to the idea itself, I'm a little concerned that in the > case of the decorator, the idea is to deemphasize the decoration in > favor of focusing on the decorated definition, while in the case of > the "in" clause, the idea is to deemphasize the definition in favor of > focusing on the "decorating" statement. There are many reasons PEP 403 is something I pick up and improve as the urge strikes me rather than something I treat with any urgency :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From namdn at socbay.com Sun Jul 15 08:18:59 2012 From: namdn at socbay.com (=?utf-8?B?xJBpbmg=?= Nho Nam) Date: Sun, 15 Jul 2012 13:18:59 +0700 (ICT) Subject: [Python-ideas] How can I get own instance of generator in itself code Message-ID: <33429676.215905.1342333139344.JavaMail.root@socbay.com> Hi Python Software Foundation! I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? detail pseudo code example: def invoke_callback(url): do something here self = get myself (invoke_callback generator) instance #how can we do as this line???? data = yield asynchronous_dowload(url, callback = self.next) do something with data How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list. def invoke_callback(url, wrapper): self = wrapper[0] data = yield asynchronous_dowload(url, callback = self.next) do something with data wrapper = [] instance = invoke_callback(url, wrapper) wrapper.append(instance) instance.next() The other way is sending instance to itself, and we invoke twice. def invoke_callback(url): self = yield data = yield asynchronous_dowload(url, callback = self.next) do something with data instance = invoke_callback(url) instance.next() #first instance.send(instance) #second But the code in both of ways is not nice. Can you add this feature in the next version of python??? Regards Namdn From simon.sapin at kozea.fr Sun Jul 15 11:36:27 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Sun, 15 Jul 2012 11:36:27 +0200 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> Message-ID: <50028F1B.1070105@kozea.fr> Le 15/07/2012 08:18, ?inh Nho Nam a ?crit : > def invoke_callback(url): > do something here > self = get myself (invoke_callback generator) instance #how can we do as this line???? > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > > How can we do as this code. How can I get self variable?. Does the > language support this feature? I cannot find it in any document in > python.org. So, I have to solve it by wrapper it in a list. Hi, Maybe you could achieve that by inspecting the stack frame, but this is generally not recommended. But taking a step back, I think that the primitive you are looking for is a more general coroutine. greenlet can do this, and gevent builds on it to add higher level async networking: http://greenlet.readthedocs.org/en/latest/index.html http://www.gevent.org/ Regards, -- Simon Sapin From grosser.meister.morti at gmx.net Sun Jul 15 15:42:09 2012 From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=) Date: Sun, 15 Jul 2012 15:42:09 +0200 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> Message-ID: <5002C8B1.5030006@gmx.net> On 07/15/2012 08:18 AM, ?inh Nho Nam wrote: > I think that this feature is not supported in classic language as C++, Java, C#.. C# and Mozillas non-standard JavaScript also support generators (maybe not called "generators") and the yield statement. From alexandre.zani at gmail.com Sun Jul 15 17:58:10 2012 From: alexandre.zani at gmail.com (Alexandre Zani) Date: Sun, 15 Jul 2012 08:58:10 -0700 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> Message-ID: <CAJVMporuyk=HVS1cfarU91WfBEmH1nVuX6WcOS2VZxf6q8Hxog@mail.gmail.com> On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote: > Hi Python Software Foundation! > I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. > Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? > detail pseudo code example: > > def invoke_callback(url): > do something here > self = get myself (invoke_callback generator) instance #how can we do as this line???? > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > > How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list. > > def invoke_callback(url, wrapper): > self = wrapper[0] > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > > wrapper = [] > instance = invoke_callback(url, wrapper) > wrapper.append(instance) > instance.next() > > The other way is sending instance to itself, and we invoke twice. > > def invoke_callback(url): > self = yield > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > instance = invoke_callback(url) > instance.next() #first > instance.send(instance) #second > > But the code in both of ways is not nice. Can you add this feature in the next version of python??? > Regards > Namdn > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas I would suggest creating a decorator for this: def self_decorator(f): @functools.wraps(f) def decorated(*args, **kwargs): return f(f, *args, **kwargs) @self_decorator def do_stuff(self, a, b, c): pass Should work for any kind of function. And C/C++ do support such functionality in the sense that you can keep state in between function invocations: // This will return 1 the first time it is called, 2, the next time and so on... int generator() { static int num = 0; num += 1; return num; } On a final note, this question more appropriate for python-list. python-ideas is for the discussion of ideas to change python itself. Alex From pyideas at rebertia.com Mon Jul 16 04:03:54 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 15 Jul 2012 19:03:54 -0700 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <33429676.215905.1342333139344.JavaMail.root@socbay.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> Message-ID: <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote: > Hi Python Software Foundation! > I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. > Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? <snip> > How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. <snip> > Can you add this feature in the next version of python??? Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which would have similarly added nice ways to get the current module, class, and regular function, was rejected, I wouldn't hold out much hope for such a feature being added any time soon. (Wrapping via decorators introduces nontrivial subtlety regarding whether you really want to refer to the decorated or undecorated version of an entity.) Cheers, Chris From aquavitae69 at gmail.com Mon Jul 16 09:12:32 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 16 Jul 2012 09:12:32 +0200 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> Message-ID: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> The problem with PEP 3130 was the PEP itself, not the idea. I believe Eric Snow was working on a patch to implement this (see http://code.activestate.com/lists/python-ideas/11234/ and http://bugs.python.org/issue12857), but I'm not sure what its current status is. David On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote: > On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote: > > Hi Python Software Foundation! > > I have learned Python for 2 years, and I love this language very much. > The structure and grammar are simple and beautiful. I used this language in > most of my projects. > > Now, I learn to use generator. This architecture is very intelligent. It > allows programmer can pause and resume function. I think that this feature > is not supported in classic language as C++, Java, C#.. In my design, we > use 1 generator in both tasks: invoke and callback when download one url. > But I have a small problem when getting instance of generator. How can I > get own instance of generator in itself code? > <snip> > > How can we do as this code. How can I get self variable?. Does the > language support this feature? I cannot find it in any document in > python.org. > <snip> > > Can you add this feature in the next version of python??? > > Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which > would have similarly added nice ways to get the current module, class, > and regular function, was rejected, I wouldn't hold out much hope for > such a feature being added any time soon. > (Wrapping via decorators introduces nontrivial subtlety regarding > whether you really want to refer to the decorated or undecorated > version of an entity.) > > Cheers, > Chris > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/05d7206f/attachment.html> From michelelacchia at gmail.com Mon Jul 16 10:55:12 2012 From: michelelacchia at gmail.com (Michele Lacchia) Date: Mon, 16 Jul 2012 01:55:12 -0700 (PDT) Subject: [Python-ideas] Where has packaging gone? Message-ID: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com> Hi, I read this: http://www.gossamer-threads.com/lists/python/dev/992979 but at the end nobody posted to it anymore. I would like to know what will now happen, since as far as I understood, the development of packaging has stopped and Bento (David Cournapeau library) has appeared. So what will happen now? Packaging will continue the development? Or will be Bento replace it? Personally I don't care which tool will become standard, but I do feel the need for a standard to be created. So I would like to point out that to me these three PEPs seem really important, and hope that will be taken into account: - PEP 345 (http://www.python.org/dev/peps/pep-0345/) - PEP 376 (http://www.python.org/dev/peps/pep-0376/) - PEP 386 (http://www.python.org/dev/peps/pep-0386/) What do you think about all this? Regards, Michele Lacchia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/25064587/attachment.html> From ncoghlan at gmail.com Mon Jul 16 14:08:30 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Jul 2012 22:08:30 +1000 Subject: [Python-ideas] Where has packaging gone? In-Reply-To: <CADiSq7dAyxfA36FaW=GHy56En9udeir1MR+EDWGRcRviY+U4Hg@mail.gmail.com> References: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com> <CADiSq7dAyxfA36FaW=GHy56En9udeir1MR+EDWGRcRviY+U4Hg@mail.gmail.com> Message-ID: <CADiSq7d9r5OotU-Y=Ki_dG9pfTH56cFf=m_y4cF+DDmAYCpWNw@mail.gmail.com> (the list address on the original email I replied to was broken) On Mon, Jul 16, 2012 at 6:55 PM, Michele Lacchia <michelelacchia at gmail.com> wrote: > Hi, > I read this: > http://www.gossamer-threads.com/lists/python/dev/992979 > > but at the end nobody posted to it anymore. I would like to know what will > now happen, since as far as I understood, the development of packaging > has stopped and Bento (David Cournapeau library) has appeared. No, the distutils2 project continues and is still being actively developed. (distutils2 *is* packaging - the name on PyPI is just different from the name that will be used when it is added to the standard library). Bento is a project that aims to tackle part of distutils that distutils2 hasn't really looked at yet: creating a more flexible build system (or, more accurately, making it easier to hook into *existing* build systems rather than defining yet another one). (In fact, the lack of attention to the build system flaws is one of the big reasons distutils2/packaging was deemed unready for stdlib inclusion in 3.3) > So what will happen now? Packaging will continue the development? Or > will be Bento replace it? Personally I don't care which tool will become > standard, > but I do feel the need for a standard to be created. The point of having documented standards in PEP form is that there can (and will) be multiple projects that produce and consume those standard formats. > - PEP 345 (http://www.python.org/dev/peps/pep-0345/) > - PEP 376 (http://www.python.org/dev/peps/pep-0376/) > - PEP 386 (http://www.python.org/dev/peps/pep-0386/) Those PEPs are *already* standards and are actively supported by external projects today. One key missing piece is standard library support, which the packaging module is intended to provide. However, it was deemed that the project was not quite mature enough for 3.3, thus it will continue to be developed (with a faster iteration cycle) as distutils2, with a view to including a more mature version of the API for 3.4. This will likely take the form of additional PEPs that more clearly define what is to be added and *why* it is being added, with one focus being making it easier to build additional tools that produce and consume metadata in the standard formats, rather then being a comparatively unwieldy monolithic system like the original distutils. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From sven at marnach.net Mon Jul 16 16:49:52 2012 From: sven at marnach.net (Sven Marnach) Date: Mon, 16 Jul 2012 15:49:52 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code Message-ID: <20120716144952.GK14370@bagheera> Currently, there is no obvious way to encode a filename in the default filesystem encoding. To pipe some filenames to the stdin of a subprocess, I effectively used encoded_name = file_name.encode(sys.getfilesystemencoding()) which mostly worked. There are cases where this fails, though: on Linux with LANG=C and filenames that contain non-ASCII characters, for example, or in any situation where the default filesystem encoding can't decode a filename. The correct way to do this seems to be something like if sys.platform == "nt": errors = "strict" else: errors = "surrogateescape" encoded_name = file_name.encode(sys.getfilesystemencoding() errors=errors) I think there should be (1) some documentation on the issue and (2) a more obvious way to do encode filenames. 1. The most useful reference I could find in the docs is http://docs.python.org/dev/c-api/unicode.html#file-system-encoding and there is a short paragraph at http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables The filename encoding applies to basically all Python library functions (including built-ins like `open()`) and should probably be documented at a more prominent spot. The "surrogateescape" error handler isn't mentioned here http://docs.python.org/dev/howto/unicode.html#unicode-filenames 2. There should be some way to access the C API functions for decoding and encoding filenames from Python. I don't have a good idea how to do this ? maybe by adding a meta-encoding "filesystem", or by adding functions to the standard library. Did I miss something? Any thoughts? Cheers, Sven From solipsis at pitrou.net Mon Jul 16 17:49:56 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Jul 2012 17:49:56 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code References: <20120716144952.GK14370@bagheera> Message-ID: <20120716174956.162a2df2@pitrou.net> On Mon, 16 Jul 2012 15:49:52 +0100 Sven Marnach <sven at marnach.net> wrote: > Currently, there is no obvious way to encode a filename in the default > filesystem encoding. To pipe some filenames to the stdin of a > subprocess, I effectively used > > encoded_name = file_name.encode(sys.getfilesystemencoding()) Well, how about os.fsencode() and os.fsdecode()? http://docs.python.org/dev/library/os.html#os.fsencode Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From ericsnowcurrently at gmail.com Mon Jul 16 18:28:48 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 16 Jul 2012 10:28:48 -0600 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> Message-ID: <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com> Yeah, this is still valuable. However, at this point it would be a 3.4 addition (_if_ it gets committed). Updating the patch in the tracker is on my todo for the next few months. -eric On Jul 16, 2012 1:13 AM, "David Townshend" <aquavitae69 at gmail.com> wrote: > The problem with PEP 3130 was the PEP itself, not the idea. I believe > Eric Snow was working on a patch to implement this (see > http://code.activestate.com/lists/python-ideas/11234/ and > http://bugs.python.org/issue12857), but I'm not sure what its current > status is. > > David > On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote: > >> On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote: >> > Hi Python Software Foundation! >> > I have learned Python for 2 years, and I love this language very much. >> The structure and grammar are simple and beautiful. I used this language in >> most of my projects. >> > Now, I learn to use generator. This architecture is very intelligent. >> It allows programmer can pause and resume function. I think that this >> feature is not supported in classic language as C++, Java, C#.. In my >> design, we use 1 generator in both tasks: invoke and callback when download >> one url. But I have a small problem when getting instance of generator. How >> can I get own instance of generator in itself code? >> <snip> >> > How can we do as this code. How can I get self variable?. Does the >> language support this feature? I cannot find it in any document in >> python.org. >> <snip> >> > Can you add this feature in the next version of python??? >> >> Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which >> would have similarly added nice ways to get the current module, class, >> and regular function, was rejected, I wouldn't hold out much hope for >> such a feature being added any time soon. >> (Wrapping via decorators introduces nontrivial subtlety regarding >> whether you really want to refer to the decorated or undecorated >> version of an entity.) >> >> Cheers, >> Chris >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120716/1e714c3b/attachment.html> From sven at marnach.net Mon Jul 16 19:04:41 2012 From: sven at marnach.net (Sven Marnach) Date: Mon, 16 Jul 2012 18:04:41 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <20120716174956.162a2df2@pitrou.net> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> Message-ID: <20120716170441.GL14370@bagheera> Antoine Pitrou schrieb am Mon, 16. Jul 2012, um 17:49:56 +0200: > On Mon, 16 Jul 2012 15:49:52 +0100 > Sven Marnach <sven at marnach.net> wrote: > > Currently, there is no obvious way to encode a filename in the default > > filesystem encoding. To pipe some filenames to the stdin of a > > subprocess, I effectively used > > > > encoded_name = file_name.encode(sys.getfilesystemencoding()) > > Well, how about os.fsencode() and os.fsdecode()? > > http://docs.python.org/dev/library/os.html#os.fsencode Oh, great, there they are! I think these functions should be mentioned in these sections to make them easier to find: [1]: http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables [2]: http://docs.python.org/dev/library/sys.html#sys.getfilesystemencoding [3]: http://docs.python.org/dev/howto/unicode.html#unicode-filenames I'll post an issue on the issue tracker. Cheers, Sven From victor.stinner at gmail.com Mon Jul 16 19:23:59 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Jul 2012 19:23:59 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <20120716170441.GL14370@bagheera> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> Message-ID: <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> >> Well, how about os.fsencode() and os.fsdecode()? >> >> http://docs.python.org/dev/library/os.html#os.fsencode > > Oh, great, there they are! I think these functions should be > mentioned in these sections to make them easier to find: > > [1]: http://docs.python.org/dev/library/os.html#file-names-command-line-arguments-and-environment-variables > [2]: http://docs.python.org/dev/library/sys.html#sys.getfilesystemencoding > [3]: http://docs.python.org/dev/howto/unicode.html#unicode-filenames > > I'll post an issue on the issue tracker. Hi, I wrote these functions when I worked in this topic for Python 3. Yes, it would be great if you write a patch to mention these functions in the doc. Someone also complained that the surrogateescape error handler is not mentionned in any FS related function. Victor From and-dev at doxdesk.com Tue Jul 17 01:00:32 2012 From: and-dev at doxdesk.com (And Clover) Date: Tue, 17 Jul 2012 00:00:32 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> Message-ID: <50049D10.5040903@doxdesk.com> On 16/07/12 18:23, Victor Stinner wrote: > I wrote these functions when I worked in this topic for Python 3. Yes, > it would be great if you write a patch to mention these functions in > the doc. Sure. But should we be encouraging their use on Windows? I would have thought it the best thing to stick with the Unicode string for paths on NT, so that the native Win32 Unicode APIs are used instead of the ANSI-code-page-bound C stdio. Encoding down to the fsencoding for Windows just means that any path including a character that isn't in the ANSI CP will fail. In lieu of some kind of abstract filepath object thatcould represent either bytes or str (depending on platform), how about a function that takes a str and only encodes it to bytes if the platform requires it? cheers, -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ gtalk:chat?jid=bobince at gmail.com From solipsis at pitrou.net Tue Jul 17 01:41:24 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Jul 2012 01:41:24 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> Message-ID: <20120717014124.2e824d59@pitrou.net> On Tue, 17 Jul 2012 00:00:32 +0100 And Clover <and-dev at doxdesk.com> wrote: > On 16/07/12 18:23, Victor Stinner wrote: > > I wrote these functions when I worked in this topic for Python 3. Yes, > > it would be great if you write a patch to mention these functions in > > the doc. > > Sure. > > But should we be encouraging their use on Windows? I would have thought > it the best thing to stick with the Unicode string for paths on NT, so > that the native Win32 Unicode APIs are used instead of the > ANSI-code-page-bound C stdio. Encoding down to the fsencoding for > Windows just means that any path including a character that isn't in the > ANSI CP will fail. Well even under Unix, these functions are only useful for very specialized cases. For normal usage, PEP 383 guarantees that all filenames, including theoretically undecodable ones, pass through properly. When piping filenames between Python processes, you can use whatever encoding you want (or you can also use json or pickle). The only remaining use case is sending some filenames to an external (non-Python) program over a bytes stream, or reading some filenames emitted by such a program. Here, you need bytes under Windows as well. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From victor.stinner at gmail.com Tue Jul 17 03:03:24 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Jul 2012 03:03:24 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <50049D10.5040903@doxdesk.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> Message-ID: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> 2012/7/17 And Clover <and-dev at doxdesk.com>: > But should we be encouraging their use on Windows? I would have thought it > the best thing to stick with the Unicode string for paths on NT, so that the > native Win32 Unicode APIs are used instead of the ANSI-code-page-bound C > stdio. Encoding down to the fsencoding for Windows just means that any path > including a character that isn't in the ANSI CP will fail. os.fsencode() should not be used explicitly on Windows. > In lieu of some kind of abstract filepath object thatcould represent either > bytes or str (depending on platform), how about a function that takes a str > and only encodes it to bytes if the platform requires it? You can use the str (Unicode) type on all platforms with Python 3, so use os.fsdecode(). os.listdir(str) does return str filenames on any platform for example. Victor From stefan_ml at behnel.de Tue Jul 17 06:57:43 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Jul 2012 06:57:43 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> Message-ID: <ju2rc7$omg$1@dough.gmane.org> Victor Stinner, 17.07.2012 03:03: > 2012/7/17 And Clover: >> But should we be encouraging their use on Windows? I would have thought it >> the best thing to stick with the Unicode string for paths on NT, so that the >> native Win32 Unicode APIs are used instead of the ANSI-code-page-bound C >> stdio. Encoding down to the fsencoding for Windows just means that any path >> including a character that isn't in the ANSI CP will fail. > > os.fsencode() should not be used explicitly on Windows. > >> In lieu of some kind of abstract filepath object thatcould represent either >> bytes or str (depending on platform), how about a function that takes a str >> and only encodes it to bytes if the platform requires it? > > You can use the str (Unicode) type on all platforms with Python 3, so > use os.fsdecode(). os.listdir(str) does return str filenames on any > platform for example. That's not the main use case I see, though. When talking to C libraries, for example, they will usually require a byte encoded file path and also return one. Getting the encoding right in this case is really not trivial. I would expect that the above functions do "the right thing" also on Windows here, unless the library really has a win32 specific file API (and that's not likely). Stefan From breamoreboy at yahoo.co.uk Tue Jul 17 15:20:57 2012 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 17 Jul 2012 14:20:57 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> Message-ID: <ju3or0$rrj$1@dough.gmane.org> On 17/07/2012 02:03, Victor Stinner wrote: > > os.fsencode() should not be used explicitly on Windows. > > Victor > Should there be a note in the docs to this effect? -- Cheers. Mark Lawrence. From ned at nedbatchelder.com Tue Jul 17 15:59:32 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 17 Jul 2012 09:59:32 -0400 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <20120716174956.162a2df2@pitrou.net> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> Message-ID: <50056FC4.6040305@nedbatchelder.com> On 7/16/2012 11:49 AM, Antoine Pitrou wrote: > On Mon, 16 Jul 2012 15:49:52 +0100 > Sven Marnach <sven at marnach.net> wrote: >> Currently, there is no obvious way to encode a filename in the default >> filesystem encoding. To pipe some filenames to the stdin of a >> subprocess, I effectively used >> >> encoded_name = file_name.encode(sys.getfilesystemencoding()) > Well, how about os.fsencode() and os.fsdecode()? > > http://docs.python.org/dev/library/os.html#os.fsencode It's too bad these are not called os.path.encode() and os.path.decode(), since they fit so nicely into os.path's charter of manipulating strings representing file paths. --Ned. > Regards > > Antoine. > > From victor.stinner at gmail.com Tue Jul 17 16:31:57 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Jul 2012 16:31:57 +0200 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <50056FC4.6040305@nedbatchelder.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <50056FC4.6040305@nedbatchelder.com> Message-ID: <CAMpsgwadeOEdQTnFgxQpJBV2bX40wBsFUkO5a_cX_z6OyNDiVg@mail.gmail.com> >> Well, how about os.fsencode() and os.fsdecode()? >> >> http://docs.python.org/dev/library/os.html#os.fsencode > > It's too bad these are not called os.path.encode() and os.path.decode(), > since they fit so nicely into os.path's charter of manipulating strings > representing file paths. os.fsencode()/fsdecode() are not specific to filesystems: you can use these functions to encode/decode command line arguments, environment variable, text from/to a console (sys.std*), etc. The "fs" letters from the name comes from the encoding used by these functions: sys.get*filesystem*encoding(). For example, os.fsencode() used by the subprocess module and posixpath.expanduser() modules, and os.fsdecode() is used by os.get_exec_path() and shutil.rmtree(). Victor From sven at marnach.net Tue Jul 17 23:25:42 2012 From: sven at marnach.net (Sven Marnach) Date: Tue, 17 Jul 2012 22:25:42 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> Message-ID: <20120717212542.GP14370@bagheera> Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200: > os.fsencode() should not be used explicitly on Windows. What else should I do to pipe filenames to another process? At least, os.fsencode() seems to work, even with cyrillic filenames. Cheers, Sven From python at mrabarnett.plus.com Tue Jul 17 23:52:57 2012 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 Jul 2012 22:52:57 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <20120717212542.GP14370@bagheera> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> <20120717212542.GP14370@bagheera> Message-ID: <5005DEB9.3020406@mrabarnett.plus.com> On 17/07/2012 22:25, Sven Marnach wrote: > Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200: >> os.fsencode() should not be used explicitly on Windows. > > What else should I do to pipe filenames to another process? At least, > os.fsencode() seems to work, even with cyrillic filenames. > Encode to UTF-8? From namdn at socbay.com Wed Jul 18 04:39:53 2012 From: namdn at socbay.com (=?utf-8?B?xJBpbmg=?= Nho Nam) Date: Wed, 18 Jul 2012 09:39:53 +0700 (ICT) Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CAJVMporuyk=HVS1cfarU91WfBEmH1nVuX6WcOS2VZxf6q8Hxog@mail.gmail.com> Message-ID: <10274497.257110.1342579193494.JavaMail.root@socbay.com> Yes, in my project, i define a decorator to call twice as I said. The code below: def asynchronous(gen_func): class _generator(object): def __init__(self, *args, **kwargs): self.__generator = gen_func(*args, **kwargs) self.__generator.next() self.__generator.send(self.__generator) #override generator method: __iter__, next, send, ... def __iter__(self): return self.__generator def next(self): return self.__generator.next() def send(self, arg): return self.__generator.send(arg) def close(): return self.__generator.close() def __repr__(self): return repr(self.__generator) return _generator @asynchronous def invoke_callback(url): code here and we only call is as function, not need call send or next method g = invoke_callback(url) But i think the code is still not nice, because any we have to define @asynchronous in any generator. ----- Original Message ----- From: "Alexandre Zani" <alexandre.zani at gmail.com> To: "?inh Nho Nam" <namdn at socbay.com> Cc: python-ideas at python.org Sent: Sunday, July 15, 2012 10:58:10 PM Subject: Re: [Python-ideas] How can I get own instance of generator in itself code On Sat, Jul 14, 2012 at 11:18 PM, ?inh Nho Nam <namdn at socbay.com> wrote: > Hi Python Software Foundation! > I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. > Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? > detail pseudo code example: > > def invoke_callback(url): > do something here > self = get myself (invoke_callback generator) instance #how can we do as this line???? > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > > How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list. > > def invoke_callback(url, wrapper): > self = wrapper[0] > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > > wrapper = [] > instance = invoke_callback(url, wrapper) > wrapper.append(instance) > instance.next() > > The other way is sending instance to itself, and we invoke twice. > > def invoke_callback(url): > self = yield > data = yield asynchronous_dowload(url, callback = self.next) > do something with data > instance = invoke_callback(url) > instance.next() #first > instance.send(instance) #second > > But the code in both of ways is not nice. Can you add this feature in the next version of python??? > Regards > Namdn > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas I would suggest creating a decorator for this: def self_decorator(f): @functools.wraps(f) def decorated(*args, **kwargs): return f(f, *args, **kwargs) @self_decorator def do_stuff(self, a, b, c): pass Should work for any kind of function. And C/C++ do support such functionality in the sense that you can keep state in between function invocations: // This will return 1 the first time it is called, 2, the next time and so on... int generator() { static int num = 0; num += 1; return num; } On a final note, this question more appropriate for python-list. python-ideas is for the discussion of ideas to change python itself. Alex From sven at marnach.net Wed Jul 18 12:36:54 2012 From: sven at marnach.net (Sven Marnach) Date: Wed, 18 Jul 2012 11:36:54 +0100 Subject: [Python-ideas] Introduce some obvious way to encode and decode filenames from Python code In-Reply-To: <5005DEB9.3020406@mrabarnett.plus.com> References: <20120716144952.GK14370@bagheera> <20120716174956.162a2df2@pitrou.net> <20120716170441.GL14370@bagheera> <CAMpsgwaw21sr==hLTgD47aE6KBjKXC6C=CM80Va_GCru3V1rLA@mail.gmail.com> <50049D10.5040903@doxdesk.com> <CAMpsgwZ=if7PQ33Sg3zaOEf6NT4hmgL6mBtppW5FRJJ8n0efTA@mail.gmail.com> <20120717212542.GP14370@bagheera> <5005DEB9.3020406@mrabarnett.plus.com> Message-ID: <20120718103654.GU14370@bagheera> MRAB schrieb am Tue, 17. Jul 2012, um 22:52:57 +0100: > On 17/07/2012 22:25, Sven Marnach wrote: > >Victor Stinner schrieb am Tue, 17. Jul 2012, um 03:03:24 +0200: > >>os.fsencode() should not be used explicitly on Windows. > > > >What else should I do to pipe filenames to another process? At least, > >os.fsencode() seems to work, even with cyrillic filenames. > > > Encode to UTF-8? I don't have control over the other process (it's ExifTool in batch mode), so I have to use whatever encoding is considered the standard to encode filenames on Windows. `os.fsencode()` works fine for this, and Victor answered off-list that it would be fine in this case. Cheers, Sven From techtonik at gmail.com Wed Jul 18 19:30:02 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Jul 2012 20:30:02 +0300 Subject: [Python-ideas] '' in 'abc' == True Message-ID: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> I've just spotted inconsistency between string and lists handling: >>> '' in 'abc' True >>> '' in 'abc'.split() False >>> [] in ['a', 'b', 'c'] False Why strings here behave differently than other sequence types? Is that by design? From matthew.lefavor at nasa.gov Wed Jul 18 19:38:31 2012 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Wed, 18 Jul 2012 12:38:31 -0500 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> Message-ID: <CC2C6BD7.F6AA%matthew.lefavor@nasa.gov> Strings have a common use case which lists do not: finding subsequences/substrings. Consider the following: >>> string = 'abcdefghijklmnop' >>> 'def' in string True >>> list('def') in list(string) False The contains operator ("in") has a different meaning than the contains operator for a list. A list contains an object if (and only if) that object is a single element of the list. A string contains another string if (and only if) the other string is a substring of the first string. Matthew Lefavor NASA GSFC [Microtel, LLC] Mail Code 699.0/Org Code 582.0 matthew.lefavor at nasa.gov (301) 614-6818 (Desk) (443) 758-4891 (Cell) On 7/18/12 1:30 PM, "anatoly techtonik" <techtonik at gmail.com> wrote: >I've just spotted inconsistency between string and lists handling: > >>>> '' in 'abc' >True >>>> '' in 'abc'.split() >False >>>> [] in ['a', 'b', 'c'] >False > >Why strings here behave differently than other sequence types? Is that >by design? >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >http://mail.python.org/mailman/listinfo/python-ideas From masklinn at masklinn.net Wed Jul 18 19:43:27 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 18 Jul 2012 19:43:27 +0200 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> Message-ID: <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> On 2012-07-18, at 19:30 , anatoly techtonik wrote: > I've just spotted inconsistency between string and lists handling: > >>>> '' in 'abc' > True >>>> '' in 'abc'.split() > False >>>> [] in ['a', 'b', 'c'] > False > > Why strings here behave differently than other sequence types? Is that > by design? Erm? yes? `in` would not be very useful for strings if you could only use it to check for a single character would it? From masklinn at masklinn.net Wed Jul 18 19:58:17 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 18 Jul 2012 19:58:17 +0200 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> Message-ID: <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> On 2012-07-18, at 19:43 , Masklinn wrote: > On 2012-07-18, at 19:30 , anatoly techtonik wrote: > >> I've just spotted inconsistency between string and lists handling: >> >>>>> '' in 'abc' >> True >>>>> '' in 'abc'.split() >> False >>>>> [] in ['a', 'b', 'c'] >> False >> >> Why strings here behave differently than other sequence types? Is that >> by design? > > Erm? yes? `in` would not be very useful for strings if you could only use > it to check for a single character would it? in fact, things used to work that way in older Python, this was specifically changed to the current behavior *as noted in the documentation*: > When s is a string or Unicode string object the in and not in > operations act like a substring test. In Python versions before 2.3, x > had to be a string of length 1. In Python 2.3 and beyond, x may be a > string of any length. A Python string, you may want to note, is a string. Not a sequence of characters. The first item of a 1-character string is itself, all basic (step-less) slices of a string are contained in itself (including itself and the empty string), you can infinitely get the first item of a non-empty string, and I'm sure I'm missing plenty. From jeanpierreda at gmail.com Wed Jul 18 20:06:59 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 18 Jul 2012 14:06:59 -0400 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> Message-ID: <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: > A Python string, you may want to note, is a string. Not a sequence of > characters. It's both (with the caveat that, in Python, a character is just a string of length 1). (See: http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy ) -- Devin From masklinn at masklinn.net Wed Jul 18 20:16:18 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 18 Jul 2012 20:16:18 +0200 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> Message-ID: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: > On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >> A Python string, you may want to note, is a string. Not a sequence of >> characters. > > It's both (with the caveat that, in Python, a character is just a > string of length 1). That's playing with words, especially comparing strings with Python 3 binaries which *do* actually have a separate "character" type (reified to an integer). So Python strings don't have reified characters, a string's item and a slice of size 1 are essentially identical which is pretty much unique to them (as far as my knowledge of Python's sequences go). Which is not a bad thing, mind you, it makes working with strings much more pleasant. From jeanpierreda at gmail.com Wed Jul 18 20:31:19 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 18 Jul 2012 14:31:19 -0400 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> Message-ID: <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com> On Wed, Jul 18, 2012 at 2:16 PM, Masklinn <masklinn at masklinn.net> wrote: >> It's both (with the caveat that, in Python, a character is just a >> string of length 1). > > That's playing with words, especially comparing strings with Python 3 > binaries which *do* actually have a separate "character" type > (reified to an integer). No it isn't. Strings are adherents to the sequence protocol. The Python datatype reference echoes what I said, nearly exactly. http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy > So Python strings don't have reified characters, a string's item and a > slice of size 1 are essentially identical which is pretty much unique > to them (as far as my knowledge of Python's sequences go). Nothing about that feature makes them not-sequences; instead, it makes them a rather special kind of sequence. -- Devin From masklinn at masklinn.net Wed Jul 18 21:02:36 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 18 Jul 2012 21:02:36 +0200 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com> Message-ID: <285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net> On 2012-07-18, at 20:31 , Devin Jeanpierre wrote: > On Wed, Jul 18, 2012 at 2:16 PM, Masklinn <masklinn at masklinn.net> wrote: >>> It's both (with the caveat that, in Python, a character is just a >>> string of length 1). >> >> That's playing with words, especially comparing strings with Python 3 >> binaries which *do* actually have a separate "character" type >> (reified to an integer). > > No it isn't. Strings are adherents to the sequence protocol. The > Python datatype reference echoes what I said, nearly exactly. This has no relevance to my messages, I have not claimed anywhere that strings weren't sequences. >> So Python strings don't have reified characters, a string's item and a >> slice of size 1 are essentially identical which is pretty much unique >> to them (as far as my knowledge of Python's sequences go). > > Nothing about that feature makes them not-sequences; instead, it makes > them a rather special kind of sequence. I'm not sure why you're saying that. Again, I have never once claimed they were not sequences (quite the opposite in fact). Why the strawmanning? From ethan at stoneleaf.us Wed Jul 18 21:32:20 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 Jul 2012 12:32:20 -0700 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> Message-ID: <50070F44.5080509@stoneleaf.us> Masklinn wrote: > On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: >> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >>> A Python string, you may want to note, is a string. Not a sequence of >>> characters. >> It's both (with the caveat that, in Python, a character is just a >> string of length 1). > > That's playing with words, especially comparing strings with Python 3 > binaries which *do* actually have a separate "character" type > (reified to an integer). Python 3 does not have a 'character' type; it has 'str' which is made up of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). ~Ethan~ From jeanpierreda at gmail.com Wed Jul 18 22:02:29 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 18 Jul 2012 16:02:29 -0400 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <CABicbJ+-tjtOUSmQUpdFuWen9NTK1wwFCmEZ7-dF+so4GJ=snQ@mail.gmail.com> <285E23FE-CE6D-43EE-B849-44185B9A801C@masklinn.net> Message-ID: <CABicbJJWh00=vuJCuMaXH+5BHC4fGFjK_X+GQUNg9kiQX_1t-g@mail.gmail.com> On Wed, Jul 18, 2012 at 3:02 PM, Masklinn <masklinn at masklinn.net> wrote: > This has no relevance to my messages, I have not claimed anywhere that > strings weren't sequences. Sorry, I misinterpreted what you said. I'm tired. :( -- Devin From aquavitae69 at gmail.com Fri Jul 20 18:11:52 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Fri, 20 Jul 2012 18:11:52 +0200 Subject: [Python-ideas] __sort__ special member Message-ID: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases: 1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict. 2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial. Is there some obvious reason why nobody has suggested this before? Is it worth pursuing? David. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/98e29be9/attachment.html> From phd at phdru.name Fri Jul 20 18:23:11 2012 From: phd at phdru.name (Oleg Broytman) Date: Fri, 20 Jul 2012 20:23:11 +0400 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> Message-ID: <20120720162311.GA13159@iskra.aviel.ru> On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com> wrote: > I was quite surprised to find that nobody seems to have suggested this > before, because it seems like an obvious idea. Basically, add a special > method __sort__ which, if specified, is used when sorted() is called. I can > think of two immediate use cases: > > 1. When an object wants sorted() to return something other than a list, > e.g. dict.__sort__ could return an OrderedDict. > 2. When there is a more efficient method of sorting a specific sequence. > E.g. sorting a range object should be trivial. > > Is there some obvious reason why nobody has suggested this before? Is it > worth pursuing? Because it's too application-specific? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From matthew.lefavor at nasa.gov Fri Jul 20 18:22:46 2012 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Fri, 20 Jul 2012 11:22:46 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> Message-ID: <CC2EFC43.F815%matthew.lefavor@nasa.gov> With regard to point 1: I think that changing the return type of sorted() would cause nasty surprises. If you can no longer assume that calling sorted() on any arbitrary container is going to return an indexed list, then we've lost all the advantages of Python's "duck typing." With regard to point 2: How would this interact with the existing signature of the sorted() function? The sorted() function, as it is right now, is responsible not just for the sort algorithm, but also for defining how comparisons should work. If the comparison keys are changed, then you can't assume any sorting operation is trivial anymore. Consider the following, using a range object: >>> import math >>> sorted(range(10), key=math.sin) [5, 4, 6, 0, 3, 9, 7, 1, 2, 8] Matthew Lefavor From: David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> Date: Friday, July 20, 2012 12:11 PM To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>> Subject: [Python-ideas] __sort__ special member I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases: 1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict. 2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial. Is there some obvious reason why nobody has suggested this before? Is it worth pursuing? David. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/0f8410f6/attachment.html> From dreamingforward at gmail.com Fri Jul 20 18:44:26 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Fri, 20 Jul 2012 11:44:26 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <20120720162311.GA13159@iskra.aviel.ru> References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> <20120720162311.GA13159@iskra.aviel.ru> Message-ID: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com> On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name> wrote: > On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend < > aquavitae69 at gmail.com> wrote: > > I was quite surprised to find that nobody seems to have suggested this > > before, because it seems like an obvious idea. Basically, add a special > > method __sort__ which, if specified, is used when sorted() is called. I > can > > think of two immediate use cases: > > > > 1. When an object wants sorted() to return something other than a list, > > e.g. dict.__sort__ could return an OrderedDict. > > 2. When there is a more efficient method of sorting a specific sequence. > > E.g. sorting a range object should be trivial. > > > > Is there some obvious reason why nobody has suggested this before? Is it > > worth pursuing? > > Because it's too application-specific? > > What are you talking about. This would solve the issue of ordering for dicts that became contentious a few years ago. If people don't like the arbitrary ordering, they can subclass it. mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/8929f1e6/attachment.html> From matthew.lefavor at nasa.gov Fri Jul 20 18:54:43 2012 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Fri, 20 Jul 2012 11:54:43 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com> Message-ID: <CC2F04A9.F85B%matthew.lefavor@nasa.gov> I'm confused: how is this related to the question of dictionary ordering? Calling sorted() on a dict orders all the keys as it stands right now. Arbitrary dictionary ordering is only an issue when iterating over a dictionary. That could already be solved by subclassing dict and overriding the __iter__ method to return the keys in some sort of order. Matthew Lefavor From: Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>> Date: Friday, July 20, 2012 12:44 PM To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>> Subject: Re: [Python-ideas] __sort__ special member On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name<mailto:phd at phdru.name>> wrote: On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> wrote: > I was quite surprised to find that nobody seems to have suggested this > before, because it seems like an obvious idea. Basically, add a special > method __sort__ which, if specified, is used when sorted() is called. I can > think of two immediate use cases: > > 1. When an object wants sorted() to return something other than a list, > e.g. dict.__sort__ could return an OrderedDict. > 2. When there is a more efficient method of sorting a specific sequence. > E.g. sorting a range object should be trivial. > > Is there some obvious reason why nobody has suggested this before? Is it > worth pursuing? Because it's too application-specific? What are you talking about. This would solve the issue of ordering for dicts that became contentious a few years ago. If people don't like the arbitrary ordering, they can subclass it. mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/c6ded7a8/attachment.html> From matthew.lefavor at nasa.gov Fri Jul 20 18:29:29 2012 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Fri, 20 Jul 2012 11:29:29 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CC2EFC43.F815%matthew.lefavor@nasa.gov> Message-ID: <CC2EFF84.F82B%matthew.lefavor@nasa.gov> (I apologize if this sent twice, my outbox is acting screwy). With regard to point 1: I think that changing the return type of sorted() would cause nasty surprises. If you can no longer assume that calling sorted() on any arbitrary container is going to return an indexed list, then we've lost all the advantages of Python's "duck typing." With regard to point 2: How would this interact with the existing signature of the sorted() function? The sorted() function, as it is right now, is responsible not just for the sort algorithm, but also for defining how comparisons should work. If the comparison keys are changed, then you can't assume any sorting operation is trivial anymore. Consider the following, using a range object: >>> import math >>> sorted(range(10), key=math.sin) [5, 4, 6, 0, 3, 9, 7, 1, 2, 8] Matthew Lefavor From: David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> Date: Friday, July 20, 2012 12:11 PM To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>> Subject: [Python-ideas] __sort__ special member I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases: 1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict. 2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial. Is there some obvious reason why nobody has suggested this before? Is it worth pursuing? David. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/5f09290c/attachment.html> From dreamingforward at gmail.com Fri Jul 20 20:29:25 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Fri, 20 Jul 2012 13:29:25 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com> References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> <20120720162311.GA13159@iskra.aviel.ru> <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com> Message-ID: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> On Fri, Jul 20, 2012 at 11:44 AM, Mark Adam <dreamingforward at gmail.com>wrote: > On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name> wrote: > >> On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend < >> aquavitae69 at gmail.com> wrote: >> > I was quite surprised to find that nobody seems to have suggested this >> > before, because it seems like an obvious idea. Basically, add a special >> > method __sort__ which, if specified, is used when sorted() is called. I >> can >> > think of two immediate use cases: >> > >> > 1. When an object wants sorted() to return something other than a list, >> > e.g. dict.__sort__ could return an OrderedDict. >> > 2. When there is a more efficient method of sorting a specific sequence. >> > E.g. sorting a range object should be trivial. >> > >> > Is there some obvious reason why nobody has suggested this before? Is >> it >> > worth pursuing? >> >> Because it's too application-specific? >> >> What are you talking about. This would solve the issue of ordering for > dicts that became contentious a few years ago. If people don't like the > arbitrary ordering, they can subclass it. > > Sorry, I'll tone down my words. But this makes more sense than sort() as a built-in. In theory, sort is for collections, not just any object. So it could belong to an abstract collections base type. This would allow users to select lexical orderings, for example, rather than ASCII. It seems strange that no one thought of it before(it that true?). mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/c6c8d0f7/attachment.html> From matthew.lefavor at nasa.gov Fri Jul 20 20:55:34 2012 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Fri, 20 Jul 2012 13:55:34 -0500 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> Message-ID: <CC2F1EF4.F87E%matthew.lefavor@nasa.gov> The sorted() function works on any object supporting an __iter__ method. It does not just sort "any object"; if the argument is not iterable, sorted() will raise a TypeError. If you insist on talking about types, you could say it operates on any instance of the abstract class collections.Iterable. This includes generators. For example: >>> import collections >>> def gen(): ... yield 3 ... yield 2 ... yield 1 ... >>> isinstance(gen(), collections.Iterable) True >>> sorted(gen()) [1, 2, 3] As for your other point, users can already support whatever ordering they wish using the "key" argument to the sorted() function. For example, lexicographic ordering is already supported by using str.lower as the key function. For example: >>> test_list = ["python 3", "PYTHON 4", "Python 2", "pYtHoN 1"] >>> sorted(test_list) ['PYTHON 4', 'Python 2', 'pYtHoN 1', 'python 3'] >>> sorted(test_list, key=str.lower) ['pYtHoN 1', 'Python 2', 'python 3', 'PYTHON 4'] If you have an object for which you wanted to specify your own default comparison behavior, you could do so by specifying your own implementations of __lt__ or __eq__. Matthew Lefavor From: Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>> Date: Friday, July 20, 2012 2:29 PM To: "python-ideas at python.org<mailto:python-ideas at python.org>" <python-ideas at python.org<mailto:python-ideas at python.org>> Subject: Re: [Python-ideas] __sort__ special member On Fri, Jul 20, 2012 at 11:44 AM, Mark Adam <dreamingforward at gmail.com<mailto:dreamingforward at gmail.com>> wrote: On Fri, Jul 20, 2012 at 11:23 AM, Oleg Broytman <phd at phdru.name<mailto:phd at phdru.name>> wrote: On Fri, Jul 20, 2012 at 06:11:52PM +0200, David Townshend <aquavitae69 at gmail.com<mailto:aquavitae69 at gmail.com>> wrote: > I was quite surprised to find that nobody seems to have suggested this > before, because it seems like an obvious idea. Basically, add a special > method __sort__ which, if specified, is used when sorted() is called. I can > think of two immediate use cases: > > 1. When an object wants sorted() to return something other than a list, > e.g. dict.__sort__ could return an OrderedDict. > 2. When there is a more efficient method of sorting a specific sequence. > E.g. sorting a range object should be trivial. > > Is there some obvious reason why nobody has suggested this before? Is it > worth pursuing? Because it's too application-specific? What are you talking about. This would solve the issue of ordering for dicts that became contentious a few years ago. If people don't like the arbitrary ordering, they can subclass it. Sorry, I'll tone down my words. But this makes more sense than sort() as a built-in. In theory, sort is for collections, not just any object. So it could belong to an abstract collections base type. This would allow users to select lexical orderings, for example, rather than ASCII. It seems strange that no one thought of it before(it that true?). mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/7bb06210/attachment.html> From g.brandl at gmx.net Fri Jul 20 22:56:08 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 20 Jul 2012 22:56:08 +0200 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <50070F44.5080509@stoneleaf.us> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> Message-ID: <jucgl8$lb$1@dough.gmane.org> On 07/18/2012 09:32 PM, Ethan Furman wrote: > Masklinn wrote: >> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: >>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >>>> A Python string, you may want to note, is a string. Not a sequence of >>>> characters. >>> It's both (with the caveat that, in Python, a character is just a >>> string of length 1). >> >> That's playing with words, especially comparing strings with Python 3 >> binaries which *do* actually have a separate "character" type >> (reified to an integer). > > Python 3 does not have a 'character' type; it has 'str' which is made up > of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). That's what he said. Could we stop the annoying "but I know it better than you without reading your message" please? Georg From ethan at stoneleaf.us Fri Jul 20 23:18:20 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 20 Jul 2012 14:18:20 -0700 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <jucgl8$lb$1@dough.gmane.org> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> Message-ID: <5009CB1C.4050201@stoneleaf.us> Georg Brandl wrote: > On 07/18/2012 09:32 PM, Ethan Furman wrote: >> Masklinn wrote: >>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: >>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >>>>> A Python string, you may want to note, is a string. Not a sequence of >>>>> characters. >>>> >>>> It's both (with the caveat that, in Python, a character is just a >>>> string of length 1). >>> >>> That's playing with words, especially comparing strings with Python 3 >>> binaries which *do* actually have a separate "character" type >>> (reified to an integer). >> >> Python 3 does not have a 'character' type; it has 'str' which is made up >> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). > > That's what he said. Could we stop the annoying "but I know it better > than you without reading your message" please? I am having trouble equating what I said with with Masklinn said. Perhaps you could explain how they say the same thing instead of assuming I didn't read his message? ~Ethan~ From andrew.svetlov at gmail.com Fri Jul 20 23:19:36 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 21 Jul 2012 00:19:36 +0300 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <jucgl8$lb$1@dough.gmane.org> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> Message-ID: <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com> Masklinn's explanation is comprehensive clean to me. On Fri, Jul 20, 2012 at 11:56 PM, Georg Brandl <g.brandl at gmx.net> wrote: > On 07/18/2012 09:32 PM, Ethan Furman wrote: >> Masklinn wrote: >>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: >>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >>>>> A Python string, you may want to note, is a string. Not a sequence of >>>>> characters. >>>> It's both (with the caveat that, in Python, a character is just a >>>> string of length 1). >>> >>> That's playing with words, especially comparing strings with Python 3 >>> binaries which *do* actually have a separate "character" type >>> (reified to an integer). >> >> Python 3 does not have a 'character' type; it has 'str' which is made up >> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). > > That's what he said. Could we stop the annoying "but I know it better > than you without reading your message" please? > > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Thanks, Andrew Svetlov From solipsis at pitrou.net Fri Jul 20 23:27:10 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Jul 2012 23:27:10 +0200 Subject: [Python-ideas] '' in 'abc' == True References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> <5009CB1C.4050201@stoneleaf.us> Message-ID: <20120720232710.55a85f11@pitrou.net> On Fri, 20 Jul 2012 14:18:20 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > Georg Brandl wrote: > > On 07/18/2012 09:32 PM, Ethan Furman wrote: > >> Masklinn wrote: > >>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: > >>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: > >>>>> A Python string, you may want to note, is a string. Not a sequence of > >>>>> characters. > >>>> > >>>> It's both (with the caveat that, in Python, a character is just a > >>>> string of length 1). > >>> > >>> That's playing with words, especially comparing strings with Python 3 > >>> binaries which *do* actually have a separate "character" type > >>> (reified to an integer). > >> > >> Python 3 does not have a 'character' type; it has 'str' which is made up > >> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). > > > > That's what he said. Could we stop the annoying "but I know it better > > than you without reading your message" please? > > I am having trouble equating what I said with with Masklinn said. > Perhaps you could explain how they say the same thing instead of > assuming I didn't read his message? "Python 3 binaries" probably means "Python 3 bytes objects" above. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From ncoghlan at gmail.com Sat Jul 21 01:19:50 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Jul 2012 09:19:50 +1000 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CC2F1EF4.F87E%matthew.lefavor@nasa.gov> References: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> <CC2F1EF4.F87E%matthew.lefavor@nasa.gov> Message-ID: <CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com> Because the decision of how sorting should occur is up to the consumer, not the provider. sorted() has the contract to produce a list, using an algorithm that makes a lot of guarantees about the results. If objects are allowed to override sorting completely, then those guarantees are no longer possible. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120721/888b5fdb/attachment.html> From raymond.hettinger at gmail.com Sat Jul 21 02:07:43 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 20 Jul 2012 17:07:43 -0700 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com> References: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> <CC2F1EF4.F87E%matthew.lefavor@nasa.gov> <CADiSq7eL6ZiEiT=Warc+r7D93parvNU=Jg0ioghY_33Y9U+AgA@mail.gmail.com> Message-ID: <56A2427D-03FB-4CB2-BC38-4BDDBC49BEB3@gmail.com> On Jul 20, 2012, at 4:19 PM, Nick Coghlan wrote: > Because the decision of how sorting should occur is up to the consumer, not the provider. sorted() has the contract to produce a list, using an algorithm that makes a lot of guarantees about the results. If objects are allowed to override sorting completely, then those guarantees are no longer possible. Well said. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120720/090710d0/attachment.html> From steve at pearwood.info Sat Jul 21 13:19:24 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 Jul 2012 21:19:24 +1000 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com> Message-ID: <500A903C.7020405@pearwood.info> Andrew Svetlov wrote: > Masklinn's explanation is comprehensive clean to me. I'm glad that it's clear to someone, because to me the straight-forward, literal meaning of Masklinn's explanation (that Python 3 has a character type, and they're integers) is wrong. Python has no built-in "Char" type, under any spelling, let alone one which is also a subset of int. The non-literal meaning is hard to understand. I *guess* that Masklinn is trying to get across that Python 3 strings are Unicode strings, and characters in Unicode are actually code points, which are implemented at the C level as integers. If not that, I have no idea. I've more or less forgotten why this was important, but I am enjoying watching people try to out-pedant each other :) > On Fri, Jul 20, 2012 at 11:56 PM, Georg Brandl <g.brandl at gmx.net> wrote: >> On 07/18/2012 09:32 PM, Ethan Furman wrote: >>> Masklinn wrote: >>>> On 2012-07-18, at 20:06 , Devin Jeanpierre wrote: >>>>> On Wed, Jul 18, 2012 at 1:58 PM, Masklinn <masklinn at masklinn.net> wrote: >>>>>> A Python string, you may want to note, is a string. Not a sequence of >>>>>> characters. >>>>> It's both (with the caveat that, in Python, a character is just a >>>>> string of length 1). >>>> That's playing with words, especially comparing strings with Python 3 >>>> binaries which *do* actually have a separate "character" type >>>> (reified to an integer). >>> Python 3 does not have a 'character' type; it has 'str' which is made up >>> of more 'str's, and it has 'byte' which is made up of 'int's (annoyingly). >> That's what he said. Could we stop the annoying "but I know it better >> than you without reading your message" please? -- Steven From jeanpierreda at gmail.com Sat Jul 21 15:59:05 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 21 Jul 2012 09:59:05 -0400 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <500A903C.7020405@pearwood.info> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com> <500A903C.7020405@pearwood.info> Message-ID: <CABicbJKpzgUcxH1u4O+SR9natMcDtJf7sytFpV8B3xtUqqFZFg@mail.gmail.com> On Sat, Jul 21, 2012 at 7:19 AM, Steven D'Aprano <steve at pearwood.info> wrote: > Andrew Svetlov wrote: >> >> Masklinn's explanation is comprehensive clean to me. > > > I'm glad that it's clear to someone, because to me the straight-forward, > literal meaning of Masklinn's explanation (that Python 3 has a character > type, and they're integers) is wrong. Python has no built-in "Char" type, > under any spelling, let alone one which is also a subset of int. The > non-literal meaning is hard to understand. I *guess* that Masklinn is trying > to get across that Python 3 strings are Unicode strings, and characters in > Unicode are actually code points, which are implemented at the C level as > integers. If not that, I have no idea. You're pretty far off. He was talking about bytes objects, not str objects. -- Devin From matt at whoosh.ca Sat Jul 21 20:59:52 2012 From: matt at whoosh.ca (Matt Chaput) Date: Sat, 21 Jul 2012 14:59:52 -0400 Subject: [Python-ideas] Atomic file.get(offset, length) Message-ID: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> I wish Python binary file objects had an atomic seek-read method, so I wouldn't have to perform my own locking everywhere to prevent other threads from moving the file pointer between seek and read. Is this something that can be bubbled up from the underlying platform? I think the Linux C equivalent is pread. I also think Java has something like this but can't find a reference now. Or does this exist and I missed it? (On a mmap file this is trivial, of course.) Has this been discussed before? Cheers, Matt From tjreedy at udel.edu Sat Jul 21 21:35:04 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 21 Jul 2012 15:35:04 -0400 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> Message-ID: <juf09k$njc$1@dough.gmane.org> On 7/21/2012 2:59 PM, Matt Chaput wrote: > I wish Python binary file objects had an atomic seek-read method, so > I wouldn't have to perform my own locking everywhere to prevent other > threads from moving the file pointer between seek and read. If you are reading a file from multiple threads, I suggest you write your own seek_and_read_with_locks function that does exactly what you need in one place. Or add a .readx method to a subclass. > Is this something that can be bubbled up from the underlying > platform? I think the Linux C equivalent is pread. If there is a standard posix function that is not yet wrapped in os, you can propose its addition. But some research to see has widespread and actually standardized it is. -- Terry Jan Reedy From storchaka at gmail.com Sat Jul 21 21:36:17 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Jul 2012 22:36:17 +0300 Subject: [Python-ideas] '' in 'abc' == True In-Reply-To: <500A903C.7020405@pearwood.info> References: <CAPkN8xLnaDMjsYTdJ=q8V07qYXHFN8FKDvA3b-pz=S=wnGKbkA@mail.gmail.com> <CC83294C-8D18-4716-BAD8-B206FD301876@masklinn.net> <8FF1DF82-96C3-44CE-936F-00E5CEE5FA42@masklinn.net> <CABicbJJ8f7H=NpBjNN5L7DaOndAN9RakhraL9ktHRSc76qdZHg@mail.gmail.com> <192B220A-46F7-4891-9DD4-2D58A38210CD@masklinn.net> <50070F44.5080509@stoneleaf.us> <jucgl8$lb$1@dough.gmane.org> <CAL3CFcWBsddH5RfN-RP+8mDwUF0aSbnZO1YeAS2E2UAmKDE8bg@mail.gmail.com> <500A903C.7020405@pearwood.info> Message-ID: <juf0bj$o0a$1@dough.gmane.org> On 21.07.12 14:19, Steven D'Aprano wrote: > I'm glad that it's clear to someone, because to me the straight-forward, > literal meaning of Masklinn's explanation (that Python 3 has a character > type, and they're integers) is wrong. Yes, but Masklinn did not claim this. From guido at python.org Sat Jul 21 22:35:21 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Jul 2012 13:35:21 -0700 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <juf09k$njc$1@dough.gmane.org> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> Message-ID: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> On Sat, Jul 21, 2012 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 7/21/2012 2:59 PM, Matt Chaput wrote: >> >> I wish Python binary file objects had an atomic seek-read method, so >> I wouldn't have to perform my own locking everywhere to prevent other >> threads from moving the file pointer between seek and read. > > > If you are reading a file from multiple threads, I suggest you write your > own seek_and_read_with_locks function that does exactly what you need in one > place. Or add a .readx method to a subclass. > > >> Is this something that can be bubbled up from the underlying >> platform? I think the Linux C equivalent is pread. > > If there is a standard posix function that is not yet wrapped in os, you can > propose its addition. But some research to see has widespread and actually > standardized it is. "man pread" on OS/X suggests it exists there too. I presume the use case is to have a large data file open for reading by multiple threads. This is a reasonable use case and it makes some sense to extend our binary readable streams (buffered and unbuffered) with an API for this purpose. However, it's probably just efficient to just have a separate open stream per thread -- I doubt that open file descriptors are scarcer resources than threads, and I presume the kernel will happily share any buffering it does on behalf of multiple open files referencing the same file. If you're worried about the buffer space, the default buffer size is 8K, which is hardly worth mentioning compared to the default thread stack allocation. Depending on your use case you may get away with an unbuffered stream just fine. This approach seems better than implementing something using locks (since the locks create contention that is not inherent in the problem) and is available right now, without waiting for Python 3.4 to be released... -- --Guido van Rossum (python.org/~guido) From jeanpierreda at gmail.com Sun Jul 22 03:08:50 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 21 Jul 2012 21:08:50 -0400 Subject: [Python-ideas] __sort__ special member In-Reply-To: <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> References: <CAEgL-ff+AscG2MaNNacMwsCDv_GNGtdXLz7kNh7QjaveJk-qrg@mail.gmail.com> <20120720162311.GA13159@iskra.aviel.ru> <CAMjeLr9hEAhORtb7FzN60itHyf2M+s-dAUND-YxQK+rQo+_4dA@mail.gmail.com> <CAMjeLr85r3w6meZxrSN2BDeiSwy9VCmONQ2o4fODp5-fm74-GQ@mail.gmail.com> Message-ID: <CABicbJK_Mgry5HzM5XMU4tUBJfTWiQ80YGpm9VbT+kY=i9=1WQ@mail.gmail.com> On Fri, Jul 20, 2012 at 2:29 PM, Mark Adam <dreamingforward at gmail.com> wrote: > Sorry, I'll tone down my words. But this makes more sense than sort() as a > built-in. In theory, sort is for collections, not just any object. So it > could belong to an abstract collections base type. This would allow users > to select lexical orderings, for example, rather than ASCII. It seems > strange that no one thought of it before(it that true?). FWIW the sort method (as in list.sort()) is polymorphic, just nobody bothers implementing it. And how many interesting specializations are there? At best I can only think of linear-time sorts, but it's pretty rare you'll know your collection only contains discrete values. -- Devin From michelelacchia at gmail.com Sun Jul 22 21:58:34 2012 From: michelelacchia at gmail.com (Michele Lacchia) Date: Sun, 22 Jul 2012 12:58:34 -0700 (PDT) Subject: [Python-ideas] Where has packaging gone? In-Reply-To: <CAPkN8xJ_776rx0nwA_zYoMCNLd82n9pnfepb+2rOrOQwNSFgCQ@mail.gmail.com> References: <ad208ce0-53ff-451a-bc94-c4cd9b075bcf@googlegroups.com> <CAPkN8xJ_776rx0nwA_zYoMCNLd82n9pnfepb+2rOrOQwNSFgCQ@mail.gmail.com> Message-ID: <dc8dcbc1-627b-4d8a-a5b9-566b59247ae8@googlegroups.com> > > Thoughts attached. > Yeah that one is great! :) And true. No, the distutils2 project continues and is still being actively > developed. (distutils2 *is* packaging - the name on PyPI is just > different from the name that will be used when it is added to the > standard library). Bento is a project that aims to tackle part of > distutils that distutils2 hasn't really looked at yet: creating a more > flexible build system (or, more accurately, making it easier to hook > into *existing* build systems rather than defining yet another one). Ok. Now the distinction is clear. Good to hear disutils2 is still actively developed. Those PEPs are *already* standards and are actively supported by > external projects today. I just found the wheel project: https://bitbucket.org/dholth/wheel It tries to implement those standards by patching distribute and pip. I run the demo and I think the resulting package format is just amazing. In a nutshell is a binary format (extensions get compiled) which in my opinion is really flexible. I hope it will become standard and will be integrated in the major software someday. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120722/b36ba8e6/attachment.html> From pyideas at rebertia.com Sun Jul 22 22:33:53 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 22 Jul 2012 13:33:53 -0700 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> Message-ID: <CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com> On Mon, Jul 16, 2012 at 12:12 AM, David Townshend <aquavitae69 at gmail.com> wrote: > On Jul 16, 2012 4:04 AM, "Chris Rebert" <pyideas at rebertia.com> wrote: <snip> >> Given that PEP 3130 (http://www.python.org/dev/peps/pep-3130/ ), which >> would have similarly added nice ways to get the current module, class, >> and regular function, was rejected, I wouldn't hold out much hope for >> such a feature being added any time soon. >> (Wrapping via decorators introduces nontrivial subtlety regarding >> whether you really want to refer to the decorated or undecorated >> version of an entity.) <snip> > The problem with PEP 3130 was the PEP itself, not the idea. Could you elaborate on how the PEP and the "idea" materially differed? > I believe Eric > Snow was working on a patch to implement this (see > http://code.activestate.com/lists/python-ideas/11234/ and > http://bugs.python.org/issue12857), but I'm not sure what its current status > is. I would hardly describe mucking around with frame objects as "nice" (particularly compared against a dedicated keyword), so I would not say that it implements the same idea; though, to be sure, what the patch accomplishes is related, still useful, and could be used to power the implementation of something like the PEP. Belated regards, Chris -- Please avoid top-posting in the future :) From victor.stinner at gmail.com Sun Jul 22 23:25:59 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 22 Jul 2012 23:25:59 +0200 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> Message-ID: <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> > "man pread" on OS/X suggests it exists there too "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() have been added to Python 3.3. Victor From guido at python.org Mon Jul 23 00:36:44 2012 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Jul 2012 15:36:44 -0700 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> Message-ID: <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner <victor.stinner at gmail.com> wrote: >> "man pread" on OS/X suggests it exists there too > > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() > have been added to Python 3.3. Awesome. :-) But does the io module offer an API that uses it? It's kind of awkward to have to call os.pread() with stream.fileno() as an argument. -- --Guido van Rossum (python.org/~guido) From cs at zip.com.au Mon Jul 23 01:04:30 2012 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 23 Jul 2012 09:04:30 +1000 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> References: <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> Message-ID: <20120722230430.GA15682@cskk.homeip.net> On 21Jul2012 13:35, Guido van Rossum <guido at python.org> wrote: | On Sat, Jul 21, 2012 at 12:35 PM, Terry Reedy <tjreedy at udel.edu> wrote: | > On 7/21/2012 2:59 PM, Matt Chaput wrote: | >> I wish Python binary file objects had an atomic seek-read method, so | >> I wouldn't have to perform my own locking everywhere to prevent other | >> threads from moving the file pointer between seek and read. [...] | >> Is this something that can be bubbled up from the underlying | >> platform? I think the Linux C equivalent is pread. [...] | "man pread" on OS/X suggests it exists there too. I presume the use | case is to have a large data file open for reading by multiple | threads. This is a reasonable use case and it makes some sense to | extend our binary readable streams (buffered and unbuffered) with an | API for this purpose. On most Linux boxen you can say: man 3p pread which will show you the POSIX man page, if it exists. And it does! So pread will exist on pretty much every UNIX platform, and I'd be amazed if it wasn't on Windows. In fact, it remarks that pread appeared in SysVr4, which is quite old. | However, it's probably just efficient to just have a separate open | stream per thread It doubles the system call count per read (if pread is a system call, which it ually will be (it is on Linux and MacOSX, and is hard to implement otherwise without an annoying and slow locking scheme concealed inside the C library). I'd be +1 for adding pread and pwrite to the os module. It seems reasonable and quite useful and should work on most platforms. Cheers, -- Cameron Simpson <cs at zip.com.au> Rimmer: It will be happened; it shall be going to be happening; it will be was an event that could will have been taken place in the future. - Red Dwarf, _Future Echoes_ From solipsis at pitrou.net Mon Jul 23 01:47:10 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Jul 2012 01:47:10 +0200 Subject: [Python-ideas] Atomic file.get(offset, length) References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> Message-ID: <20120723014710.61fd5e12@pitrou.net> On Sun, 22 Jul 2012 15:36:44 -0700 Guido van Rossum <guido at python.org> wrote: > On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner > <victor.stinner at gmail.com> wrote: > >> "man pread" on OS/X suggests it exists there too > > > > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() > > have been added to Python 3.3. > > Awesome. :-) But does the io module offer an API that uses it? It's > kind of awkward to have to call os.pread() with stream.fileno() as an > argument. It doesn't. I guess we could add an "offset" keyword-only argument to read() and write(), but then we need to provide a Windows implementation as well (it seems using overlapped I/O with ReadFile() / WriteFile() could make it possible). Also, I'm not sure it makes sense for buffered I/O, or only unbuffered. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From guido at python.org Mon Jul 23 02:16:57 2012 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Jul 2012 17:16:57 -0700 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <20120723014710.61fd5e12@pitrou.net> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> <20120723014710.61fd5e12@pitrou.net> Message-ID: <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sun, 22 Jul 2012 15:36:44 -0700 > Guido van Rossum <guido at python.org> wrote: >> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner >> <victor.stinner at gmail.com> wrote: >> >> "man pread" on OS/X suggests it exists there too >> > >> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() >> > have been added to Python 3.3. >> >> Awesome. :-) But does the io module offer an API that uses it? It's >> kind of awkward to have to call os.pread() with stream.fileno() as an >> argument. > > It doesn't. I guess we could add an "offset" keyword-only argument to > read() and write(), but then we need to provide a Windows > implementation as well (it seems using overlapped I/O with > ReadFile() / WriteFile() could make it possible). Also, I'm not sure it > makes sense for buffered I/O, or only unbuffered. Given that the use case is to avoid race conditions when more than one thread is doing random-access reads on the same open file, I think it makes some sense to implement it for both buffered and unbuffered streams -- and even for text streams, since those support seek() as well, so the race condition exists for those too. But note that the pread() man page (at least the one I checked :-) specifies that pread() doesn't affect the file pointer. So I suppose it should also not affect the buffer. That may make it hard to implement it for text streams (which IIRC rely quite heavy on buffering for their implementation), but it should be easy for buffered streams: it should just be passed on to the underlying unbuffered stream. (For those jumping in the middle of the thread: I know it's past the feature freeze, so these considerations are for 3.4. Also, os.pread() is in 3.3.) -- --Guido van Rossum (python.org/~guido) From g.rodola at gmail.com Mon Jul 23 06:03:24 2012 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Mon, 23 Jul 2012 06:03:24 +0200 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins Message-ID: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> This would be similar to 'key' argument already available for min(), max() and sorted() and would let user decide what must be considered True and what not. Some use cases: all(a, b, c, ... key=callable) all(a, b, c, ... key=lambda x: isinstance(x, int)) any(a, b, c, ... key=lambda x: x < 10) a = b = c = None for x in foo: if cond1: a = 0 elif cond2: b = 0 elif cond3: c = 0 # meaning all cond* has been satisfied and it's ok to consider 0 a meaningful value assert all(a, b, c, key=lambda x: x != None) Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/530d1aaf/attachment.html> From raymond.hettinger at gmail.com Mon Jul 23 06:36:28 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 22 Jul 2012 23:36:28 -0500 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins In-Reply-To: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> Message-ID: <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com> On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote: > This would be similar to 'key' argument already available for min(), max() and sorted() and would let user decide what must be considered True and what not. There's no need. We already have: >>> all(predicate(x) for x in iterable) >>> any(predicate(x) for x in iterable) Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120722/9c380f04/attachment.html> From pyideas at rebertia.com Mon Jul 23 06:39:13 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 22 Jul 2012 21:39:13 -0700 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins In-Reply-To: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> Message-ID: <CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com> On Sun, Jul 22, 2012 at 9:03 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote: > This would be similar to 'key' argument already available for min(), max() > and sorted() and would let user decide what must be considered True and what > not. > Some use cases: > > > all(a, b, c, ... key=callable) You're missing some brackets or parens there: all([a, b, c, ...], key=callable) <snip> > Thoughts? I see negligible advantage over just writing the generator expression directly: all(callable(item) for item in iterable) === all(iterable, key=callable) (which is about comparable to the difference between a map() invocation vs. a list comprehension, and the choice there seems to be largely a matter of taste) min() & max() return the result object satisfying the constraint, so the `key` argument makes sense for when you want to e.g. find the cheapest house rather than the price of the cheapest house. In contrast, any() & all() *always* just return a simple bool result, not the object whose truthiness determined the predicate's result, so `key` would be pointless since the result gets converted to a plain bool anyway (or at least, we can conceptualize the implementation as if it worked that way). Cheers, Chris From aquavitae69 at gmail.com Mon Jul 23 09:07:01 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 23 Jul 2012 09:07:01 +0200 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> <CAMZYqRR620e8xyeck8uFcXsf8eeF7MHYKZqrNfieZZjTEPiMqA@mail.gmail.com> Message-ID: <CAEgL-fdmxyerPVydU=v9Yvi9Nzm1iYjAF7Xgy+i8MBz580CCOw@mail.gmail.com> > > Could you elaborate on how the PEP and the "idea" materially differed? > IIRC, this was explained quite well in the thread I linked to. Basically the idea of accessing functions objects from within the function was fine, but the PEP didn't adequately answer all the questions it should have and tried to cover three ideas in one. To quote the rejection notice from the PEP: This PEP is rejected. It is not clear how it should be implemented or what the precise semantics should be in edge cases, and there aren't enough important use cases given. response has been lukewarm at best. > I would hardly describe mucking around with frame objects as "nice" > (particularly compared against a dedicated keyword), so I would not > say that it implements the same idea; though, to be sure, what the > patch accomplishes is related, still useful, and could be used to > power the implementation of something like the PEP. > Agreed, but it is a good start. > Please avoid top-posting in the future :) > Sorry about that - tends to happen when I reply on my phone if I'm not concentrating. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/0e1c2520/attachment.html> From solipsis at pitrou.net Mon Jul 23 11:52:16 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Jul 2012 11:52:16 +0200 Subject: [Python-ideas] Atomic file.get(offset, length) References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> <20120723014710.61fd5e12@pitrou.net> <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com> Message-ID: <20120723115216.33cc8ed5@pitrou.net> On Sun, 22 Jul 2012 17:16:57 -0700 Guido van Rossum <guido at python.org> wrote: > On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > On Sun, 22 Jul 2012 15:36:44 -0700 > > Guido van Rossum <guido at python.org> wrote: > >> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner > >> <victor.stinner at gmail.com> wrote: > >> >> "man pread" on OS/X suggests it exists there too > >> > > >> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() > >> > have been added to Python 3.3. > >> > >> Awesome. :-) But does the io module offer an API that uses it? It's > >> kind of awkward to have to call os.pread() with stream.fileno() as an > >> argument. > > > > It doesn't. I guess we could add an "offset" keyword-only argument to > > read() and write(), but then we need to provide a Windows > > implementation as well (it seems using overlapped I/O with > > ReadFile() / WriteFile() could make it possible). Also, I'm not sure it > > makes sense for buffered I/O, or only unbuffered. > > Given that the use case is to avoid race conditions when more than one > thread is doing random-access reads on the same open file, I think it > makes some sense to implement it for both buffered and unbuffered > streams -- and even for text streams, since those support seek() as > well, so the race condition exists for those too. > > But note that the pread() man page (at least the one I checked :-) > specifies that pread() doesn't affect the file pointer. So I suppose > it should also not affect the buffer. That may make it hard to > implement it for text streams (which IIRC rely quite heavy on > buffering for their implementation), but it should be easy for > buffered streams: it should just be passed on to the underlying > unbuffered stream. Indeed, it should not affect the buffer. That's why I'm questioning the addition of this feature to buffered streams (whose whole point is their implicit buffer management). Also, there are implementation subtleties when e.g. reading from an area which overlaps the current buffer :-) As you pointed out, I think a reasonable solution to the race condition problem is to use several file descriptors. It may not work so well if you also write to the file, though. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From g.rodola at gmail.com Mon Jul 23 13:28:08 2012 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Mon, 23 Jul 2012 13:28:08 +0200 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins In-Reply-To: <CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com> References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> <CAMZYqRRv0zwyObYu-QmqCwMXjhTG5Q+PtafqRUNhad22v_xH8A@mail.gmail.com> Message-ID: <CAFYqXL93gpRK0B0jxXqbVpEio4Kq--=DiUiP3hE=sTHmrt35iQ@mail.gmail.com> 2012/7/23 Chris Rebert <pyideas at rebertia.com> > > On Sun, Jul 22, 2012 at 9:03 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote: > > This would be similar to 'key' argument already available for min(), max() > > and sorted() and would let user decide what must be considered True and what > > not. > > Some use cases: > > > > > > all(a, b, c, ... key=callable) > > You're missing some brackets or parens there: > all([a, b, c, ...], key=callable) > > <snip> > > Thoughts? > > I see negligible advantage over just writing the generator expression directly: > all(callable(item) for item in iterable) <snip> > min() & max() return the result object satisfying the constraint, so > the `key` argument makes sense for when you want to e.g. find the > cheapest house rather than the price of the cheapest house. > In contrast, any() & all() *always* just return a simple bool result, > not the object whose truthiness determined the predicate's result, so > `key` would be pointless since the result gets converted to a plain > bool anyway (or at least, we can conceptualize the implementation as > if it worked that way). Agreed. Please ignore my proposal then. --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/ From guido at python.org Mon Jul 23 16:55:32 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Jul 2012 07:55:32 -0700 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins In-Reply-To: <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com> References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com> Message-ID: <CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com> On Sun, Jul 22, 2012 at 9:36 PM, Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > > On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote: > > This would be similar to 'key' argument already available for min(), max() > and sorted() and would let user decide what must be considered True and what > not. > > There's no need. We already have: > > >>> all(predicate(x) for x in iterable) > >>> any(predicate(x) for x in iterable) In addition, there's a reason why the key= parameter to sort, sorted, max and min can't be replaced with a similar idiom: all these return the *original* object(s) and the key= parameter affects only the comparison; for example, try max(1,-2,3,-4, key=abs). Whereas for the current proposal, the value returned by the key would also determine the outcome -- and then there is no advantage over using the comprehension. More general, if you can first apply the transformation (the key function) and then the aggregation function (min/max/etc.), there's no need to combine the transformation and the aggregation. But in all the cases that have a key= parameter, this is not so. -- --Guido van Rossum (python.org/~guido) From guido at python.org Mon Jul 23 17:00:37 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Jul 2012 08:00:37 -0700 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <20120723115216.33cc8ed5@pitrou.net> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> <20120723014710.61fd5e12@pitrou.net> <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com> <20120723115216.33cc8ed5@pitrou.net> Message-ID: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com> On Mon, Jul 23, 2012 at 2:52 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sun, 22 Jul 2012 17:16:57 -0700 > Guido van Rossum <guido at python.org> wrote: > >> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: >> > On Sun, 22 Jul 2012 15:36:44 -0700 >> > Guido van Rossum <guido at python.org> wrote: >> >> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner >> >> <victor.stinner at gmail.com> wrote: >> >> >> "man pread" on OS/X suggests it exists there too >> >> > >> >> > "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() >> >> > have been added to Python 3.3. >> >> >> >> Awesome. :-) But does the io module offer an API that uses it? It's >> >> kind of awkward to have to call os.pread() with stream.fileno() as an >> >> argument. >> > >> > It doesn't. I guess we could add an "offset" keyword-only argument to >> > read() and write(), but then we need to provide a Windows >> > implementation as well (it seems using overlapped I/O with >> > ReadFile() / WriteFile() could make it possible). Also, I'm not sure it >> > makes sense for buffered I/O, or only unbuffered. >> >> Given that the use case is to avoid race conditions when more than one >> thread is doing random-access reads on the same open file, I think it >> makes some sense to implement it for both buffered and unbuffered >> streams -- and even for text streams, since those support seek() as >> well, so the race condition exists for those too. >> >> But note that the pread() man page (at least the one I checked :-) >> specifies that pread() doesn't affect the file pointer. So I suppose >> it should also not affect the buffer. That may make it hard to >> implement it for text streams (which IIRC rely quite heavy on >> buffering for their implementation), but it should be easy for >> buffered streams: it should just be passed on to the underlying >> unbuffered stream. > > Indeed, it should not affect the buffer. That's why I'm questioning the > addition of this feature to buffered streams (whose whole point is their > implicit buffer management). Also, there are implementation subtleties > when e.g. reading from an area which overlaps the current buffer :-) > > As you pointed out, I think a reasonable solution to the race condition > problem is to use several file descriptors. It may not work so well if > you also write to the file, though. If you write and read a file from multiple threads you're crazy. -- --Guido van Rossum (python.org/~guido) From breamoreboy at yahoo.co.uk Mon Jul 23 17:12:05 2012 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 23 Jul 2012 16:12:05 +0100 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> <20120723014710.61fd5e12@pitrou.net> <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com> <20120723115216.33cc8ed5@pitrou.net> <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com> Message-ID: <jujpgu$2pr$1@dough.gmane.org> On 23/07/2012 16:00, Guido van Rossum wrote: > On Mon, Jul 23, 2012 at 2:52 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: >> On Sun, 22 Jul 2012 17:16:57 -0700 >> Guido van Rossum <guido at python.org> wrote: >> >>> On Sun, Jul 22, 2012 at 4:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: >>>> On Sun, 22 Jul 2012 15:36:44 -0700 >>>> Guido van Rossum <guido at python.org> wrote: >>>>> On Sun, Jul 22, 2012 at 2:25 PM, Victor Stinner >>>>> <victor.stinner at gmail.com> wrote: >>>>>>> "man pread" on OS/X suggests it exists there too >>>>>> >>>>>> "man pread" or "import os; help(os.pread" ;-) pread() and pwrite() >>>>>> have been added to Python 3.3. >>>>> >>>>> Awesome. :-) But does the io module offer an API that uses it? It's >>>>> kind of awkward to have to call os.pread() with stream.fileno() as an >>>>> argument. >>>> >>>> It doesn't. I guess we could add an "offset" keyword-only argument to >>>> read() and write(), but then we need to provide a Windows >>>> implementation as well (it seems using overlapped I/O with >>>> ReadFile() / WriteFile() could make it possible). Also, I'm not sure it >>>> makes sense for buffered I/O, or only unbuffered. >>> >>> Given that the use case is to avoid race conditions when more than one >>> thread is doing random-access reads on the same open file, I think it >>> makes some sense to implement it for both buffered and unbuffered >>> streams -- and even for text streams, since those support seek() as >>> well, so the race condition exists for those too. >>> >>> But note that the pread() man page (at least the one I checked :-) >>> specifies that pread() doesn't affect the file pointer. So I suppose >>> it should also not affect the buffer. That may make it hard to >>> implement it for text streams (which IIRC rely quite heavy on >>> buffering for their implementation), but it should be easy for >>> buffered streams: it should just be passed on to the underlying >>> unbuffered stream. >> >> Indeed, it should not affect the buffer. That's why I'm questioning the >> addition of this feature to buffered streams (whose whole point is their >> implicit buffer management). Also, there are implementation subtleties >> when e.g. reading from an area which overlaps the current buffer :-) >> >> As you pointed out, I think a reasonable solution to the race condition >> problem is to use several file descriptors. It may not work so well if >> you also write to the file, though. > > If you write and read a file from multiple threads you're crazy. > Or Dutch? :) -- Cheers. Mark Lawrence. From p.f.moore at gmail.com Mon Jul 23 17:18:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Jul 2012 16:18:11 +0100 Subject: [Python-ideas] Provide a 'key' argument for all() and any() builtins In-Reply-To: <CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com> References: <CAFYqXL8mzn35cyiY_-Sdr8awLrKvb4-absx0bcdWx_Hqpgrp3Q@mail.gmail.com> <B79E541B-CFD6-496D-B28F-85A9F32D7A59@gmail.com> <CAP7+vJLLhTY=z8HoPfJB5YJMOogUyFZgr=9-ps-8Ck34=1s66g@mail.gmail.com> Message-ID: <CACac1F-u=mkGQSeZFEVA0swdD4J4oq9dRjQcNKv7BCiP7rVddg@mail.gmail.com> On 23 July 2012 15:55, Guido van Rossum <guido at python.org> wrote: > On Sun, Jul 22, 2012 at 9:36 PM, Raymond Hettinger > <raymond.hettinger at gmail.com> wrote: >> >> On Jul 22, 2012, at 11:03 PM, Giampaolo Rodol? wrote: >> >> This would be similar to 'key' argument already available for min(), max() >> and sorted() and would let user decide what must be considered True and what >> not. >> >> There's no need. We already have: >> >> >>> all(predicate(x) for x in iterable) >> >>> any(predicate(x) for x in iterable) > > In addition, there's a reason why the key= parameter to sort, sorted, > max and min can't be replaced with a similar idiom: all these return > the *original* object(s) and the key= parameter affects only the > comparison; for example, try max(1,-2,3,-4, key=abs). > > Whereas for the current proposal, the value returned by the key would > also determine the outcome -- and then there is no advantage over > using the comprehension. More general, if you can first apply the > transformation (the key function) and then the aggregation function > (min/max/etc.), there's no need to combine the transformation and the > aggregation. But in all the cases that have a key= parameter, this is > not so. To put it yet another way, the key= parameter encapsulates the decorate-process-undecorate pattern, and that pattern doesn't apply to any/all. Paul. From ubershmekel at gmail.com Mon Jul 23 18:50:29 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 23 Jul 2012 19:50:29 +0300 Subject: [Python-ideas] Atomic file.get(offset, length) In-Reply-To: <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com> References: <BD328860-0320-415B-8FF2-1B0DB9AC1A08@whoosh.ca> <juf09k$njc$1@dough.gmane.org> <CAP7+vJL=x+agA9CLN1yPwzM74p68w8f6TYWF3kkLPwkd_PFhOg@mail.gmail.com> <CAMpsgwbjBcC1eeMFK+nC=DraG=5w9sm0_vhGx9OnNzGPb0zveA@mail.gmail.com> <CAP7+vJ+4P298_N_4vL=Cw3WAduRm5XN-Qk9+eSY2yJS5oSX8Eg@mail.gmail.com> <20120723014710.61fd5e12@pitrou.net> <CAP7+vJ+cv9mWUc=rT51_E+6vsYdBcOMShs=wS-eBNbkEwOasvA@mail.gmail.com> <20120723115216.33cc8ed5@pitrou.net> <CAP7+vJJF0-mC5GyruqvDuP5mozmL094y3JSg-gajak5OfNStXA@mail.gmail.com> Message-ID: <CANSw7KzUDSpasvUutZekxpaBdu+q9EBn7Vj9VG+sRFnWaEBzkQ@mail.gmail.com> On Mon, Jul 23, 2012 at 6:00 PM, Guido van Rossum <guido at python.org> wrote: > If you write and read a file from multiple threads you're crazy. > Perhaps, though I could imagine a fine-grained-locking DB doing this with constant sized data structures. Though that might be a good time to pull this one out: http://en.wikipedia.org/wiki/Readers%E2%80%93writer_lock -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120723/4a12fd4b/attachment.html> From ericsnowcurrently at gmail.com Tue Jul 24 02:59:21 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 23 Jul 2012 18:59:21 -0600 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com> Message-ID: <CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com> On Mon, Jul 16, 2012 at 10:28 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > Yeah, this is still valuable. However, at this point it would be a 3.4 > addition (_if_ it gets committed). Updating the patch in the tracker is on > my todo for the next few months. FWIW, the possibility of exposing something like __function__ in the function locals is less likely to garner support than the initial f_func approach. Said Guido[1]: <quote> If it were expected that people would start writing recursive calls using __function__ routinely, in situations where a name reference works, I'd be very unhappy with the new feature. (And if someone wants to make the argument that recursive calls using __function__ are actually better in some way I am willing to filibuster.) </quote> So there's a metric that f_func passes much more readily. That said, the f_func patch would also make "__function__" easier to implement if that feature passed muster. -eric [1] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html From ncoghlan at gmail.com Tue Jul 24 05:57:34 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jul 2012 13:57:34 +1000 Subject: [Python-ideas] How can I get own instance of generator in itself code In-Reply-To: <CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com> References: <33429676.215905.1342333139344.JavaMail.root@socbay.com> <CAMZYqRRC+dqLZNnBhKE0fiyqz1phd-FpcJ2GfmkP8YF0K1N9LA@mail.gmail.com> <CAEgL-ffr9qW00a-UNPMbW0UZTw=XfguY9gE62MWt3AQJAe859g@mail.gmail.com> <CALFfu7CeX67Z1FOp7PU2Q6DHBm+XwaSu47XDOqswkyf41qmr4g@mail.gmail.com> <CALFfu7DZALezM9EsFqkFWczgBcui9o7exLKXTiJOcxL0icTysQ@mail.gmail.com> Message-ID: <CADiSq7etDhjrsNuXssbtP7CgkOY05_qZ9KXcAMKMsKaJ2Z8m_g@mail.gmail.com> On Tue, Jul 24, 2012 at 10:59 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > On Mon, Jul 16, 2012 at 10:28 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote: >> Yeah, this is still valuable. However, at this point it would be a 3.4 >> addition (_if_ it gets committed). Updating the patch in the tracker is on >> my todo for the next few months. > > FWIW, the possibility of exposing something like __function__ in the > function locals is less likely to garner support than the initial > f_func approach. Said Guido[1]: > > <quote> > If it were expected that people would start writing recursive > calls using __function__ routinely, in situations where a name > reference works, I'd be very unhappy with the new feature. (And if > someone wants to make the argument that recursive calls using > __function__ are actually better in some way I am willing to > filibuster.) > </quote> > > So there's a metric that f_func passes much more readily. That said, > the f_func patch would also make "__function__" easier to implement if > that feature passed muster. Python 3 already has an implicit lexical __class__ reference (courtesy of PEP 3135) and it refers to the inner *undecorated* version of the class. References by name, on the other hand, refer to the postdecorated version bound in the outer scope (which may or may not be the same object, depending on the specific decorators). A PEP for 3.4 that proposes a mechanism that more cleanly incorporates __class__ into the language data model (thus addressing http://bugs.python.org/issue12370) would be most welcome. Such a PEP could also attempt to make a symmetry argument for a __def__ reference that refers directly to the undecorated function object (as it's a lexical reference, it's worth emphasising the keyword over the kind of object). An implicit reference to the *decorated* object would not be acceptable, as decorators need to be able to call the function, even if it relies on the implicit reference. (This is the same reason why __class__ refers to the undecorated version: so that the class definition is complete while the decorator is running, which means they can instantiate it and call methods that use the new shorthand style super() invocation) The module self reference aspect discussed in PEP 3130 should just be dropped as using "mod = importlib.import_module(__name__)" is enough to avoid the odd behaviour associated with trying to use "__import__" directly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fetchinson at googlemail.com Tue Jul 24 15:23:15 2012 From: fetchinson at googlemail.com (Daniel Fetchinson) Date: Tue, 24 Jul 2012 15:23:15 +0200 Subject: [Python-ideas] Python Package Collections Message-ID: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> Hi folks, here is an idea that came out of (not quite) thin air: Pypi, setuptools, distribute & Co are all very useful for finding, installing, upgrading, etc. of python packages. They even handle dependencies! Yaaaay! However, there is one more thing that could be built on top of them that would make them even more useful. This is the <drum roll please> Python Package Collections! This would be an online service to organize packages into collections allowing one to download, install, build, upgrade, etc. many packages at once. Say somebody finds package1, package2, package3, package4 very useful and figure they naturally belong together by topic. And so our test person always wants to have them all installed so when he downloads a new python version which will make his python arsenal not only 2.4 2.5 2.6 2.7 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages separately again but he can just install the collection with one command or one click. You might ask why can't our test person just do "install package1" then "install package2" then "install package3" then "install package4"? Well, certainly he could do that but if he found that these four packages nicely go together chances are others will find it handy to have them all too. So our brilliant test person will want to share his collection with others telling them essentially "if you like working with X" or "you frequently develop for Y" then "you can just install my collection and you will be ready to go"! Anybody could organize packages into collections and I bet there would be very popular ones like "mathematics" including all sorts of math packages from pypi, or "pdfgoodies" including everything pdf related that one typically needs for work with pdf. These don't have to depend on each other (dependencies are handled quite well by existing tools), they just need to have the property that they are very handy to have together. To be honest the idea is not mine, take a look at https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for firefox are organized into collections. It's basically the same idea but for python packages. This whole thing could be integrated into pypi or initially the idea could be tested on a separate site to see if it catches on. What do you all think? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown From ubershmekel at gmail.com Tue Jul 24 15:26:19 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 24 Jul 2012 16:26:19 +0300 Subject: [Python-ideas] Python Package Collections In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> Message-ID: <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com> On Tue, Jul 24, 2012 at 4:23 PM, Daniel Fetchinson < fetchinson at googlemail.com> wrote: > Hi folks, here is an idea that came out of (not quite) thin air: > > Pypi, setuptools, distribute & Co are all very useful for finding, > installing, upgrading, etc. of python packages. They even handle > dependencies! Yaaaay! > > However, there is one more thing that could be built on top of them > that would make them even more useful. This is the <drum roll please> > Python Package Collections! This would be an online service to > organize packages into collections allowing one to download, install, > build, upgrade, etc. many packages at once. Say somebody finds > package1, package2, package3, package4 very useful and figure they > naturally belong together by topic. And so our test person always > wants to have them all installed so when he downloads a new python > version which will make his python arsenal not only 2.4 2.5 2.6 2.7 > 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages > separately again but he can just install the collection with one > command or one click. > > You might ask why can't our test person just do "install package1" > then "install package2" then "install package3" then "install > package4"? Well, certainly he could do that but if he found that these > four packages nicely go together chances are others will find it handy > to have them all too. So our brilliant test person will want to share > his collection with others telling them essentially "if you like > working with X" or "you frequently develop for Y" then "you can just > install my collection and you will be ready to go"! > > Anybody could organize packages into collections and I bet there would > be very popular ones like "mathematics" including all sorts of math > packages from pypi, or "pdfgoodies" including everything pdf related > that one typically needs for work with pdf. These don't have to depend > on each other (dependencies are handled quite well by existing tools), > they just need to have the property that they are very handy to have > together. > > To be honest the idea is not mine, take a look at > https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for > firefox are organized into collections. It's basically the same idea > but for python packages. > > This whole thing could be integrated into pypi or initially the idea > could be tested on a separate site to see if it catches on. > > What do you all think? > > Cheers, > Daniel > > > -- > Psss, psss, put it down! - http://www.cafepress.com/putitdown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Sounds like a great idea for a separate site. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/655b338c/attachment.html> From ncoghlan at gmail.com Tue Jul 24 15:33:57 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jul 2012 23:33:57 +1000 Subject: [Python-ideas] Python Package Collections In-Reply-To: <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com> References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com> Message-ID: <CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com> On Tue, Jul 24, 2012 at 11:26 PM, Yuval Greenfield <ubershmekel at gmail.com> wrote: > On Tue, Jul 24, 2012 at 4:23 PM, Daniel Fetchinson > <fetchinson at googlemail.com> wrote: >> This whole thing could be integrated into pypi or initially the idea >> could be tested on a separate site to see if it catches on. > > Sounds like a great idea for a separate site. Certainly sounds like a concept worth exploring. Similar to sites like crate.io and pythonpackages, your best bet would be to start with a separate site that uses the PyPI APIs to access the relevant information. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From michelelacchia at gmail.com Tue Jul 24 16:06:24 2012 From: michelelacchia at gmail.com (Michele Lacchia) Date: Tue, 24 Jul 2012 07:06:24 -0700 (PDT) Subject: [Python-ideas] Python Package Collections In-Reply-To: <CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com> References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> <CANSw7KxSeAzTr0HgGscyMOq5LcQNf+EcZDrxUBi7mXheZhPbfw@mail.gmail.com> <CADiSq7cgG0dFLq3Uy_X51_Y-QE3tdt=xL63RjL9v90JJ6ykwiA@mail.gmail.com> Message-ID: <78662cc3-2b86-404a-9a9b-28b35a165711@googlegroups.com> Crate.io supports so-called lists of packages. But I don't know if you can share them. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/0a92de02/attachment.html> From bkabrda at redhat.com Tue Jul 24 15:55:40 2012 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Tue, 24 Jul 2012 09:55:40 -0400 (EDT) Subject: [Python-ideas] Python Package Collections In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> Message-ID: <924466335.1936859.1343138140069.JavaMail.root@redhat.com> ----- Original Message ----- > Hi folks, here is an idea that came out of (not quite) thin air: > > Pypi, setuptools, distribute & Co are all very useful for finding, > installing, upgrading, etc. of python packages. They even handle > dependencies! Yaaaay! > > However, there is one more thing that could be built on top of them > that would make them even more useful. This is the <drum roll please> > Python Package Collections! This would be an online service to > organize packages into collections allowing one to download, install, > build, upgrade, etc. many packages at once. Say somebody finds > package1, package2, package3, package4 very useful and figure they > naturally belong together by topic. And so our test person always > wants to have them all installed so when he downloads a new python > version which will make his python arsenal not only 2.4 2.5 2.6 2.7 > 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages > separately again but he can just install the collection with one > command or one click. > > You might ask why can't our test person just do "install package1" > then "install package2" then "install package3" then "install > package4"? Well, certainly he could do that but if he found that > these > four packages nicely go together chances are others will find it > handy > to have them all too. So our brilliant test person will want to share > his collection with others telling them essentially "if you like > working with X" or "you frequently develop for Y" then "you can just > install my collection and you will be ready to go"! > > Anybody could organize packages into collections and I bet there > would > be very popular ones like "mathematics" including all sorts of math > packages from pypi, or "pdfgoodies" including everything pdf related > that one typically needs for work with pdf. These don't have to > depend > on each other (dependencies are handled quite well by existing > tools), > they just need to have the property that they are very handy to have > together. > > To be honest the idea is not mine, take a look at > https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for > firefox are organized into collections. It's basically the same idea > but for python packages. > > This whole thing could be integrated into pypi or initially the idea > could be tested on a separate site to see if it catches on. > > What do you all think? > > Cheers, > Daniel > I think you can emulate this behaviour by creating a pypi "metapackage" - an empty package that only "install_requires" the others. Having said that, I agree that it might be nice to support this as a new feature (by more light-weight solution that the mentioned "metapackages"). -- Regards, Bohuslav "Slavek" Kabrda. From ianb at colorstudy.com Tue Jul 24 20:16:41 2012 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 24 Jul 2012 13:16:41 -0500 Subject: [Python-ideas] Python Package Collections In-Reply-To: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> References: <CADjSo4SRo8mPR-AoRdiNx1VQWdj3=JM5HW4JY96UubaVWG29KQ@mail.gmail.com> Message-ID: <CAHtfchqAUFeZi59CbGr8a=L2aY4K81QN3b8+KVESE8=TW+3Cxw@mail.gmail.com> You can also publish pip requirement files at a URL, and use: pip install -r http://pypackagecollections.org/scraping-tools.txt It works with upgrade/uninstall/etc. But typically if you want to get someone started on a new task using a new set of tools, you want to also generate or at least unpack starter files for them ? installing libraries isn't helpful if you don't know what they do or how to use them, or distinguish between libraries-you-use and libraries-your-libraries-use. Which is to say, I'm not surprised that a public index of requirement files hasn't emerged, because it makes more sense for specific starter guides to be written, and those guides may use other tools. On Tue, Jul 24, 2012 at 8:23 AM, Daniel Fetchinson < fetchinson at googlemail.com> wrote: > Hi folks, here is an idea that came out of (not quite) thin air: > > Pypi, setuptools, distribute & Co are all very useful for finding, > installing, upgrading, etc. of python packages. They even handle > dependencies! Yaaaay! > > However, there is one more thing that could be built on top of them > that would make them even more useful. This is the <drum roll please> > Python Package Collections! This would be an online service to > organize packages into collections allowing one to download, install, > build, upgrade, etc. many packages at once. Say somebody finds > package1, package2, package3, package4 very useful and figure they > naturally belong together by topic. And so our test person always > wants to have them all installed so when he downloads a new python > version which will make his python arsenal not only 2.4 2.5 2.6 2.7 > 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages > separately again but he can just install the collection with one > command or one click. > > You might ask why can't our test person just do "install package1" > then "install package2" then "install package3" then "install > package4"? Well, certainly he could do that but if he found that these > four packages nicely go together chances are others will find it handy > to have them all too. So our brilliant test person will want to share > his collection with others telling them essentially "if you like > working with X" or "you frequently develop for Y" then "you can just > install my collection and you will be ready to go"! > > Anybody could organize packages into collections and I bet there would > be very popular ones like "mathematics" including all sorts of math > packages from pypi, or "pdfgoodies" including everything pdf related > that one typically needs for work with pdf. These don't have to depend > on each other (dependencies are handled quite well by existing tools), > they just need to have the property that they are very handy to have > together. > > To be honest the idea is not mine, take a look at > https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for > firefox are organized into collections. It's basically the same idea > but for python packages. > > This whole thing could be integrated into pypi or initially the idea > could be tested on a separate site to see if it catches on. > > What do you all think? > > Cheers, > Daniel > > > -- > Psss, psss, put it down! - http://www.cafepress.com/putitdown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120724/d09e2ea1/attachment.html> From miki.tebeka at gmail.com Wed Jul 25 22:51:25 2012 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Wed, 25 Jul 2012 13:51:25 -0700 (PDT) Subject: [Python-ideas] compatibility triples for binary distributions In-Reply-To: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com> References: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com> Message-ID: <92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com> IMO implementation names should be compatible with PEP-421<http://www.python.org/dev/peps/pep-0421/>. Also instead of a cryptic long string, maybe some structured one?. For example (JSON): {"implementation":"cpython","machine":"x86_64"} It's longer but more readable IMO (and easier to parse). On Wednesday, July 25, 2012 7:55:34 AM UTC-7, Daniel Holth wrote: > > I am working on a specification that might be useful in the PKG-INFO > Platform or Supported-Platform tags. It tries to express which Python > versions a built distribution works on with a triple > (Implementation+Language Version, ABI, Architecture), anticipating an > ABI that is not tied to a single CPython release. > > A typical tag would be py27-noabi-noarch or cp33-cp33dmu-linux_x86_64. > > cp27-noabi-noarch is a package that requires CPython 2.7 but doesn't > include extension modules. > > The Python implementation is abbreviated. Each implementation has a > two-letter code: > > py: Generic Python > cp: CPython > ip: IronPython > pp: PyPy > jy: Jython > > concatenated with py_version_nodot ?27?. The minor version can be > omitted ?py2? or ?py3? when appropriate. > > The ABI tag is an abbreviated SOABI ?cp33m?, or, for ?pure Python? > packages, ?noabi?. For pypy it was more or less suggested that the ABI > would have to be a hash of the source code revision and the > translation/compilation options. > > The platform tag is distutils.util.get_platform() with all periods and > hyphens replaced with underscore, or the string ?noarch?. > > I use the term "built distribution" instead of "binary distribution" > because it includes pure python packages that for example have had > 2to3 run ahead of time and are installable without setup.py. > > Will it work? The ABI in particular may be so messy that it should be > lumped with architecture. What is the ABI for packages that use ctypes or > cffi? > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120725/bdb4be4e/attachment.html> From steve at pearwood.info Thu Jul 26 03:57:25 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 26 Jul 2012 11:57:25 +1000 Subject: [Python-ideas] compatibility triples for binary distributions In-Reply-To: <92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com> References: <4a1e9d73-5778-44a2-a934-cfca092fb976@googlegroups.com> <92fbab11-ce4e-4c8a-b7cb-3f79ca1e0bc2@googlegroups.com> Message-ID: <5010A405.10704@pearwood.info> Sorry for breaking threading, I actually wanted to reply to the post from Daniel Holth, but for some reason I don't have the original. Miki Tebeka wrote: > IMO implementation names should be compatible with PEP-421<http://www.python.org/dev/peps/pep-0421/>. > > > Also instead of a cryptic long string, maybe some structured one?. For > example (JSON): {"implementation":"cpython","machine":"x86_64"} > It's longer but more readable IMO (and easier to parse). +1 See below. > On Wednesday, July 25, 2012 7:55:34 AM UTC-7, Daniel Holth wrote: [...] >> The Python implementation is abbreviated. Each implementation has a >> two-letter code: >> >> py: Generic Python >> cp: CPython >> ip: IronPython >> pp: PyPy >> jy: Jython Who would be responsible for registering and tracking these two-letter codes? What counts as "generic python" -- surely if anything is "generic", it is CPython? I can see that you have missed out on a number of existing or experimental implementations, including: Stackless WPython ActivePython CLPython PyMite Python-iPod TinyPy Unladen Swallow CapPython HotPy Skulpt HoPe Berp among others. Some of these may be dead projects; others may be working but not maintained; some are actively maintained. Even if they only have a small number of users, they are still legitimate Python implementations. If there is need to distinguish implementations at all, surely there is a need to distinguish (say) TinyPy from CapPython? I think that this demonstrates the folly of using a central registry of codes for implementations. There are more implementations than just the "big four", and although some of them have the life-expectancy of mayflies, some of them go on for many years. -- Steven From matt at whoosh.ca Fri Jul 27 00:40:33 2012 From: matt at whoosh.ca (Matt Chaput) Date: Thu, 26 Jul 2012 18:40:33 -0400 Subject: [Python-ideas] Marvelous movable memoryview Message-ID: <5011C761.3060606@whoosh.ca> By which I mean a memoryview that lets you change the start and end offsets of its view of the underlying object (not modifying the underlying object). Let's say I want to slice a long string s into trigrams and write them to disk (or something). for i in xrange(0, len(s) - 3): x = s[i:i + 3] myfile.write(x) At each step the slice copies the bytes of the string, even though all I do is write them to disk. I could avoid copying with memoryviews... for i in xrange(0, len(s) - 3): x = memoryview(s)[i:i + 3] myfile.write(x) ...but this is actually much slower (3x slower in some quick tests). I'm guessing it's because of all the object creation (while string slicing probably uses fast paths). Shouldn't I be able to do this? m = memoryview(s) for i in xrange(0, len(s) - 3): m.start = i m.end = i + 3 myfile.write(m) Cheers, Matt From dholth at gmail.com Fri Jul 27 00:52:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 26 Jul 2012 18:52:31 -0400 Subject: [Python-ideas] Package collections Message-ID: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com> See http://good-py.appspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120726/0c7ac2db/attachment.html> From solipsis at pitrou.net Fri Jul 27 01:25:56 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 27 Jul 2012 01:25:56 +0200 Subject: [Python-ideas] Marvelous movable memoryview References: <5011C761.3060606@whoosh.ca> Message-ID: <20120727012556.77a3a998@pitrou.net> On Thu, 26 Jul 2012 18:40:33 -0400 Matt Chaput <matt at whoosh.ca> wrote: > By which I mean a memoryview that lets you change the start and end > offsets of its view of the underlying object (not modifying the > underlying object). > > Let's say I want to slice a long string s into trigrams and write them > to disk (or something). > > for i in xrange(0, len(s) - 3): > x = s[i:i + 3] > myfile.write(x) > > At each step the slice copies the bytes of the string, even though all I > do is write them to disk. To be honest, copying three bytes is a trivial operation compared to all the rest (interpreting bytecode, instantiating and destructing bytes objects, calling I/O routines, etc.). > Shouldn't I be able to do this? > > m = memoryview(s) > for i in xrange(0, len(s) - 3): > m.start = i > m.end = i + 3 > myfile.write(m) You wouldn't win anything over the simple bytes slicing approach, IMO. (even with PyPy or another interpreter) Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From solipsis at pitrou.net Fri Jul 27 01:26:45 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 27 Jul 2012 01:26:45 +0200 Subject: [Python-ideas] Package collections References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com> Message-ID: <20120727012645.3e465fd7@pitrou.net> On Thu, 26 Jul 2012 18:52:31 -0400 Daniel Holth <dholth at gmail.com> wrote: > See http://good-py.appspot.com Perhaps you sent your message too early, what specifically are you proposing? Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From ncoghlan at gmail.com Fri Jul 27 03:37:10 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Jul 2012 11:37:10 +1000 Subject: [Python-ideas] Marvelous movable memoryview In-Reply-To: <5011C761.3060606@whoosh.ca> References: <5011C761.3060606@whoosh.ca> Message-ID: <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com> On Fri, Jul 27, 2012 at 8:40 AM, Matt Chaput <matt at whoosh.ca> wrote: > I could avoid copying with memoryviews... > > for i in xrange(0, len(s) - 3): > x = memoryview(s)[i:i + 3] > myfile.write(x) > > ...but this is actually much slower (3x slower in some quick tests). I'm > guessing it's because of all the object creation (while string slicing > probably uses fast paths). memorview objects are pretty big - they have to store a lot of pointers and other objects that describe their view of the underlying buffer. Bytes objects, on the other hand, have minimal overhead. (These numbers are for 3.3, which uses a revamped memoryview implementation) >>> import sys >>> x = b"foo" >>> sys.getsizeof(x) 36 >>> sys.getsizeof(memoryview(x)) 184 There's also the execution speed overhead that comes from the indirection when accessing the contents. Thus, using views instead of copying really only starts to pay off once you're talking about comparatively large chunks of data: >>> x *= 1000 >>> sys.getsizeof(x) 3033 >>> sys.getsizeof(memoryview(x)) 184 > Shouldn't I be able to do this? > > m = memoryview(s) > for i in xrange(0, len(s) - 3): > m.start = i > m.end = i + 3 > myfile.write(m) No, because making memorviews mutable would be a huge increase in complexity (and they're complex enough already - only with Stefan Krah's work in 3.3 have we finally worked most of the kinks out of the implementation). What you *can* do with a memoryview, though, is slice it, and the resulting object will be a memoryview that references a subset of the original object. This can be done with full slicing flexibility in 3.3, or in a more limited fashion in earlier versions. For example, processing a data sequence in chunks in 3.3 without copying and without inadvertently keeping a potentially large data object alive (and/or locked into immutability) by hanging on to a buffer reference: chunk_len = 512 # For small chunks, copying is likely faster. Measure it! with memorview(data) as m: for offset in range(0, len(m), chunk_len): with m[offset:offset+chunk_len] as x: process_chunk(x) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jul 27 03:50:03 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Jul 2012 11:50:03 +1000 Subject: [Python-ideas] Package collections In-Reply-To: <20120727012645.3e465fd7@pitrou.net> References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com> <20120727012645.3e465fd7@pitrou.net> Message-ID: <CADiSq7cVSsQK+SihQH-FOmkdHcfOkZPtXj+gxGZYwWHGsFZMww@mail.gmail.com> On Fri, Jul 27, 2012 at 9:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 26 Jul 2012 18:52:31 -0400 > Daniel Holth <dholth at gmail.com> wrote: >> See http://good-py.appspot.com > > Perhaps you sent your message too early, what specifically are you > proposing? I don't think it's a specific proposal yet, I believe it is just a "here's a follow up for those that were reading the previous thread". A little more prose in the OP would have helped make that clear, though... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jul 27 07:41:08 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 27 Jul 2012 07:41:08 +0200 Subject: [Python-ideas] Package collections References: <CAG8k2+7dqLhN1MZbd660q+KrWDw=dONG=FDhEVHY7vTr2A9u0A@mail.gmail.com> <20120727012645.3e465fd7@pitrou.net> <CADiSq7cVSsQK+SihQH-FOmkdHcfOkZPtXj+gxGZYwWHGsFZMww@mail.gmail.com> Message-ID: <20120727074108.3b68ce86@pitrou.net> On Fri, 27 Jul 2012 11:50:03 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > On Fri, Jul 27, 2012 at 9:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > On Thu, 26 Jul 2012 18:52:31 -0400 > > Daniel Holth <dholth at gmail.com> wrote: > >> See http://good-py.appspot.com > > > > Perhaps you sent your message too early, what specifically are you > > proposing? > > I don't think it's a specific proposal yet, I believe it is just a > "here's a follow up for those that were reading the previous thread". Ah, right, I hadn't seen Daniel's message was a followup to another thread. It came unthreaded here. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From lists at cheimes.de Fri Jul 27 14:26:19 2012 From: lists at cheimes.de (Christian Heimes) Date: Fri, 27 Jul 2012 14:26:19 +0200 Subject: [Python-ideas] Marvelous movable memoryview In-Reply-To: <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com> References: <5011C761.3060606@whoosh.ca> <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com> Message-ID: <juu1db$d82$1@dough.gmane.org> Am 27.07.2012 03:37, schrieb Nick Coghlan: > Thus, using views instead of copying really only starts to pay off > once you're talking about comparatively large chunks of data: > >>>> x *= 1000 >>>> sys.getsizeof(x) > 3033 >>>> sys.getsizeof(memoryview(x)) > 184 It can pay off A LOT. I've recently added PEP 3118 buffer support to my smc.freeimage library (Cython wrapper of FreeImage and LCMS). smc.freeimage needs 0.006 sec for the loop as it returns a writable buffer of the internal pixel data (zero copy). PIL runs 1.165 sec as it uses the old buffer interface and copies its data every time. Benchmark code: https://bitbucket.org/tiran/smc.freeimage/src/b97e13e9f04c/contrib/benchmark.py RW_COUNT = 300 tiff = Image(TIFF) start = time() for i in xrange(RW_COUNT): arr = asarray(tiff) # change last BGR -> RGB arr = arr[..., ::-1] bytescale(arr, 64, 192) end = time() - start tiff = pil_open(TIFF) tiff.load() start = time() for i in xrange(RW_COUNT): arr = asarray(tiff) bytescale(arr, 64, 192) end = time() - start > No, because making memorviews mutable would be a huge increase in > complexity (and they're complex enough already - only with Stefan > Krah's work in 3.3 have we finally worked most of the kinks out of the > implementation). +1 for Nick's -1 The new buffer interface and memoryview's are already very complex. I haven't found a library yet that supports all edge cases. Even NumPy doesn't support multidimensional and non-contiguous buffer with suboffsets. I wanted to use a sub-offset of 3 on the last dimension with a negative stride on the last dimension to convert BGR to RGB inside the buffer code. It didn't work because NumPy ignore the information. Christian From sturla at molden.no Sat Jul 28 23:09:13 2012 From: sturla at molden.no (Sturla Molden) Date: Sat, 28 Jul 2012 23:09:13 +0200 Subject: [Python-ideas] Marvelous movable memoryview In-Reply-To: <juu1db$d82$1@dough.gmane.org> References: <5011C761.3060606@whoosh.ca> <CADiSq7f1ndXeUXfuBaUsxmxygspC5GCGa7i8v+0jsSeXMGwwew@mail.gmail.com> <juu1db$d82$1@dough.gmane.org> Message-ID: <501454F9.2040802@molden.no> Den 27.07.2012 14:26, skrev Christian Heimes: > The new buffer interface and memoryview's are already very complex. I > haven't found a library yet that supports all edge cases. Even NumPy > doesn't support multidimensional and non-contiguous buffer with > suboffsets. I wanted to use a sub-offset of 3 on the last dimension > with a negative stride on the last dimension to convert BGR to RGB > inside the buffer code. It didn't work because NumPy ignore the > information. For those who write C extensions with Cython, it might be useful to know that it can compile "Python-like" code that uses PEP 3118 buffers to very efficient C code: http://docs.cython.org/src/userguide/memoryviews.html On a simple test posted to the cython-user list, https://github.com/jakevdp/memview_benchmarks I only got 2.2 % increased run-time compared to plain C pointer arithmetics. That involved creating two million PEP 3118 buffer objects and computing one million dot products with length 1000. This takes most of the complexity of using PEP 3118 buffers in C extensions away, and allows us to write C extensions with a syntax that are very similar to Python with NumPy arrays. (On the other hand, the old "NumPy array syntax" in Cython is considerably slower, particularly when slicing the arrays.) Sturla From x at jwp.io Sun Jul 29 02:47:31 2012 From: x at jwp.io (James William Pye) Date: Sat, 28 Jul 2012 17:47:31 -0700 Subject: [Python-ideas] Python Environment Initialization Scripts Message-ID: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist. Some quick searches on -ideas and -dev didn't reveal anything either. Has the idea of [python] environment initialization been pitched/rejected before or does it already exist? Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization. Without modifying libpython, I can achieve what I'm trying to do by: * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()") * Overriding the python3 executable (not desirable; consider embedding) * Provide an alternate executable. (not desirable; again embedding and lack of availability in scripts pointing directly to python3) * LD_PRELOAD overrides? (portability problems abound, I imagine. mangling seems preferable) * Perform installation in the dependent (duplication of installation in each dependent) [1] https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/CCHe8_WHRt8 From ericsnowcurrently at gmail.com Sun Jul 29 02:56:53 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 28 Jul 2012 18:56:53 -0600 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> Message-ID: <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> On Sat, Jul 28, 2012 at 6:47 PM, James William Pye <x at jwp.io> wrote: > I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist. > Some quick searches on -ideas and -dev didn't reveal anything either. > > Has the idea of [python] environment initialization been pitched/rejected before or does it already exist? > > Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization. > > Without modifying libpython, I can achieve what I'm trying to do by: > > * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()") No need to modify site.py. Either create a sitecustomize.py in your site-packages. Here are some references: http://docs.python.org/library/site.html http://www.python.org/dev/peps/pep-0370 Nothing new is needed. As to metapath hooks, one alternative is to make a helper function that puts the hooks in place (or otherwise manages your hooks). Then you can call that API in each script that needs those hooks. -eric From x at jwp.io Sun Jul 29 03:54:43 2012 From: x at jwp.io (James William Pye) Date: Sat, 28 Jul 2012 18:54:43 -0700 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> Message-ID: <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > No need to modify site.py. Either create a sitecustomize.py in your > site-packages. I had recently forgotten about that. Thanks. =) However, from the documentation: "After these path manipulations, an attempt is made to import a module named site customize," "which can perform arbitrary site-specific customizations. It is typically created by a system "administrator in the site-packages directory." The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py. In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon `pip install metapathawesomeness` which may already be in use by the system administrator. That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way. > As to metapath hooks, one alternative is to make a helper function > that puts the hooks in place (or otherwise manages your hooks). Then > you can call that API in each script that needs those hooks. Yeah, I referred to that in my list of alternatives. My gripe with it is the duplication across the set of dependent scripts/packages. However, it would seem to be the most appropriate solution ATM. =\ From ericsnowcurrently at gmail.com Sun Jul 29 05:17:47 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 28 Jul 2012 21:17:47 -0600 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> Message-ID: <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> On Sat, Jul 28, 2012 at 7:54 PM, James William Pye <x at jwp.io> wrote: > > On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: >> No need to modify site.py. Either create a sitecustomize.py in your >> site-packages. > > I had recently forgotten about that. Thanks. =) > > However, from the documentation: > > "After these path manipulations, an attempt is made to import a module named site customize," > "which can perform arbitrary site-specific customizations. It is typically created by a system > "administrator in the site-packages directory." > > The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py. > > In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon > `pip install metapathawesomeness` which may already be in use by the system administrator. > That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way. That's where PEP 370 comes in. Stick your sitecustomize.py into ~/.local/lib/pythonX.Y/site-packages. Then it only applies when python is run as you. -eric From x at jwp.io Sun Jul 29 07:22:19 2012 From: x at jwp.io (James William Pye) Date: Sat, 28 Jul 2012 22:22:19 -0700 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> Message-ID: <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io> On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > That's where PEP 370 comes in. Stick your sitecustomize.py into > ~/.local/lib/pythonX.Y/site-packages. Then it only applies when > python is run as you. But it's not desirable for it to be limited one user... The situation is that the meta_path hook is provided in a package that is depended on by other packages that contain modules that are importable by the hook. It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the system administrator's sitecustomize or the user's sitecustomize. No? Now, that's not to say that I'm unwilling to be inappropriate. >8) From ncoghlan at gmail.com Sun Jul 29 08:57:01 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 29 Jul 2012 16:57:01 +1000 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io> Message-ID: <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com> On Sun, Jul 29, 2012 at 3:22 PM, James William Pye <x at jwp.io> wrote: > On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: >> That's where PEP 370 comes in. Stick your sitecustomize.py into >> ~/.local/lib/pythonX.Y/site-packages. Then it only applies when >> python is run as you. > > But it's not desirable for it to be limited one user... > > The situation is that the meta_path hook is provided in a package that is depended on > by other packages that contain modules that are importable by the hook. > > It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the > system administrator's sitecustomize or the user's sitecustomize. No? > > Now, that's not to say that I'm unwilling to be inappropriate. >8) The short answer is that, no, a coherent proposal for application specific preinitialisation hasn't been put forward as a PEP. The slightly longer answer is that there are related problems in terms of initialising debuggers, warnings, logging, faulthandler, coverage, profiling, etc, during development where it would be incredibly convenient to be able to execute a short snippet of Python code prior to execution. See http://bugs.python.org/issue14803 (I'm nervous about the security implications of a PYTHONRUNFIRST environment variable that is enabled by default, but the idea of a -C option doesn't pose any such concerns) In general though, messing with process global state (including the import system) implicitly is just *not cool*. It leads to applications that have a dependency on a particular tool being present, but don't actually make sure that it is available. Thus you end up with manual instructions that say "before running this script, make sure infrastructure X is in place to automagically take care of things I should be doing myself". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From x at jwp.io Sun Jul 29 11:44:55 2012 From: x at jwp.io (James William Pye) Date: Sun, 29 Jul 2012 02:44:55 -0700 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io> <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com> Message-ID: <C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io> On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > The short answer is that, no, a coherent proposal for application > specific preinitialisation hasn't been put forward as a PEP. =) > The slightly longer answer is that there are related problems in terms of > initialising debuggers, warnings, logging, faulthandler, coverage, > profiling, etc, during development where it would be incredibly > convenient to be able to execute a short snippet of Python code prior > to execution. Yeah, that's been painful. > See http://bugs.python.org/issue14803 (I'm nervous about the security > implications of a PYTHONRUNFIRST environment variable that is enabled > by default, Rightly so. > but the idea of a -C option doesn't pose any such > concerns) Cool. I implemented something like that for "-m postgresql.bin.pg_python"[1] Never ended up using it much. But I could see myself using it with the Python executable: alias py='python3 -C ?' Tucking that into a sh source file that sets up my dev context. > In general though, messing with process global state (including the > import system) implicitly is just *not cool*. It leads to applications > that have a dependency on a particular tool being present, but don't > actually make sure that it is available. Thus you end up with manual > instructions that say "before running this script, make sure > infrastructure X is in place to automagically take care of things I > should be doing myself". Even in the case of some designated mechanism where there is some attempt to expose/record changes to process global state so that it can be easily referenced by a dependency, is there a win over making an explicit attempt to validate that the state change occurred by the dependency? I think in this particular case involving meta_path, there would be a small win with regards to being able to perform the check at package installation time as opposed to every time the package is imported, given a "designated mechanism" of sorts. Essentially, the package being installed requests assurance that a state change *will* occur upon Python startup. [1] https://github.com/jwp/py-postgresql/blob/v1.1/postgresql/python/command.py From shibturn at gmail.com Mon Jul 30 18:44:09 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Mon, 30 Jul 2012 17:44:09 +0100 Subject: [Python-ideas] Better support for finalization with weakrefs Message-ID: <jv6dl2$v8r$1@dough.gmane.org> I would like to see better support for the use of weakrefs callbacks for object finalization. The current issues with weakref callbacks include: 1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles). When it is an option, using a __del__ method is far less hassle. 2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None. 3. Sometimes you want the callback to be called at program exit even if the referent is still alive. 4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?) (Note that problems 2-4 also apply to using __del__ methods.) If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be >>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny! Prototype at https://gist.github.com/3208245 Cheers, Richard From solipsis at pitrou.net Mon Jul 30 19:27:34 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Jul 2012 19:27:34 +0200 Subject: [Python-ideas] Better support for finalization with weakrefs References: <jv6dl2$v8r$1@dough.gmane.org> Message-ID: <20120730192734.73454970@pitrou.net> Hi, On Mon, 30 Jul 2012 17:44:09 +0100 Richard Oudkerk <shibturn at gmail.com> wrote: > > 2. Callbacks invoked during interpreter shutdown are troublesome. For > instance they can throw exceptions because module globals have been > replaced by None. A solution was proposed a long time ago at http://bugs.python.org/issue812369 > 4. Exceptions thrown in callbacks do not produce a traceback. This > can make debugging rather awkward. (Maybe this is because printing > tracebacks is problematic during interpreter shutdown?) No, it's just because PyErr_WriteUnraisable() doesn't print the traceback. We could just fix that. > (Note that problems 2-4 also apply to using __del__ methods.) > > If possible I would like to see the weakref module provide a finalize > class to address these issues. Trivial usage might be > > >>> class Kenny: pass > ... > >>> kenny = Kenny() > >>> finalize(kenny, print, "you killed kenny!") > <finalize.finalize object at 0xffed4f2c> > >>> del kenny > you killed kenny! Looks good! Regards Antoine. -- Software development and contracting: http://pro.pitrou.net From ericsnowcurrently at gmail.com Tue Jul 31 07:49:06 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 30 Jul 2012 23:49:06 -0600 Subject: [Python-ideas] only raise ImportError out of imports Message-ID: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com> Currently, if module 'eggs' has bad syntax and you import it from module 'spam', then you will get a SyntaxError in module spam: --------------------------------------------------------- $ cat > eggs.py << EOF a + EOF $ cat > spam.py << EOF import eggs EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax --------------------------------------------------------- My proposal is that, instead, an error during import always results in an ImportError. So in the above example, the error would have been an ImportError rather than a SyntaxError. However, rather than ignore the cause of the import error, we would use exception chaining to save the SyntaxError. To get this effect currently would look something like this: --------------------------------------------------------- $ cat > spam.py << EOF try: import eggs except SyntaxError as e: raise ImportError("failed to import eggs") from e EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax The above exception was the direct cause of the following exception: Traceback (most recent call last): File "spam.py", line 4, in <module> raise ImportError("failed to import eggs") from e ImportError: failed to import eggs --------------------------------------------------------- With exception chaining the SyntaxError is stored on the ImportError: --------------------------------------------------------- $ cat > ham.py << EOF try: import spam except ImportError as e: print("cause: {}".format(e.__cause__)) print("context: {}".format(e.__context__)) EOF $ python ham.py cause: invalid syntax (eggs.py, line 1) context: invalid syntax (eggs.py, line 1) --------------------------------------------------------- The main benefit of this change would be to isolate errors during import to the module where they happen. The main challenge would be with backward compatibility, so I won't hold my breath to see this in Python 3. While all feedback is welcome, I'd particularly like to hear from those that actually use non-ImportError exceptions that come out of the import statement. Thanks! -eric From ncoghlan at gmail.com Tue Jul 31 08:00:44 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2012 16:00:44 +1000 Subject: [Python-ideas] only raise ImportError out of imports In-Reply-To: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com> References: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com> Message-ID: <CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com> On Tue, Jul 31, 2012 at 3:49 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > The main benefit of this change would be to isolate errors during > import to the module where they happen. The main challenge would be > with backward compatibility, so I won't hold my breath to see this in > Python 3. > > While all feedback is welcome, I'd particularly like to hear from > those that actually use non-ImportError exceptions that come out of > the import statement. Thanks! The problem specifically with doing this for SyntaxError is that you would break every interactive interpreter out there (including the default one). They rely on seeing the SyntaxError directly in order to highlight the offending piece of syntax. Also dig into pkgutil - that broke when I started tinkering with this kind of thing while migrating it over to using importlib. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericsnowcurrently at gmail.com Tue Jul 31 08:14:30 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 31 Jul 2012 00:14:30 -0600 Subject: [Python-ideas] only raise ImportError out of imports In-Reply-To: <CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com> References: <CALFfu7AtHLT2Unw4DPtUX+oNKLgbuhq96cmH3P4iYT_daYuuHg@mail.gmail.com> <CADiSq7cENEeg5L7MA_j6e0wwa=aT8njtjO0Hdp4UNSFFg=CZHA@mail.gmail.com> Message-ID: <CALFfu7Dh0weJ88uSxkGG5BM=eBRAbZG94Vn1HKTsgsJirvSUVA@mail.gmail.com> On Tue, Jul 31, 2012 at 12:00 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Tue, Jul 31, 2012 at 3:49 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: >> The main benefit of this change would be to isolate errors during >> import to the module where they happen. The main challenge would be >> with backward compatibility, so I won't hold my breath to see this in >> Python 3. >> >> While all feedback is welcome, I'd particularly like to hear from >> those that actually use non-ImportError exceptions that come out of >> the import statement. Thanks! > > The problem specifically with doing this for SyntaxError is that you > would break every interactive interpreter out there (including the > default one). They rely on seeing the SyntaxError directly in order to > highlight the offending piece of syntax. > > Also dig into pkgutil - that broke when I started tinkering with this > kind of thing while migrating it over to using importlib. Cool. I'll have a look. -eric From ironfroggy at gmail.com Tue Jul 31 15:28:53 2012 From: ironfroggy at gmail.com (Calvin Spealman) Date: Tue, 31 Jul 2012 09:28:53 -0400 Subject: [Python-ideas] Better support for finalization with weakrefs In-Reply-To: <jv6dl2$v8r$1@dough.gmane.org> References: <jv6dl2$v8r$1@dough.gmane.org> Message-ID: <CAGaVwhTQQmWKhdgUdt0VESSY=nhnj-E8rVcUkM_8BeCW-RU-hQ@mail.gmail.com> On Mon, Jul 30, 2012 at 12:44 PM, Richard Oudkerk <shibturn at gmail.com> wrote: > I would like to see better support for the use of weakrefs callbacks > for object finalization. > > The current issues with weakref callbacks include: > > 1. They are rather low level, and working out how to use them > correctly requires a bit of head scratching. One must find > somewhere to store the weakref till after the referent is dead, and > without accidentally keeping the referent alive. Then one must > ensure that the callback frees the weakref (without leaving any > remnant ref-cycles). > > When it is an option, using a __del__ method is far less hassle. > > 2. Callbacks invoked during interpreter shutdown are troublesome. For > instance they can throw exceptions because module globals have been > replaced by None. > > 3. Sometimes you want the callback to be called at program exit even > if the referent is still alive. > > 4. Exceptions thrown in callbacks do not produce a traceback. This > can make debugging rather awkward. (Maybe this is because printing > tracebacks is problematic during interpreter shutdown?) > > (Note that problems 2-4 also apply to using __del__ methods.) > > If possible I would like to see the weakref module provide a finalize > class to address these issues. Trivial usage might be > > >>> class Kenny: pass > ... > >>> kenny = Kenny() > >>> finalize(kenny, print, "you killed kenny!") > <finalize.finalize object at 0xffed4f2c> > >>> del kenny > you killed kenny! > > Prototype at https://gist.github.com/3208245 I like how simple it is, but it might be too simple. That is, shouldn't we have a way to unregister the callback? > Cheers, > > Richard > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From x at jwp.io Tue Jul 31 23:59:24 2012 From: x at jwp.io (James William Pye) Date: Tue, 31 Jul 2012 14:59:24 -0700 Subject: [Python-ideas] Python Environment Initialization Scripts In-Reply-To: <C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io> References: <808F1494-5BC8-4DB5-BC68-31B6F558B372@jwp.io> <CALFfu7CGOgK73XUSkFOth_Oog5_5yaLdTjrSy6wOB-fazXc2aQ@mail.gmail.com> <95A84799-151A-4AA9-B1FC-68346309C602@jwp.io> <CALFfu7A+kfS2LAVJdNr97d1OFM9PsUatrQ+87=r7fvVjPs6kxQ@mail.gmail.com> <C0DD22F1-7E4B-4601-89F6-88308556BCF4@jwp.io> <CADiSq7dre1PY51AWq-iNTzxU14S2YufRA144=Qkp2D8XZv2bFQ@mail.gmail.com> <C526AD43-C385-482E-A0FF-F908720E93A1@jwp.io> Message-ID: <B5E2A6BA-128E-465F-9330-B76DA8BDDC9C@jwp.io> On Jul 29, 2012, at 2:44 AM, James William Pye <x at jwp.io> wrote: > On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> The short answer is that, no, a coherent proposal for application >> specific preinitialisation hasn't been put forward as a PEP. sigh. *.pth files in site-packages/ can do it. They can run imports.. I think this is the second time I've been surprised by that? =\