From abarnert at yahoo.com Sun Dec 1 00:42:31 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 30 Nov 2013 15:42:31 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <5298390B.30506@canterbury.ac.nz> References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: On Nov 28, 2013, at 22:49, Greg Ewing wrote: > spir wrote: >> (But then, possibly, some would wonder why this new ord(s,i) is not a string method ;-) > > For the same reason that the existing ord() function isn't > a string method, whatever that is! http://docs.python.org/2/faq/design.html#why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list The short version is: historical reasons, and no reason to change it compelling enough to be worth the costs in backward compat, bikeshedding, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 1 00:48:57 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 30 Nov 2013 15:48:57 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: On Sat, Nov 30, 2013 at 3:42 PM, Andrew Barnert wrote: > On Nov 28, 2013, at 22:49, Greg Ewing wrote: > > spir wrote: > > (But then, possibly, some would wonder why this new ord(s,i) is not a > string method ;-) > > > For the same reason that the existing ord() function isn't > a string method, whatever that is! > > > > http://docs.python.org/2/faq/design.html#why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list > > The short version is: historical reasons, and no reason to change it > compelling enough to be worth the costs in backward compat, bikeshedding, > etc. > That FAQ entry does not do the real motivation justice. While OO "purists" may argue about it, to me it's obvious that in quite a few cases the function spelling is more readable than the method spelling. I found it more readable 23 years ago, and I still find it more readable today. So it's not historical reasons to me, even if not everybody agrees (obviously :-). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Dec 1 02:23:17 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 30 Nov 2013 20:23:17 -0500 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: On 11/30/2013 6:48 PM, Guido van Rossum wrote: > > On Sat, Nov 30, 2013 at 3:42 PM, Andrew Barnert > > wrote: > > On Nov 28, 2013, at 22:49, Greg Ewing > > wrote: > >> spir wrote: >>> (But then, possibly, some would wonder why this new ord(s,i) is >>> not a string method ;-) ord and chr are inverse functions. Would you have chr be an int method? (Even if yes, history comes into play here as ints did not have methods until 2.2(?, or close to that). >> For the same reason that the existing ord() function isn't >> a string method, whatever that is! > > http://docs.python.org/2/faq/design.html#why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list > > The short version is: historical reasons, and no reason to change it > compelling enough to be worth the costs in backward compat, > bikeshedding, etc. > > > That FAQ entry does not do the real motivation justice. While OO > "purists" may argue about it, to me it's obvious that in quite a few > cases the function spelling is more readable than the method spelling. I > found it more readable 23 years ago, and I still find it more readable > today. So it's not historical reasons to me, even if not everybody > agrees (obviously :-). Sometimes functions are more functional (pun intented) than methods. len(ob) checks that ob.__len__() returns an int or something that can be 'interpreted' as such. reversed(itable) checks for itable.__reversed__ before using a default version. -- Terry Jan Reedy From greg at krypto.org Sun Dec 1 03:03:21 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 30 Nov 2013 18:03:21 -0800 Subject: [Python-ideas] Replacing the if __name__ == "__main__" idiom (was Re: making a module callable) In-Reply-To: References: <20131125071932.GA65531@cskk.homeip.net> <20131125141220.GE2085@ando> <20131125144244.1cb160f0@anarchist> <20131125171603.1812e9fd@anarchist> <5294CD94.6050008@gmail.com> Message-ID: On Wed, Nov 27, 2013 at 9:58 AM, Gregory Salvan wrote: > my cent: > I think it's most often a bad practice (whereas it's convenient) to mix > "executable" and "importable" code. > Providing an easier way to do it, may encourage a bad practice. > > I would prefer a solution which encourage separation of "executable" and > "importable" code. > The only way to do that is to force people to put their executable code in a __main__.py file and prevent class and function definitions within that... Gross. People need the test for main or not (imported or not) because they write unit tests for _all_ of their code, including their main program. Just because it can be imported does not mean it is a library that anyone outside of their tests _should_ import. Sure, there are some python stdlib modules and others that work as an importable library or via python -m modulename to run a program but I wouldn't call that the common case. I would not want that to stop working either. -gps > > > > 2013/11/26 Andrew Barnert > >> On Nov 26, 2013, at 8:34, Alan Cristhian Ruiz >> wrote: >> >> > I think the need to change * if __ name__ == "__main__": * is >> capricious and obsessive. The current rule is better than anything that has >> been suggested so far. I never had any problems with the * if __ name__ == >> "__main__": *. Also in python there are other things much more difficult to >> learn and use, such as metaclasses. >> >> Although I agree with your main point, I don't think that's a very good >> argument. >> >> __main__ is something novices have to learn early and use in code >> regularly; metaclasses are something only experienced developers use, and >> not that often (and that's even if you count using stdlib metaclasses to, >> e.g., create ABCs, which doesn't really require you to understand how they >> work). It's perfectly reasonable for an "expert" feature to be more >> difficult to learn than a novice feature. >> >> Also, Python doesn't have a queue of improvements to be scheduled to a >> team of developers. Things get improved if someone is motivated enough to >> write the code and drive the idea to consensus and/or BDFL approval. So, >> improving this would have very little bearing on improving things you care >> about more. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sun Dec 1 03:04:05 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 30 Nov 2013 18:04:05 -0800 Subject: [Python-ideas] Replacing the if __name__ == "__main__" idiom (was Re: making a module callable) In-Reply-To: References: <20131125071932.GA65531@cskk.homeip.net> <20131125141220.GE2085@ando> Message-ID: On Mon, Nov 25, 2013 at 6:22 AM, Nick Coghlan wrote: > On 26 November 2013 00:12, Steven D'Aprano wrote: > > > > I'm wondering if we should add a link to Steven's post from > http://www.python.org/dev/peps/pep-0299/ (and perhaps even update the > PEP text itself) > > As the status of PEP 299 shows, Guido has rejected the idea of a > special main function before, but I think Steven's post does a better > job of spelling out "Why not?" than any of the previous discussions. > +1 (if you haven't already done it) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Dec 1 05:33:08 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 30 Nov 2013 20:33:08 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> On Nov 30, 2013, at 15:48, Guido van Rossum wrote: > > On Sat, Nov 30, 2013 at 3:42 PM, Andrew Barnert wrote: >> On Nov 28, 2013, at 22:49, Greg Ewing wrote: >> >>> spir wrote: >>>> (But then, possibly, some would wonder why this new ord(s,i) is not a string method ;-) >>> >>> For the same reason that the existing ord() function isn't >>> a string method, whatever that is! >> >> http://docs.python.org/2/faq/design.html#why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list >> >> The short version is: historical reasons, and no reason to change it compelling enough to be worth the costs in backward compat, bikeshedding, etc. > > That FAQ entry does not do the real motivation justice. While OO "purists" may argue about it, to me it's obvious that in quite a few cases the function spelling is more readable than the method spelling. I found it more readable 23 years ago, and I still find it more readable today. So it's not historical reasons to me, even if not everybody agrees (obviously :-). I kind of read it the other way around--making _too many_ things methods is the historical quirk, because at some point (probably after making str and tuple and so on into full types) you and the other core devs went a bit farther with the OO-style "everything should be a method if at all possible" than you would have earlier or later in history. But I may have been assuming too much. A lot of other "multi-paradigm" languages went through a phase like that and have a historical legacy, like C++'s ridiculous string class and JavaScript's excess of Array methods. But Python does seen to have ended up with a lot fewer quirky choices than those languages. Still, I think the various index/find/count/etc. methods didn't all need to be methods. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 1 06:03:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Dec 2013 15:03:51 +1000 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> Message-ID: On 1 December 2013 14:33, Andrew Barnert wrote: > On Nov 30, 2013, at 15:48, Guido van Rossum wrote: > That FAQ entry does not do the real motivation justice. While OO "purists" > may argue about it, to me it's obvious that in quite a few cases the > function spelling is more readable than the method spelling. I found it more > readable 23 years ago, and I still find it more readable today. So it's not > historical reasons to me, even if not everybody agrees (obviously :-). > > I kind of read it the other way around--making _too many_ things methods is > the historical quirk, because at some point (probably after making str and > tuple and so on into full types) you and the other core devs went a bit > farther with the OO-style "everything should be a method if at all possible" > than you would have earlier or later in history. > > But I may have been assuming too much. A lot of other "multi-paradigm" > languages went through a phase like that and have a historical legacy, like > C++'s ridiculous string class and JavaScript's excess of Array methods. But > Python does seen to have ended up with a lot fewer quirky choices than those > languages. Still, I think the various index/find/count/etc. methods didn't > all need to be methods. One of the other things to keep in mind is that many of the builtin function vs method choices predated the iterator protocol. A few have since been updated to work with it (list.sort -> sorted, list.reverse -> reversed), but others live on as Sequence ABC methods rather than as builtins or itertools. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Sun Dec 1 06:53:54 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 30 Nov 2013 21:53:54 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> Message-ID: On Nov 30, 2013, at 21:03, Nick Coghlan wrote: > On 1 December 2013 14:33, Andrew Barnert wrote: >> On Nov 30, 2013, at 15:48, Guido van Rossum wrote: >> That FAQ entry does not do the real motivation justice. While OO "purists" >> may argue about it, to me it's obvious that in quite a few cases the >> function spelling is more readable than the method spelling. I found it more >> readable 23 years ago, and I still find it more readable today. So it's not >> historical reasons to me, even if not everybody agrees (obviously :-). >> >> I kind of read it the other way around--making _too many_ things methods is >> the historical quirk, because at some point (probably after making str and >> tuple and so on into full types) you and the other core devs went a bit >> farther with the OO-style "everything should be a method if at all possible" >> than you would have earlier or later in history. >> >> But I may have been assuming too much. A lot of other "multi-paradigm" >> languages went through a phase like that and have a historical legacy, like >> C++'s ridiculous string class and JavaScript's excess of Array methods. But >> Python does seen to have ended up with a lot fewer quirky choices than those >> languages. Still, I think the various index/find/count/etc. methods didn't >> all need to be methods. > > One of the other things to keep in mind is that many of the builtin > function vs method choices predated the iterator protocol. A few have > since been updated to work with it (list.sort -> sorted, list.reverse > -> reversed), but others live on as Sequence ABC methods rather than > as builtins or itertools. Sure, that's the part of "historical reasons" that seems accurate. There are a few methods that would be functions if they were added today (e.g., because the iterator protocol didn't exist), and a few functions that might be methods (e.g., because some of the standard types weren't class-like). Many of them would be the same even if they were added today; it's only the handful of exceptions that people even think to ask about. If you read the FAQ as saying that Python should be Java/Ruby-style pure OO (or that it should be Haskell-style pure functional), but isn't because of historical reasons, then it's unfair and incorrect. But I just read it as saying the exact balance of methods vs. functions might turn out slightly different. From denis.spir at gmail.com Sun Dec 1 09:15:41 2013 From: denis.spir at gmail.com (spir) Date: Sun, 01 Dec 2013 09:15:41 +0100 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: <529AF02D.5080007@gmail.com> On 12/01/2013 12:42 AM, Andrew Barnert wrote: > On Nov 28, 2013, at 22:49, Greg Ewing wrote: > >> spir wrote: >>> (But then, possibly, some would wonder why this new ord(s,i) is not a string method ;-) >> >> For the same reason that the existing ord() function isn't >> a string method, whatever that is! > > http://docs.python.org/2/faq/design.html#why-does-python-use-methods-for-some-functionality-e-g-list-index-but-functions-for-other-e-g-len-list > > The short version is: historical reasons, and no reason to change it compelling enough to be worth the costs in backward compat, bikeshedding, etc. I'm very ok with extending the builtin func ord to get an index param: ord(i). Solves the issue without trouble, and backward compatibile. What do you think? Denis From denis.spir at gmail.com Sun Dec 1 09:21:59 2013 From: denis.spir at gmail.com (spir) Date: Sun, 01 Dec 2013 09:21:59 +0100 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> Message-ID: <529AF1A7.4090803@gmail.com> On 12/01/2013 12:48 AM, Guido van Rossum wrote: > That FAQ entry does not do the real motivation justice. While OO "purists" > may argue about it, to me it's obvious that in quite a few cases the > function spelling is more readable than the method spelling. I found it > more readable 23 years ago, and I still find it more readable today. So > it's not historical reasons to me, even if not everybody agrees (obviously > :-). I was told ord() is a func because it works only on single-code (historically single-byte) strings (yes, ord throws an error otherwise, I just checked!). From a personal, nearly esthetic, pov, I'd rather make ord a method [s.ord(i)], and join not [join(strings, sep)]. But I do _not_ want to argue on this ;-). Denis From denis.spir at gmail.com Sun Dec 1 09:37:24 2013 From: denis.spir at gmail.com (spir) Date: Sun, 01 Dec 2013 09:37:24 +0100 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> References: <52960290.6090809@gmail.com> <52972C49.1040909@gmail.com> <20131129004527.GZ2085@ando> <52981EB6.50003@canterbury.ac.nz> <52983764.9010204@gmail.com> <5298390B.30506@canterbury.ac.nz> <6751972E-5222-4B02-AD87-3FBC404C1317@yahoo.com> Message-ID: <529AF544.9020808@gmail.com> On 12/01/2013 05:33 AM, Andrew Barnert wrote: > But I may have been assuming too much. A lot of other "multi-paradigm" languages went through a phase like that and have a historical legacy, like C++'s ridiculous string class and JavaScript's excess of Array methods. But Python does seen to have ended up with a lot fewer quirky choices than those languages. Still, I think the various index/find/count/etc. methods didn't all need to be methods. Something must be good to be criticisable... Python is clean, so its quircks so-to-say jump at one's eyes. When facing C++, or similar mess, I don't know where to start the critics ;-) Denis From denis.spir at gmail.com Sun Dec 1 13:45:52 2013 From: denis.spir at gmail.com (spir) Date: Sun, 01 Dec 2013 13:45:52 +0100 Subject: [Python-ideas] start, test, init Message-ID: <529B2F80.1070605@gmail.com> Hello, This is a proposal and opinion I have thought at sending for a long time already, but did not because it is not exactly proper to Python (there are a few more of this category ;-). What decided me is the thread about the idiom "if __name__ == '__main__':", which had much success! Below a short summary or my views, followed by a longer series of comments. Summary: There should be 3 top-level functions predefined by the language: * 'start' : start func for a module run as an app (or 'main') * 'test' : main func of a module's test suite * 'init' : module init code, rather when imported In my view, all 3 correspond to clearly definite & distinct functionalities. Each of them provides clarity and all together permit getting rid of "lost code" roaming around at the top-level of modules; which I dislike, even for scripts; however, the proposal does not force anyone to follow such a style. Every module using such functions is then a set of definitions: assignments, def's, classes, plus imports and such. Possibly 1, 2, 3 of them are language-defined main functions. Very clean... The execution logics would be: * if imported module: ~ run init if any ~ else, run nothing * if executed module: ~ run test, if any (see below) ~ else, run start, if any ~ else, run nothing (error?) What do you think? === why start? === It looks nice to have hello world reduced to: print("Hello, world!") However, this is a very superficial nicety. I prefere the (very big!) complication of: def start(): print("Hello, world!") This code is self-understanding even for a novice programmer and nicely introduces some structural feature of the language. [0] This function 'start' would have an optional argument for command-line args; it may indeed also return an exit code. About the name 'start', well, it's a question of meaning; and to avoid confusion I'd rather reserved 'main' for a package's main module. Also, every of these 3 funcs is a 'main' one, depending on the actual case. But I would *not* fight on this point, call it 'main' if you like. (In assembly, it's commonly start as well: both ends of the expressivity scale meet here ;-). When compiling with start, in standard the language may also automagically strip out development or control instructions, like assertions. (There also may be a differenciated debug print command... but that's another point. Also different error messages for end-users.) === why test? === If you practice testing (by code), you know why. Nearly all of my modules end up ending with: # === t e s t ======================= def test_abc (): ... def test_def (): ... def test_ghi (): ... ... def test (): test_abc() test_def() test_ghi() ... if __name__ == '__main__': test() Then, I comment out the last 2 lines when all runs fine; or else, if it's the main module of an app, I replace it with a call to start. This is a second form or running a module as a stand-alone prog (from the command-line). Actually, I guess, anyone practicing a minimum of testing may so-to-say press the 'test' button constantly, far more often than we lauch an app in usage or trial mode (if ever it's an app). We switch to normal usage execution once only per development phase, when all is fine and we prepare a user release. This is why, in standard, when both exist, 'test' has precedence over 'start'. There may be a builtin config var (eg __main__) to set (eg __main__ = start), or any other to switch to start (but preferably from code, see also [1]). This function 'test' _may_ have an optional argument for command-line args; it may also return an exit code, here meaning test success / failure (why not number of failures?). Args may be handy to drive testing differently: exact funcs to run, form of output, depth of testing..., in a purely user-defined way (no language-defined meaning, else we'd never stop arguing on the topic; a typical bikeshed issue). [1] === why init? === There is much less use, in my personal practice, for such an init function typically run when a module is imported. (I know it from the D language.) But I guess whenever we need it, having it is very nice. I simulate it (1) to init program elements from external data (2) to import, scan, process, big data files like unicode tables, avoiding huge code-data files and/or security issues (3) for a usage similar to compile-time computations in static langs that provide that. Anyway, it is a clearly defined functionality, just like start & test. === flexibility === An init func may be run by test or start, conditionnally or not. A test func may be run by init or start; maybe self-testing just once on first launch of a just-installed app or package. === alternative === Apart from clarity and practicality (see below), such flexibility is also why I prefere such predefined function names to a proposal by Steven D'Aprano, on the mentionned thread, of builtin conditionals like: if __main__: ... if is_main(): ... if is_main: ... We could however trivially extend this proposal with eg is_tested & is_imported builtin conditionals. Why not chose the direct simplicity of builtin func names, though? This is a common Python usage (it would also introduce novices to this idea of special func names, then they are ready for "magic methods"). Another issue is practicle: writing or not such a function, or chosing between test and start, is something we control from code itself. To launch tests for instance, there is thus no need to run a module with a special command-line option, or to modify it in our favorite programming editor's settings, or to change environment variables or python config files... all pretty annoying things to do (and I never remember which is to be done, where, how exactly...). Instead, predefined conditionnals depend on such settings or data external to the code. [2] === why simple names? === Again, I would not fight for simple names; it may well in fact be preferable to have weird names with underscores. In my view, the need in python for names like __all__ or __str__ is due to the fact that there is no difference in the language between defining and redefining a symbol. If Python used eg ':=' to redefine symbols, then: foo = ... # error, symbol 'foo' already exists foo := ... # ok, intentional redefinition of builtin 'foo' Waiting for this, it may indeed be preferable to have weird names. === trial mode === There is one mode not covered by this: what I call trial, as opposed to in-code tests. This corresponds more or less to a beta-version: a simulation of normal usage by developpers or (power-)users, to further find bugs or more generally control the software as is. I don't know of cases where this would require a special dedicated main func: this is rather opposite to the idea of checking the program really as is. But there may be code parts interpreted differently or conditionnally compiled, differently from the case of an actual user release: assertions, debug prints, stats, benchmarking, profiling... all kinds of "meta-programming" issues. === all together === ''' d o c ''' import foo __all__ = ... code... code... code... def test_abc (): ... def test_def (): ... def test_ghi (): ... ... def test (): test_abc() test_def() test_ghi() ... def init (): ... # __main__ = start def start (args): init() # process args ... What do you think? (bis) Denis [0] Python may lack such higher-level code structure, more generally; this is an open question. [1] This feature is for command-line args fans, but I prefere to control testing from code itself. For information, I have 2 std test config params, which I define in code; whether to: * write successful checks also, for comparison * continue on check failures, or else stop (They affect a custom check func, similar to an assert, used for testing only.) Turning them to off transforms a module's diagnosis test suite into (part of) a regression test suite. [2] More generally, I want to control all what concerns code and its development from code itself. Command-line args and such (better command-line args only) are good only for very special needs like producing doc, a parse-tree, compile-only, profiling, etc. I definitely dislike languages that force people to constantly change the command line to compile/run or worse change env vars or edit config files (and I don't even evoque makefiles and such...). From ncoghlan at gmail.com Sun Dec 1 14:07:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Dec 2013 23:07:58 +1000 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: The fundamental problem with this idea (outside of backwards compatibility issues) is that requiring people to write functions completely misses the core reason that scripting languages are easier to learn than those with a declarative compile time syntax: it's far easier to use a function than it is to define one. More on that topic: http://www.curiousefficiency.org/posts/2011/08/scripting-languages-and-suitable.html Aside from a couple of hints to the tokenizer and compiler in the file header, Python has no declarative mode - this is a feature, not a bug. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Dec 1 14:49:58 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 1 Dec 2013 05:49:58 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: <272F2B9E-13B7-4C50-9BBC-22C6F72A4672@yahoo.com> On Dec 1, 2013, at 4:45, spir wrote: > Every module using such functions is then a set of definitions: assignments, def's, classes, plus imports and such. Possibly 1, 2, 3 of them are language-defined main functions. Very clean... > > The execution logics would be: > * if imported module: > ~ run init if any > ~ else, run nothing > * if executed module: > ~ run test, if any (see below) > ~ else, run start, if any > ~ else, run nothing (error?) > > What do you think? I assume you realize that function and class definitions, assignments, etc. are code, and there is no separate execution phase for defining code vs. running code in Python. So, are you proposing that top-level code only be allowed to run some subset of the language, or that it just be encouraged to do so as a convention? Meanwhile, your proposal to make hello world a two-liner instead of a one-liner may not seem that big a deal, but it means that every novice has to learn how to define functions before they can write their first program. And it means that everyone who wants to use Python as an admin/scripting language in place of perl or awk or whatever has to write that extra line of code, making Python less usable. And so on. And meanwhile, in realistic large programs, you're already going to define and call the function anyway, so you're only saving the call. I don't think saving 1 line in 100-line scripts is worth adding 1 line to 1-line scripts. From denis.spir at gmail.com Sun Dec 1 15:13:14 2013 From: denis.spir at gmail.com (spir) Date: Sun, 01 Dec 2013 15:13:14 +0100 Subject: [Python-ideas] start, test, init In-Reply-To: <272F2B9E-13B7-4C50-9BBC-22C6F72A4672@yahoo.com> References: <529B2F80.1070605@gmail.com> <272F2B9E-13B7-4C50-9BBC-22C6F72A4672@yahoo.com> Message-ID: <529B43FA.10207@gmail.com> On 12/01/2013 02:49 PM, Andrew Barnert wrote: > On Dec 1, 2013, at 4:45, spir wrote: > >> Every module using such functions is then a set of definitions: assignments, def's, classes, plus imports and such. Possibly 1, 2, 3 of them are language-defined main functions. Very clean... >> >> The execution logics would be: >> * if imported module: >> ~ run init if any >> ~ else, run nothing >> * if executed module: >> ~ run test, if any (see below) >> ~ else, run start, if any >> ~ else, run nothing (error?) >> >> What do you think? > > I assume you realize that function and class definitions, assignments, etc. are code, and there is no separate execution phase for defining code vs. running code in Python. > > So, are you proposing that top-level code only be allowed to run some subset of the language, or that it just be encouraged to do so as a convention? > > Meanwhile, your proposal to make hello world a two-liner instead of a one-liner may not seem that big a deal, but it means that every novice has to learn how to define functions before they can write their first program. And it means that everyone who wants to use Python as an admin/scripting language in place of perl or awk or whatever has to write that extra line of code, making Python less usable. And so on. > > And meanwhile, in realistic large programs, you're already going to define and call the function anyway, so you're only saving the call. I don't think saving 1 line in 100-line scripts is worth adding 1 line to 1-line scripts. Well, from the initial post: "however, the proposal does not force anyone to follow such a style" Nothing prevents you to write hello-world the usual way. Nothing prevents using Python as a calculator, neither, nore as a programmer-friendly Bash; however, for the latter case, i'd still use 'test' and 'start'. Instead, it is a structural addition to the language; a backward compatible one if ever special names, like '__test__', are used rather than my preferred ones, like 'test'. The last point of your post (saving a func call) has nothing to do with the proposal or its spirit. It is about code clarity and high-level structuration, replacing "if __name=='__main__'", allowing code flow logics to be all cleanly placed in funcs, such things... (did you reply too fast, maybe?) denis From guido at python.org Sun Dec 1 17:52:31 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 1 Dec 2013 08:52:31 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: <529B43FA.10207@gmail.com> References: <529B2F80.1070605@gmail.com> <272F2B9E-13B7-4C50-9BBC-22C6F72A4672@yahoo.com> <529B43FA.10207@gmail.com> Message-ID: spir, If you are serious about your idea I would recommend that you try to build a library that implements your convention. It's a clear that you have put a lot of thought in your proposal. It is not so clear that it will be an improvement. It is also not so clear that there won't be implementation roadblocks. However, it *is* clear that most people do not think it is worth the effort. You can be sure that the effort would be significant -- not just in coding it up, but in documentation, education, getting people to adopt it, and so on. So, since this is open source, if you really, truly believe in your idea, build it. You will definitely learn something. You may end up rejecting your own idea. Or you may find a better way. Or you may pave the road for adoption. There is only one way to find out. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Dec 1 21:32:52 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 01 Dec 2013 14:32:52 -0600 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: On 12/01/2013 06:45 AM, spir wrote: > # __main__ = start > def start (args): > init() > # process args > ... > > > What do you think? (bis) I think you are describing good programming practices for larger modules. And I think most people who have been programming in python for any length of time develop some form of what you are describing. The only difference is that they wouldn't need the conditional logic at the bottom of the module. But that also serves as bit of self documentation as to what the module or script does, is more flexible, and only adds a line or two if the rest is put into functions or classes. As for the testing case... I'd like for python to have a -t option that only sets a global name __test__ to True. Then that covers most of your use cases without adding a whole lot. Sometimes less is more. if __test__: test() elif __name__ == '__main__': main() That's just one possible combination of using __test__ and __name__. Cheers, Ron From ncoghlan at gmail.com Sun Dec 1 22:13:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 07:13:56 +1000 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: On 2 Dec 2013 06:34, "Ron Adam" wrote: > > > > On 12/01/2013 06:45 AM, spir wrote: >> >> # __main__ = start >> def start (args): >> init() >> # process args >> ... >> >> >> What do you think? (bis) > > > I think you are describing good programming practices for larger modules. And I think most people who have been programming in python for any length of time develop some form of what you are describing. > > The only difference is that they wouldn't need the conditional logic at the bottom of the module. But that also serves as bit of self documentation as to what the module or script does, is more flexible, and only adds a line or two if the rest is put into functions or classes. > > > As for the testing case... I'd like for python to have a -t option that only sets a global name __test__ to True. Then that covers most of your use cases without adding a whole lot. Sometimes less is more. > > if __test__: > test() > elif __name__ == '__main__': > main() > > That's just one possible combination of using __test__ and __name__. Oh, interesting. Such an option could also alter -O and -OO to keep assert statements. I'd love to see that idea elaborated further, as there are a variety of questions around how it might work in practice (Compile time constant like __debug__? Always False builtin shadowed in the main module? If it works like __debug__ and/or affects -O and -OO, does it need a new naming convention in the pycache directory?) Cheers, Nick. > > Cheers, > Ron > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Dec 1 21:55:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 01 Dec 2013 12:55:58 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <20131129001248.GY2085@ando> References: <52960290.6090809@gmail.com> <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> Message-ID: <529BA25E.6000509@stoneleaf.us> On 11/28/2013 04:12 PM, Steven D'Aprano wrote: > > And some benchmarks: > > py> from timeit import Timer > py> setup = "from __main__ import match" > py> t1 = Timer("match('abcdef', 'cde', 2, -1)", setup) > py> t2 = Timer("s[2:-1] == 'cde'", "s = 'abcdef'") > py> min(t1.repeat(repeat=5)) > 1.2987589836120605 > py> min(t2.repeat(repeat=5)) > 0.25656223297119141 > > Slicing is about three times faster. Wouldn't that be 5 times faster? I tried to get startswith to be quicker, and compared a `s.startswith('cde', 2)` (so no function call, no plethora of other objects being created and destroyed), and slicing was still twice as fast. -- ~Ethan~ From ethan at stoneleaf.us Sun Dec 1 22:12:10 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 01 Dec 2013 13:12:10 -0800 Subject: [Python-ideas] Replacing the if __name__ == "__main__" idiom (was Re: making a module callable) In-Reply-To: References: <20131125071932.GA65531@cskk.homeip.net> <20131125141220.GE2085@ando> <20131125144244.1cb160f0@anarchist> <20131125171603.1812e9fd@anarchist> <5294CD94.6050008@gmail.com> Message-ID: <529BA62A.3030204@stoneleaf.us> On 11/30/2013 06:03 PM, Gregory P. Smith wrote: > On Wed, Nov 27, 2013 at 9:58 AM, Gregory Salvan wrote: > >> my cent: >> I think it's most often a bad practice (whereas it's convenient) to mix "executable" and "importable" code. >> Providing an easier way to do it, may encourage a bad practice. >> >> I would prefer a solution which encourage separation of "executable" and "importable" code. > > The only way to do that is to force people to put their executable code in a __main__.py file and prevent class and > function definitions within that... Gross. > > People need the test for main or not (imported or not) because they write unit tests for _all_ of their code, including > their main program. Just because it can be imported does not mean it is a library that anyone outside of their tests > _should_ import. Sure, there are some python stdlib modules and others that work as an importable library or via python > -m modulename to run a program but I wouldn't call that the common case. I would not want that to stop working either. +1 to gps' statements. -- ~Ethan~ From ethan at stoneleaf.us Sun Dec 1 22:18:35 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 01 Dec 2013 13:18:35 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: <529BA7AB.6010507@stoneleaf.us> If somebody wants this type of behavior it is easy enough to put in the `if __name__ == "__main__"` section. I do not see that adding it to the language buys us anything. -- ~Ethan~ From tjreedy at udel.edu Sun Dec 1 23:16:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Dec 2013 17:16:05 -0500 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: On 12/1/2013 7:45 AM, spir wrote: > === why test? === > > If you practice testing (by code), you know why. Nearly all of my > modules end up ending with: > > # === t e s t ======================= > > def test_abc (): > ... > def test_def (): > ... > def test_ghi (): > ... > ... > def test (): > test_abc() > test_def() > test_ghi() > ... > if __name__ == '__main__': > test() The style above is obsolete. The older test.regrtest module (private, intended for CPython test suite only) required a test_main function in which one collected together the test classes, etc, but one might forget something. The newer unittest module does the test discovery for us, so no master test function is needed. Just put unittest.main() at the bottom instead of test(). Much easier. -- Terry Jan Reedy From bruce at leapyear.org Sun Dec 1 22:55:55 2013 From: bruce at leapyear.org (Bruce Leban) Date: Sun, 1 Dec 2013 13:55:55 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: On Sun, Dec 1, 2013 at 1:13 PM, Nick Coghlan wrote: > > As for the testing case... I'd like for python to have a -t option that > only sets a global name __test__ to True. Then that covers most of your > use cases without adding a whole lot. Sometimes less is more. > > > > if __test__: > I like this idea but the specific proposal means __test__ needs to be defined in all modules. A simple alternative: if __name__ == '__test__': --- Bruce SALE! My brother's puzzle company Puzzazz is having a BlackFriday/CyberMonday sale http://j.mp/182R4kG - FREE puzzle books and BOGO -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Dec 2 01:02:00 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Dec 2013 11:02:00 +1100 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <529BA25E.6000509@stoneleaf.us> References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> Message-ID: <20131202000200.GR2085@ando> On Sun, Dec 01, 2013 at 12:55:58PM -0800, Ethan Furman wrote: > >py> from timeit import Timer > >py> setup = "from __main__ import match" > >py> t1 = Timer("match('abcdef', 'cde', 2, -1)", setup) > >py> t2 = Timer("s[2:-1] == 'cde'", "s = 'abcdef'") > >py> min(t1.repeat(repeat=5)) > >1.2987589836120605 > >py> min(t2.repeat(repeat=5)) > >0.25656223297119141 > > > >Slicing is about three times faster. > > Wouldn't that be 5 times faster? I'm reminded of a joke. What's 9/2? The mathematician says it's 4.5. The scientist says it's 4.5 ? 0.1. The engineer says it's between 4 and 5, but we'll call it 6 to be on the safe side. Apparently I'm an engineer at heart :-) > I tried to get startswith to be quicker, and compared a > `s.startswith('cde', 2)` (so no function call, no plethora of other objects > being created and destroyed), and slicing was still twice as fast. Did you try it with huge strings? Say, a million Unicode characters? -- Steven From guido at python.org Mon Dec 2 01:21:40 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 1 Dec 2013 16:21:40 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <20131202000200.GR2085@ando> References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> Message-ID: On Sun, Dec 1, 2013 at 4:02 PM, Steven D'Aprano wrote: > > I'm reminded of a joke. What's 9/2? > > The mathematician says it's 4.5. The scientist says it's 4.5 ? 0.1. The > engineer says it's between 4 and 5, but we'll call it 6 to be on the > safe side. > I think you missed one. The programmer says it's 4. :-) > Apparently I'm an engineer at heart :-) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Mon Dec 2 01:55:28 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 01 Dec 2013 18:55:28 -0600 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: On 12/01/2013 03:13 PM, Nick Coghlan wrote: > > As for the testing case... I'd like for python to have a -t option that > only sets a global name __test__ to True. Then that covers most of your > use cases without adding a whole lot. Sometimes less is more. > > > > if __test__: > > test() > > elif __name__ == '__main__': > > main() > > > > That's just one possible combination of using __test__ and __name__. > > Oh, interesting. Such an option could also alter -O and -OO to keep assert > statements. You could use the __test__ flag as the key to remove asserts rather than -O, -OO, and __debug__. Is that what you are thinking? If so, I think it would be equivalent to this. assert (invariant_condition) if __test__ else 1, fail_message > I'd love to see that idea elaborated further, as there are a variety of > questions around how it might work in practice (Compile time constant like > __debug__? Isn't __debug__ set at Python's compile time, not module compile time? I think that makes __debug__ good for testing C extensions and python's own C code, but it's too static to be very useful in python modules. Always False builtin shadowed in the main module? That would limit the __test__ check to the __main__ module. I'd like to be able to have that check available anywhere. For example if I wanted to insert some print functions and use the __test__ flag to turn them on when I need to see them... if __test__: print(this) print(that) The idea could go one step further and allow a value to be passed with the -t flag. Then you could select tests based on that value. > If it works > like __debug__ and/or affects -O and -OO, does it need a new naming > convention in the pycache directory?) I don't think it should effect -O or -OO in any significant way. And -O or -OO shouldn't effect __test__ either. I think the easiest way to implement it is to just to have __test__ set in the modules attributes at load/import time. If it's always to be available, then why not? I consider testing an imortant part of any program, so having __test__ in a module doesn't seem out of place to me. If anything... maybe if it's there... it's more likely to be used. ;-) I also like how easy it is to access. For many small scripts, including an argument parser is more work than I want to do, but including a few small tests to call if "__test__ == True" is perfect for those cases. if __test__: import doctest doctest.testmod() elif __name__ == '__main__': main() And I think it makes using __name__ here seem more natIt may also work as a name in the sys module. sys.__test__, but I prefer the simpler spelling of just __test__ because it looks nicer and allows for setting __test__ at the top of a module.ural. Ok, maybe that one is just me. It just seems, to me, some of the oddness comes from always seeing it spelled *exactly the same everywhere*. So the "if __name__ == '__main__':" line just begs for being reduced to something more minimal. Of course that does not mean it should be changes just that we will consider doing that to anything that stands out as being duplicated a lot. It's just what good programmers do. ;-) Cheers, Ron From python at mrabarnett.plus.com Mon Dec 2 02:07:40 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 02 Dec 2013 01:07:40 +0000 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> Message-ID: <529BDD5C.4040409@mrabarnett.plus.com> On 02/12/2013 00:21, Guido van Rossum wrote: > On Sun, Dec 1, 2013 at 4:02 PM, Steven D'Aprano > wrote: > > > I'm reminded of a joke. What's 9/2? > > The mathematician says it's 4.5. The scientist says it's 4.5 ? 0.1. The > engineer says it's between 4 and 5, but we'll call it 6 to be on the > safe side. > > > I think you missed one. The programmer says it's 4. :-) > No. The programmer used to say it was 4, but now says it's 4.5, give or take a little rounding error. :-) > Apparently I'm an engineer at heart :-) > From dreamingforward at gmail.com Mon Dec 2 02:19:36 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Sun, 1 Dec 2013 17:19:36 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: > There should be 3 top-level functions predefined by the language: > * 'start' : start func for a module run as an app (or 'main') > * 'test' : main func of a module's test suite > * 'init' : module init code, rather when imported 1) I'd make "main" a keyword, used to denote the program start point, and let the interpreter bail-out if it encounters more than one "main" section. Yes that keyword would only be used for that one, simple purpose, but it *is* a special purpose. This would also solve the circular import issues, because now you have a well-defined *root* in which to construct a graph and check for cycles. 2) I think testing should also be more "built-into" the interpreter environment. I like the idea of a python "-t" flag or something which makes it runs doctests. In which case, I'd also what a test() built-in, so that I can use it *within* the interpreter environment also, making sure invariants are preserved when modifying and re-using code. 3) with a "main" keyword, this wouldn't be absolutely necessary. You'll know where your top-level scope is. -- MarkJ Tacoma, Washington From breamoreboy at yahoo.co.uk Mon Dec 2 02:45:30 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 02 Dec 2013 01:45:30 +0000 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <529BDD5C.4040409@mrabarnett.plus.com> References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> <529BDD5C.4040409@mrabarnett.plus.com> Message-ID: On 02/12/2013 01:07, MRAB wrote: > On 02/12/2013 00:21, Guido van Rossum wrote: >> On Sun, Dec 1, 2013 at 4:02 PM, Steven D'Aprano >> > > wrote: >> >> >> I'm reminded of a joke. What's 9/2? >> >> The mathematician says it's 4.5. The scientist says it's 4.5 ? >> 0.1. The >> engineer says it's between 4 and 5, but we'll call it 6 to be on the >> safe side. >> >> >> I think you missed one. The programmer says it's 4. :-) >> > No. The programmer used to say it was 4, but now says it's 4.5, give or > take a little rounding error. :-) > The newbie says it should be 4.5 but gets 4.499999999999999999999 so raises issue on bug tracker stating that Python can't do arithmetic properly. In [5]: 9/2 Out[5]: 4.5 Blast, lousy example :) -- Python is the second best programming language in the world. But the best has yet to be invented. Christian Tismer Mark Lawrence From ron3200 at gmail.com Mon Dec 2 02:53:32 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 01 Dec 2013 19:53:32 -0600 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: On 12/01/2013 06:55 PM, Ron Adam wrote: > > And I think it makes using __name__ here seem more natIt may also work as a > name in the sys module. sys.__test__, but I prefer the simpler spelling of > just __test__ because it looks nicer and allows for setting __test__ at the > top of a module.ural. Hmmm... still have that annoying bug of having stuff I cut pasted back in at random places in Thunderbird. That should have read... "And I think it makes using __name__ here seem more natural." The part that was pasted in was a paragraph I deleted considering using sys.__test__, but I decided it wasn't really needed. Cheers, Ron From ethan at stoneleaf.us Mon Dec 2 06:28:07 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 01 Dec 2013 21:28:07 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: <529C1A67.3010509@stoneleaf.us> On 12/01/2013 04:55 PM, Ron Adam wrote: > > Isn't __debug__ set at Python's compile time, not module compile time? No. __debug__ is true when Python [1] is started normally, and False when started with -O or -OO. -- ~Ethan~ [1] Cpython to be specific; and there may be other flags that affect __debug__. From ncoghlan at gmail.com Mon Dec 2 08:37:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 17:37:37 +1000 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> <529BDD5C.4040409@mrabarnett.plus.com> Message-ID: On 2 December 2013 11:45, Mark Lawrence wrote: > The newbie says it should be 4.5 but gets 4.499999999999999999999 so raises > issue on bug tracker stating that Python can't do arithmetic properly. > > In [5]: 9/2 > Out[5]: 4.5 > > Blast, lousy example :) Getting a modern Python to fall into that trap at all is actually quite difficult - the way str.__repr__ works was changed a while ago to favour the nearest terminating approximation that is represented using the same IEEE754 bit pattern :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Mon Dec 2 09:42:13 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 2 Dec 2013 00:42:13 -0800 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: On Dec 1, 2013, at 17:19, Mark Janssen wrote: > 1) I'd make "main" a keyword, used to denote the program start point, > and let the interpreter bail-out if it encounters more than one "main" > section. Yes that keyword would only be used for that one, simple > purpose, but it *is* a special purpose. This would also solve the > circular import issues, because now you have a well-defined *root* in > which to construct a graph and check for cycles. How does that solve circular import issues? There's already a well-defined root, __main__, today. And if there's a cycle, it's a cycle whether it involves the root or not. And if you're talking about moving imports into a function instead of at the top level, people already do that today, and it wouldn't work any better. From abarnert at yahoo.com Mon Dec 2 09:45:06 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 2 Dec 2013 00:45:06 -0800 Subject: [Python-ideas] string codes & substring equality In-Reply-To: References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> <529BDD5C.4040409@mrabarnett.plus.com> Message-ID: On Dec 1, 2013, at 23:37, Nick Coghlan wrote: > On 2 December 2013 11:45, Mark Lawrence wrote: >> The newbie says it should be 4.5 but gets 4.499999999999999999999 so raises >> issue on bug tracker stating that Python can't do arithmetic properly. >> >> In [5]: 9/2 >> Out[5]: 4.5 >> >> Blast, lousy example :) > > Getting a modern Python to fall into that trap at all is actually > quite difficult - the way str.__repr__ works was changed a while ago > to favour the nearest terminating approximation that is represented > using the same IEEE754 bit pattern :) I think the trap you're looking for is "9/2 == 4.5". Except that one doesn't fail either. From denis.spir at gmail.com Mon Dec 2 12:36:21 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 12:36:21 +0100 Subject: [Python-ideas] start, test, init In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: <529C70B5.2050308@gmail.com> On 12/01/2013 01:45 PM, spir wrote: > [much stuff] About flexibility, I spoke of intra-module flexibility, but forgot the inter-module side: a module's 'test' or 'start' (I think in particular at regression test suites, with a std 'test' name for unit tests). denis From denis.spir at gmail.com Mon Dec 2 12:42:12 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 12:42:12 +0100 Subject: [Python-ideas] start, test, init -- in lib In-Reply-To: <529B2F80.1070605@gmail.com> References: <529B2F80.1070605@gmail.com> Message-ID: <529C7214.2040708@gmail.com> Hello, Below is a minimal simulation of the proposal's core principal. Using my best knowledge of Python to have things light for the client, the necessity for it to make use of globals() and sys (see below, end of mod.py) nevertheless removes much of the proposal's nicety. Anyway. It establishes the control logic as expressed in the proposal: * if imported module: ~ run 'init' if any ~ else, run nothing * if executed module: ~ run 'test', if any (see below) ~ else, run 'start', if any ~ else, run nothing (error?) If both test & start exist, start is run if the magic var __run_fn__ == start. The implementation also forwards command-line args to either test or start. Below code: * I called the control logic module 'top_fn', with a func of same name. * To test it, a module 'mod'. * To test the case of import, a module 'main'. Try it by: * running mod as is (with some command-line args): should run test * running mod after uncommenting __run_fun__: should run start * running main, importing mod: should run init Non-nice aspects: * The obligation to include in client module: import sys from topfn import topfn topfn(globals(), sys.argv) This should be compiler magic, certainly (eg make C main be the right func). * No further possible compiler magic about differential compilation or such (see original proposal). To be clear, the most important feature of the proposal is to establish standard magic names for high-level software structure, which are correctly taken into account by the language (read: compiler, probably). If this feature exists, it will be used, then recommanded. Thank you, Denis ============================================================================== # topfn.py from types import FunctionType def topfn(env, argv): ''' Control which top function is executed in importing module. ''' # relevant vars: name = env["__name__"] # know if module was run as app init = env.get("init", None) # init func test = env.get("test", None) # test func start = env.get("start", None) # start func for app run run_fn = env.get("__run_fn__", None) # choice between test & start # control logic: if name == "__main__": if isinstance(start, FunctionType) and run_fn == start: start(argv) elif isinstance(test, FunctionType): test(argv) elif isinstance(init, FunctionType): init() ============================================================================== # mod.py print("in mod") def init (): print("Executing init.") def test1 (): print("Executing test1.") def test2 (): print("Executing test1.") def test3 (): print("Executing test1.") def test (argv): print("Executing test with argv: %s." % argv) test1() test2() test3() def start (argv): print("Executing start with argv: %s." % argv) #~ __run_fn__ = start import sys from topfn import topfn topfn(globals(), sys.argv) ============================================================================== # main.py print("in main") import mod From denis.spir at gmail.com Mon Dec 2 12:55:42 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 12:55:42 +0100 Subject: [Python-ideas] string codes & substring equality In-Reply-To: <529BDD5C.4040409@mrabarnett.plus.com> References: <529632E5.80907@mrabarnett.plus.com> <52966558.4030002@gmail.com> <529671F0.5040200@stoneleaf.us> <529685A0.3010305@stoneleaf.us> <20131128105308.GW2085@ando> <5297ACAA.2070803@stoneleaf.us> <20131129001248.GY2085@ando> <529BA25E.6000509@stoneleaf.us> <20131202000200.GR2085@ando> <529BDD5C.4040409@mrabarnett.plus.com> Message-ID: <529C753E.5090801@gmail.com> On 12/02/2013 02:07 AM, MRAB wrote: >> I think you missed one. The programmer says it's 4. :-) >> > No. The programmer used to say it was 4, but now says it's 4.5, give or > take a little rounding error. :-) [off topic] Yop, I just discover that: which division is executed (integral or real) is now determined by operator. Great reparation of a long-time Python bug, in my view: python 3.3.1: spir at ospir:~$ python3 Python 3.3.1 (default, Sep 25 2013, 19:29:01) [GCC 4.7.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> 9/2 4.5 >>> 9//2 4 >>> 9.0/2.0 4.5 >>> 9.0//2.0 4.0 >>> python 2.7.4: spir at ospir:~$ python2 Python 2.7.4 (default, Sep 26 2013, 03:20:26) [GCC 4.7.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> 9/2 4 >>> 9//2 4 >>> 9.0/2.0 4.5 >>> 9.0//2.0 4.0 [There remains a slight inconsistency, does't it? real div always returns a float independently of args' types, while integral division return a float if float args; is it in purpose? anyway, we'll survive it.] Denis From ncoghlan at gmail.com Mon Dec 2 13:28:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 22:28:25 +1000 Subject: [Python-ideas] start, test, init In-Reply-To: <529C70B5.2050308@gmail.com> References: <529B2F80.1070605@gmail.com> <529C70B5.2050308@gmail.com> Message-ID: On 2 Dec 2013 21:37, "spir" wrote: > > On 12/01/2013 01:45 PM, spir wrote: >> >> [much stuff] > > > About flexibility, I spoke of intra-module flexibility, but forgot the inter-module side: a module's 'test' or 'start' (I think in particular at regression test suites, with a std 'test' name for unit tests). That's already covered by unittest test discovery. Higher level structure beyond the individual module is more the province of distutils-sig than it is python-ideas or python-dev. Cheers, Nick. > > denis > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis.spir at gmail.com Mon Dec 2 13:35:31 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 13:35:31 +0100 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: <529C7E93.6030602@gmail.com> On 12/01/2013 09:32 PM, Ron Adam wrote: > > > On 12/01/2013 06:45 AM, spir wrote: >> # __main__ = start >> def start (args): >> init() >> # process args >> ... >> >> >> What do you think? (bis) > > I think you are describing good programming practices for larger modules. And I > think most people who have been programming in python for any length of time > develop some form of what you are describing. > > The only difference is that they wouldn't need the conditional logic at the > bottom of the module. But that also serves as bit of self documentation as to > what the module or script does, is more flexible, and only adds a line or two if > the rest is put into functions or classes. > > > As for the testing case... I'd like for python to have a -t option that only > sets a global name __test__ to True. Then that covers most of your use cases > without adding a whole lot. Sometimes less is more. > > if __test__: > test() > elif __name__ == '__main__': > main() > > That's just one possible combination of using __test__ and __name__. That is all right, in my view; except maybe the point on flexibility (the solution of std func names is highly flexible). However, this does not address 2 related issues: * Why has the idiom "(el)if __name__ == '__main__':" has to be that obscure? and involve unnecessary knwoledge of Python arcanes (for a newcomer) * Having a std (albeit not required) high-level software structure is a net gain for the community of Python programmers (instead of everyone doing their own way... when they do). Denis From denis.spir at gmail.com Mon Dec 2 13:37:20 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 13:37:20 +0100 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: <529C7F00.5050100@gmail.com> On 12/01/2013 10:13 PM, Nick Coghlan wrote: > On 2 Dec 2013 06:34, "Ron Adam" wrote: >> >> >> >> On 12/01/2013 06:45 AM, spir wrote: >>> >>> # __main__ = start >>> def start (args): >>> init() >>> # process args >>> ... >>> >>> >>> What do you think? (bis) >> >> >> I think you are describing good programming practices for larger modules. > And I think most people who have been programming in python for any length > of time develop some form of what you are describing. >> >> The only difference is that they wouldn't need the conditional logic at > the bottom of the module. But that also serves as bit of self > documentation as to what the module or script does, is more flexible, and > only adds a line or two if the rest is put into functions or classes. >> >> >> As for the testing case... I'd like for python to have a -t option that > only sets a global name __test__ to True. Then that covers most of your > use cases without adding a whole lot. Sometimes less is more. >> >> if __test__: >> test() >> elif __name__ == '__main__': >> main() >> >> That's just one possible combination of using __test__ and __name__. > > Oh, interesting. Such an option could also alter -O and -OO to keep assert > statements. > > I'd love to see that idea elaborated further, as there are a variety of > questions around how it might work in practice (Compile time constant like > __debug__? Always False builtin shadowed in the main module? If it works > like __debug__ and/or affects -O and -OO, does it need a new naming > convention in the pycache directory?) This is very similar to my proposal, except that you'd control that from the code. No need to change command-line args. Denis From daniel at daniel-watkins.co.uk Mon Dec 2 13:39:48 2013 From: daniel at daniel-watkins.co.uk (Daniel Watkins) Date: Mon, 2 Dec 2013 12:39:48 +0000 Subject: [Python-ideas] start, test, init In-Reply-To: <529C7E93.6030602@gmail.com> References: <529B2F80.1070605@gmail.com> <529C7E93.6030602@gmail.com> Message-ID: <20131202123948.GI22882@daniel-watkins.co.uk> On Mon, Dec 02, 2013 at 01:35:31PM +0100, spir wrote: > * Why has the idiom "(el)if __name__ == '__main__':" has to be that > obscure? and involve unnecessary knwoledge of Python arcanes (for a > newcomer) It seems to me that having to know the string "if __name__ == '__main__'" is about the same level of arcana as knowing the name of a magic function that does the same. > * Having a std (albeit not required) high-level software structure > is a net gain for the community of Python programmers (instead of > everyone doing their own way... when they do). "if __name__ == '__main__'" is, by necessity, standard across all Python programs. And if I want to run the tests for a project, I would expect './setup.py test' to run them for me. If a project has more than one entry point, how would I know which one to run to get the tests to work? Or would they all have to implement the test function identically? Dan From denis.spir at gmail.com Mon Dec 2 13:48:47 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 13:48:47 +0100 Subject: [Python-ideas] start, test, init In-Reply-To: References: <529B2F80.1070605@gmail.com> Message-ID: <529C81AF.4090105@gmail.com> On 12/02/2013 02:19 AM, Mark Janssen wrote: >> There should be 3 top-level functions predefined by the language: >> * 'start' : start func for a module run as an app (or 'main') >> * 'test' : main func of a module's test suite >> * 'init' : module init code, rather when imported > > 1) I'd make "main" a keyword, used to denote the program start point, > and let the interpreter bail-out if it encounters more than one "main" > section. Yes that keyword would only be used for that one, simple > purpose, but it *is* a special purpose. This would also solve the > circular import issues, because now you have a well-defined *root* in > which to construct a graph and check for cycles. > 2) I think testing should also be more "built-into" the interpreter > environment. I like the idea of a python "-t" flag or something which > makes it runs doctests. In which case, I'd also what a test() > built-in, so that I can use it *within* the interpreter environment > also, making sure invariants are preserved when modifying and re-using > code. > 3) with a "main" keyword, this wouldn't be absolutely necessary. > You'll know where your top-level scope is. I guess that is close to my point of view, if I undertand yours correctly: there could be an minimal amount of high-level software structure, at least about different running modes (exec, test, import), built into the language. Denis From ron3200 at gmail.com Mon Dec 2 16:37:37 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 02 Dec 2013 09:37:37 -0600 Subject: [Python-ideas] start, test, init In-Reply-To: <529C7E93.6030602@gmail.com> References: <529B2F80.1070605@gmail.com> <529C7E93.6030602@gmail.com> Message-ID: On 12/02/2013 06:35 AM, spir wrote: >> >> As for the testing case... I'd like for python to have a -t option that only >> sets a global name __test__ to True. Then that covers most of your use >> cases >> without adding a whole lot. Sometimes less is more. >> >> if __test__: >> test() >> elif __name__ == '__main__': >> main() >> >> That's just one possible combination of using __test__ and __name__. > > That is all right, in my view; except maybe the point on flexibility (the > solution of std func names is highly flexible). However, this does not > address 2 related issues: > * Why has the idiom "(el)if __name__ == '__main__':" has to be that > obscure? and involve unnecessary knwoledge of Python arcanes (for a newcomer) I don't see it as obscure or arcane to compare a string to variable which bound to a string. It's a very common and basic operation. The only extra knowledge you need is to know that the __name__ attribute is set to "__main__" in the main module. And that enables us to check if "this" is the main module in a direct way. >>> dir() ['__builtins__', '__doc__', '__loader__', '__name__', '__package__'] >>> __name__ '__main__' >>> __name__ == '__main__' True The difference between python and compiled languages that must use functions to get things started, is that python is already executing the code, so it a matter of avoiding some parts at different times. > * Having a std (albeit not required) high-level software structure is a net > gain for the community of Python programmers (instead of everyone doing > their own way... when they do). Most beginner books on python explain objects and name binding fairly early. Knowing a module is also an object and has a __name__ attribute is something a python programmer should learn very early. It helps them learn and understand how everything works in a more direct way. So rather than adding ... "You been doing this for a while, but many people do this other way ... ". So it would need to be explained anyway. Cheers, Ron From apieum at gmail.com Mon Dec 2 16:55:22 2013 From: apieum at gmail.com (Gregory Salvan) Date: Mon, 2 Dec 2013 16:55:22 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) Message-ID: Hi, I've made a try of a function that check object/class members for duck typing. For now I've basically called it isducktype(X, A) it returns true if: - X has all attributes of A - X methods have same number of arguments than corresponding A methods or if A.__ducktypecheck__(X) returns True Behaviour looks like isinstance and issubclass (support tuples...). To test it: The patch with example is here: https://gist.github.com/apieum/7751279 Or you can use my branch here: https://bitbucket.org/gsalvan/cpython/commits/branch/default Any opinion about a feature like this ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 2 16:59:25 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 2 Dec 2013 07:59:25 -0800 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: Sounds like a cute thing to post on PyPI and see how popular it becomes. You have until the 3.5 release cycle starts to gather votes and improvements. There are lots of edge cases I could see going wrong, so I'm kind of skeptical of having this in the stdlib without strong caveats about its usefulness. Maybe it could go in the inspect module, which is traditionally full of heuristics. On Mon, Dec 2, 2013 at 7:55 AM, Gregory Salvan wrote: > Hi, > I've made a try of a function that check object/class members for duck > typing. > > For now I've basically called it isducktype(X, A) > it returns true if: > - X has all attributes of A > - X methods have same number of arguments than corresponding A methods > > or if A.__ducktypecheck__(X) returns True > > Behaviour looks like isinstance and issubclass (support tuples...). > > To test it: > The patch with example is here: > https://gist.github.com/apieum/7751279 > > Or you can use my branch here: > https://bitbucket.org/gsalvan/cpython/commits/branch/default > > > Any opinion about a feature like this ? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Dec 2 17:11:48 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 2 Dec 2013 17:11:48 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: <20131202161148.GA19607@phdru.name> Hi! On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan wrote: > For now I've basically called it isducktype(X, A) > it returns true if: > - X has all attributes of A Isn't the requirement too strong? When I need a file-like object I seldom need more than .read() and .write() methods, and even of those I seldom need both at once -- I usually need one or the other. Testing for strict conformance is not duck typing IMO -- it's interface. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From rosuav at gmail.com Mon Dec 2 17:22:51 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 Dec 2013 03:22:51 +1100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <20131202161148.GA19607@phdru.name> References: <20131202161148.GA19607@phdru.name> Message-ID: On Tue, Dec 3, 2013 at 3:11 AM, Oleg Broytman wrote: > On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan wrote: >> For now I've basically called it isducktype(X, A) >> it returns true if: >> - X has all attributes of A > > Isn't the requirement too strong? When I need a file-like object I > seldom need more than .read() and .write() methods, and even of those I > seldom need both at once -- I usually need one or the other. > Testing for strict conformance is not duck typing IMO -- it's > interface. But you could have a "ReadableFile" that has a .read() method and whatever else you need, thus using that class as a sort of interface. With a little careful scripting and introspection, you might even be able to craft that by source code analysis, which would be extremely cool (eg you run this script over your source code and it notes that elements of this list are indexed with these five keywords, so it constructs a class with those five so you can check what goes into the list). Whether it's actually _useful_ or not remains to be seen, but it would certainly be cool. ChrisA From apieum at gmail.com Mon Dec 2 17:33:10 2013 From: apieum at gmail.com (Gregory Salvan) Date: Mon, 2 Dec 2013 17:33:10 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <20131202161148.GA19607@phdru.name> References: <20131202161148.GA19607@phdru.name> Message-ID: Ok for pypi. (I've never tried to make python C extension, I feel it's going to be epic :) ) @Oleg you can override default behaviour with __ducktypecheck__ it's not as strict as an interface as var names can be differents and can have defaults. then instead of declaring an interface with "read" and "write", you just create a base class with these methods. What solution do you suggest ? I've made a function that returns public members only, maybe this method can be exposed in the api too. 2013/12/2 Oleg Broytman > Hi! > > On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan > wrote: > > For now I've basically called it isducktype(X, A) > > it returns true if: > > - X has all attributes of A > > Isn't the requirement too strong? When I need a file-like object I > seldom need more than .read() and .write() methods, and even of those I > seldom need both at once -- I usually need one or the other. > Testing for strict conformance is not duck typing IMO -- it's > interface. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 2 17:38:26 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 3 Dec 2013 03:38:26 +1100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <20131202161148.GA19607@phdru.name> Message-ID: On Tue, Dec 3, 2013 at 3:33 AM, Gregory Salvan wrote: > Ok for pypi. (I've never tried to make python C extension, I feel it's going > to be epic :) Is there any reason this can't be implemented in pure Python? It'd be a lot easier to figure things out that way. Maybe a C implementation could come later, but for a first cut, this ought to be possible in Python. ChrisA From apieum at gmail.com Mon Dec 2 17:50:57 2013 From: apieum at gmail.com (Gregory Salvan) Date: Mon, 2 Dec 2013 17:50:57 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <20131202161148.GA19607@phdru.name> Message-ID: Yes it can be done in python. It's actually in C, essentially because I wanted to learn python internal, and as it's quite similar to isinstance and issubclass. 2013/12/2 Chris Angelico > On Tue, Dec 3, 2013 at 3:33 AM, Gregory Salvan wrote: > > Ok for pypi. (I've never tried to make python C extension, I feel it's > going > > to be epic :) > > Is there any reason this can't be implemented in pure Python? It'd be > a lot easier to figure things out that way. Maybe a C implementation > could come later, but for a first cut, this ought to be possible in > Python. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Dec 2 18:36:09 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 02 Dec 2013 19:36:09 +0200 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: 02.12.13 17:55, Gregory Salvan ???????(??): > Hi, > I've made a try of a function that check object/class members for duck > typing. > > For now I've basically called it isducktype(X, A) > it returns true if: > - X has all attributes of A > - X methods have same number of arguments than corresponding A methods > > or if A.__ducktypecheck__(X) returns True > > Behaviour looks like isinstance and issubclass (support tuples...). class FooReader: def read(self, n=None): ... class BarReader: def read(self, amount=-1): ... class BazReader: def read(self, size=-1, chars=-1, firstline=False): ... Instances of all three classes can be used where needed an object with the read() method which are called with one positional argument or without arguments. Is it possible to use isducktype() for such case? From phd at phdru.name Mon Dec 2 18:54:35 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 2 Dec 2013 18:54:35 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <20131202161148.GA19607@phdru.name> Message-ID: <20131202175435.GA22609@phdru.name> On Mon, Dec 02, 2013 at 05:33:10PM +0100, Gregory Salvan wrote: > @Oleg > you can override default behaviour with __ducktypecheck__ > it's not as strict as an interface as var names can be differents and can > have defaults. > then instead of declaring an interface with "read" and "write", you just > create a base class with these methods. > > What solution do you suggest ? > I've made a function that returns public members only, maybe this method > can be exposed in the api too. > > 2013/12/2 Oleg Broytman > > > Hi! > > > > On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan > > wrote: > > > For now I've basically called it isducktype(X, A) > > > it returns true if: > > > - X has all attributes of A > > > > Isn't the requirement too strong? When I need a file-like object I > > seldom need more than .read() and .write() methods, and even of those I > > seldom need both at once -- I usually need one or the other. > > Testing for strict conformance is not duck typing IMO -- it's > > interface. I don't suggest any solution as I don't have a problem to solve with interfaces. You are going to implement another interface checker in addition to existing ones? https://pypi.python.org/pypi?%3Aaction=search&term=interface&submit=search No objections from me. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From elazarg at gmail.com Mon Dec 2 19:04:19 2013 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Mon, 2 Dec 2013 20:04:19 +0200 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <20131202161148.GA19607@phdru.name> References: <20131202161148.GA19607@phdru.name> Message-ID: On Dec 2, 2013 6:12 PM, "Oleg Broytman" wrote: > > Hi! > > On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan wrote: > > For now I've basically called it isducktype(X, A) > > it returns true if: > > - X has all attributes of A > > Isn't the requirement too strong? When I need a file-like object I > seldom need more than .read() and .write() methods, and even of those I > seldom need both at once -- I usually need one or the other. > Testing for strict conformance is not duck typing IMO -- it's > interface. Interfaces are for nominal languages. Technically it should be called x.is_structurally_subtype() Elazar _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Dec 2 19:10:52 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 2 Dec 2013 19:10:52 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <20131202161148.GA19607@phdru.name> Message-ID: <20131202181052.GA23868@phdru.name> On Mon, Dec 02, 2013 at 08:04:19PM +0200, ?????????? wrote: > On Dec 2, 2013 6:12 PM, "Oleg Broytman" wrote: > > On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan > wrote: > > > For now I've basically called it isducktype(X, A) > > > it returns true if: > > > - X has all attributes of A > > > > Isn't the requirement too strong? When I need a file-like object I > > seldom need more than .read() and .write() methods, and even of those I > > seldom need both at once -- I usually need one or the other. > > Testing for strict conformance is not duck typing IMO -- it's > > interface. > Interfaces are for nominal languages. Technically it should be called > x.is_structurally_subtype() I'm sure the terminology in Python is quite stable. There are interfaces (there are many implementations, there is one rejected PEP and one deferred PEP) and there is duck typing. Formalizing duck typing is IMO constructing an interface. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From abarnert at yahoo.com Mon Dec 2 19:42:22 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 2 Dec 2013 10:42:22 -0800 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> Shouldn't this be tied to ABCs rather than redesigning and reimplementing most of ABC to add a little bit on top? Sent from a random iPhone On Dec 2, 2013, at 7:55, Gregory Salvan wrote: > Hi, > I've made a try of a function that check object/class members for duck typing. > > For now I've basically called it isducktype(X, A) > it returns true if: > - X has all attributes of A > - X methods have same number of arguments than corresponding A methods > > or if A.__ducktypecheck__(X) returns True > > Behaviour looks like isinstance and issubclass (support tuples...). > > To test it: > The patch with example is here: > https://gist.github.com/apieum/7751279 > > Or you can use my branch here: > https://bitbucket.org/gsalvan/cpython/commits/branch/default > > > Any opinion about a feature like this ? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis.spir at gmail.com Mon Dec 2 19:51:13 2013 From: denis.spir at gmail.com (spir) Date: Mon, 02 Dec 2013 19:51:13 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: <529CD6A1.8080202@gmail.com> On 12/02/2013 04:55 PM, Gregory Salvan wrote: > I've made a try of a function that check object/class members for duck > typing. > > For now I've basically called it isducktype(X, A) > it returns true if: > - X has all attributes of A > - X methods have same number of arguments than corresponding A methods Just a question: I wonder about actual usefulness. For a strict safe check you'd also want to control method param types and names, not only number. Indeed, param types are out of question here, but what about param names? Since you implement that in C, is it at all possible to get method param names? (Well, in fact, maybe even Python's self-examination power allows that, does it?) Denis From zuo at chopin.edu.pl Tue Dec 3 00:10:47 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 03 Dec 2013 00:10:47 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> Message-ID: <821805489b4fb932405a0f26749201dd@chopin.edu.pl> 02.12.2013 19:42, Andrew Barnert wrote: > Shouldn't this be tied to ABCs rather than redesigning and > reimplementing most of ABC to add a little bit on top? +1. *j From apieum at gmail.com Tue Dec 3 02:21:15 2013 From: apieum at gmail.com (Gregory Salvan) Date: Tue, 3 Dec 2013 02:21:15 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: @Serhiy Not really (only with FooReader, BarReader) as it checks if functions co_argcount and co_kwonlyargcount are equals. In this case it requires BazReader.__ducktypecheck__ to be implemented. I had a doubt when doing this, I agree with you isducktype(BazReader, FooReader) can return True (whereas isducktype(FooReader, BazReader) should stay false) This can easily be changed. @Oleg: The fact there is a lot of interface implementations means certainly it's something needed. Each implementation I've seen add burden on objects whereas the only thing personnally I need, is to be sure I'll be able to call some methods off injected objects. @spir: Names can easily be checked, but I can't see why doing this, it add constraints to developpers without adding safety. @Andrew +1 Maybe inspect is more appropriate as it provides ArgSpec and getmembers ? My code is crappy, I've just do it like that to train me in C then tought it might be interesting to get feedback. 2013/12/2 Serhiy Storchaka > 02.12.13 17:55, Gregory Salvan ???????(??): > > Hi, >> I've made a try of a function that check object/class members for duck >> typing. >> >> For now I've basically called it isducktype(X, A) >> it returns true if: >> - X has all attributes of A >> - X methods have same number of arguments than corresponding A methods >> >> or if A.__ducktypecheck__(X) returns True >> >> Behaviour looks like isinstance and issubclass (support tuples...). >> > > class FooReader: > def read(self, n=None): ... > > class BarReader: > def read(self, amount=-1): ... > > class BazReader: > def read(self, size=-1, chars=-1, firstline=False): ... > > Instances of all three classes can be used where needed an object with the > read() method which are called with one positional argument or without > arguments. Is it possible to use isducktype() for such case? > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Dec 3 08:56:37 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 2 Dec 2013 23:56:37 -0800 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: Message-ID: <341911A7-D99C-4BE5-A158-6901864AE09E@yahoo.com> On Dec 2, 2013, at 17:21, Gregory Salvan wrote: > @Andrew +1 > Maybe inspect is more appropriate as it provides ArgSpec and getmembers ? Sure, using inspect for the implementation _also_ makes sense. But my point is more about the interface to your new thing. Your thing checks most of the same stuff that ABCs check, but maybe not all of it, and also adds checking for compatible argspecs. (I believe that's the only new feature it adds; am I right?) Meanwhile, it uses a new isducktype function instead of working with normal issubclass/isinstance, and a completely different way of registering explicit compliance, and so on. Instead, why not just extend ABCMeta with a new subclass that just adds the check for compatible argspecs (maybe using inspect, as you say) and takes advantage of all the other stuff ABC already does? Besides being less code, which isn't that big of a deal, it's less of a radical conceptual change, which is a bigger deal, and it means that all kinds of code that relies on isinstance today can work with no changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 3 14:05:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 23:05:52 +1000 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> Message-ID: On 3 December 2013 04:42, Andrew Barnert wrote: > Shouldn't this be tied to ABCs rather than redesigning and reimplementing > most of ABC to add a little bit on top? This seems like an apropos place for this link: http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ That's the existing "formalised ducktyping" hook, that already allows things like the following: >>> class MyClass: ... def __len__(self): ... return 0 ... >>> from collections.abc import Sized >>> isinstance(MyClass(), Sized) True >>> issubclass(MyClass, Sized) True The advantage ABCs have over ducktyping alone is that subclassing and explicit registration allow ambiguities like the one between the Sequence and Mapping interfaces to be resolved (you can't always just ducktype those, as they have the same methods - they differ only in how the __*item__ methods handle slice objects). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From apieum at gmail.com Wed Dec 4 06:22:06 2013 From: apieum at gmail.com (Gregory Salvan) Date: Wed, 4 Dec 2013 06:22:06 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> Message-ID: Sorry Andrew, I'm not sure to understand all your point. I don't believe it reimplements things of ABC, or I don't see what. I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol. Before reading your comments on ABC, I've released the python version of "isducktype": https://github.com/apieum/ducktype I've modified some behaviours within your comments and made it more stable. It would be interesting to have the version wich use ABCMeta, so I'm going to implement it. I've thought to name it SignedMeta, or ProtocolMeta, any opinion ? 2013/12/3 Nick Coghlan > On 3 December 2013 04:42, Andrew Barnert wrote: > > Shouldn't this be tied to ABCs rather than redesigning and reimplementing > > most of ABC to add a little bit on top? > > This seems like an apropos place for this link: > http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ > > That's the existing "formalised ducktyping" hook, that already allows > things like the following: > > >>> class MyClass: > ... def __len__(self): > ... return 0 > ... > >>> from collections.abc import Sized > >>> isinstance(MyClass(), Sized) > True > >>> issubclass(MyClass, Sized) > True > > The advantage ABCs have over ducktyping alone is that subclassing and > explicit registration allow ambiguities like the one between the > Sequence and Mapping interfaces to be resolved (you can't always just > ducktype those, as they have the same methods - they differ only in > how the __*item__ methods handle slice objects). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Dec 4 09:19:30 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Dec 2013 00:19:30 -0800 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> Message-ID: <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> On Dec 3, 2013, at 21:22, Gregory Salvan wrote: > Sorry Andrew, I'm not sure to understand all your point. > I don't believe it reimplements things of ABC, or I don't see what. Almost everything you did reimplements features of ABCs. The builtin isinstance and issubclass functions can check whether you implement the methods of a type, and you can override that by explicitly declaring that you implement the type. That's the core feature set of your proposal, and of ABCs; you've just implemented the same thing with a different API, plus and minus a few minor details. If you haven't read PEP 3119, you really should. (It's also worth reading rejected predecessors like 245/246 and successors like 3124, and of course the pre-existing third party stuff like PyProtocols and the Interface types in Twisted and Zope.) > I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol. I don't understand what you mean here. ABCs can check methods. They do so at runtime. You're free to define or not define any protocol you want. What you have to add is checking the argspecs of the methods rather than just their names. That's a great idea, but it's a much better idea as an addition to what's already there than as a complete parallel API to very similar functionality. And if this were added to the stdlib, I'm pretty sure it would be a modification/extension of the abc module, not a competing and similar but not identical module. > Before reading your comments on ABC, I've released the python version of "isducktype": https://github.com/apieum/ducktype > I've modified some behaviours within your comments and made it more stable. > > It would be interesting to have the version wich use ABCMeta, so I'm going to implement it. > I've thought to name it SignedMeta, or ProtocolMeta, any opinion ? First, it sounds like you're focusing on the wrong part of the name--ABCMeta is a metaclass for implementing ABCs; leaving out the ABC part sounds like you're designing a metaclass for something completely unrelated. Less seriously, ProtocolMeta sounds like it's going to add more PyProtocols-style functionality (in particular, adapters from one protocol to another), but there's only so many synonyms available and they're all taken, so I don't think that's a major problem. As for SignedMeta, I don't even know what that's meant to imply. Is it to do with method signatures? If so, I think "signature" is a better word; you don't really think about anything being "signed" in a method signature; that's dragging in associations from a different meaning of the word. But I don't have any better suggestions. Protocol, interface, prototype, type, abstract type, abstract base... As I said, all of the names have already been used up, repeatedly, and none of them have any particular connotations that would explain how your module, metaclass, function, etc. differ from abc. In fact, I'd probably just come up with a stupid package name like abc2 and stick in modules named abc, collections.abc, and numbers and expose names identical to the stdlib ones, so I could play around with it by just using abc2.isinstance, abc2.abc.ABCMeta, abc2.collections.abc.Mapping, etc. Or I could just toss some "from abc2 import"s around. But for any serious use beyond playing around, that would be quite confusing. > 2013/12/3 Nick Coghlan >> On 3 December 2013 04:42, Andrew Barnert wrote: >> > Shouldn't this be tied to ABCs rather than redesigning and reimplementing >> > most of ABC to add a little bit on top? >> >> This seems like an apropos place for this link: >> http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ >> >> That's the existing "formalised ducktyping" hook, that already allows >> things like the following: >> >> >>> class MyClass: >> ... def __len__(self): >> ... return 0 >> ... >> >>> from collections.abc import Sized >> >>> isinstance(MyClass(), Sized) >> True >> >>> issubclass(MyClass, Sized) >> True >> >> The advantage ABCs have over ducktyping alone is that subclassing and >> explicit registration allow ambiguities like the one between the >> Sequence and Mapping interfaces to be resolved (you can't always just >> ducktype those, as they have the same methods - they differ only in >> how the __*item__ methods handle slice objects). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Dec 4 13:20:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Dec 2013 22:20:56 +1000 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> Message-ID: On 4 December 2013 18:19, Andrew Barnert wrote: > I don't understand what you mean here. ABCs can check methods. They do so at > runtime. You're free to define or not define any protocol you want. > > What you have to add is checking the argspecs of the methods rather than > just their names. That's a great idea, but it's a much better idea as an > addition to what's already there than as a complete parallel API to very > similar functionality. And if this were added to the stdlib, I'm pretty sure > it would be a modification/extension of the abc module, not a competing and > similar but not identical module. It should also be based on the rich function inspect.signature API (introduced in http://www.python.org/dev/peps/pep-0362/), not on the older low level interfaces (the function signature API has been backported to Python 2.x as the funcsigs module on PyPI). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Wed Dec 4 17:09:43 2013 From: brett at python.org (Brett Cannon) Date: Wed, 4 Dec 2013 11:09:43 -0500 Subject: [Python-ideas] 2 more admins for the list Message-ID: FYI both Ethan Furman and Tal Einat have joined Titus and myself in administering this list. And while I'm thinking about it, just a reminder about how to let us know about CoC violations. Please email/cc python-ideas-owner@ when you think there has been an infraction. There is enough volume on the list you can't assume we read every email. We also try to stay objective on this topic so chances are we won't bring up any CoC issues in order to stay neutral. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apieum at gmail.com Thu Dec 5 07:48:35 2013 From: apieum at gmail.com (Gregory Salvan) Date: Thu, 5 Dec 2013 07:48:35 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> Message-ID: Thank you for taking time to answer. I trust you, but I've sometimes the feeling we're talking about different things. I didn't know PEP 3124 and PEP 0362, they are great ressources. Signature is particularly usefull, it's a pity it's not documented, I'll suggest something further. I'm ok with what you said (I talked effectively about signatures). The matter I see, is that attributes and methods can be set dynamically (that's why I talked about runtime), either I don't manage these cases, or I have to extend ABCMeta and override __instancecheck__ and __subclasscheck__. The second solution would make duplicate code unless I extract ABCMeta cache and registry operations in a dedicated object. This would be a lot of change and despite _abc_* attributes are protected, I'm afraid some library use them. I've made an example of failure to illustrate dynamic setting matter: https://gist.github.com/apieum/7800992 Maybe it's more effective for attributes which are often defined in __init__ uniquely. Whereas all of this is really interesting we are quite far from the initial target which was simply to avoid these cases: >>> class MyClass: ... def __len__(self, enum): ... return 0 ... >>> from collections.abc import Sized >>> isinstance(MyClass(), Sized) True >>> issubclass(MyClass, Sized) True >>> len(MyClass()) Traceback (most recent call last): File "", line 1, in TypeError: __len__() missing 1 required positional argument: 'enum' I'll dig if I can find a better proposal considering all you suggestion. 2013/12/4 Andrew Barnert > On Dec 3, 2013, at 21:22, Gregory Salvan wrote: > > Sorry Andrew, I'm not sure to understand all your point. > I don't believe it reimplements things of ABC, or I don't see what. > > > Almost everything you did reimplements features of ABCs. The builtin > isinstance and issubclass functions can check whether you implement the > methods of a type, and you can override that by explicitly declaring that > you implement the type. That's the core feature set of your proposal, and > of ABCs; you've just implemented the same thing with a different API, plus > and minus a few minor details. > > If you haven't read PEP 3119, you really should. (It's also worth reading > rejected predecessors like 245/246 and successors like 3124, and of course > the pre-existing third party stuff like PyProtocols and the Interface types > in Twisted and Zope.) > > I agree ABC solution is the cleanest, whereas I would be able to check > functions, stay on a runtime checking and being free to define or not a > kind of protocol. > > > I don't understand what you mean here. ABCs can check methods. They do so > at runtime. You're free to define or not define any protocol you want. > > What you have to add is checking the argspecs of the methods rather than > just their names. That's a great idea, but it's a much better idea as an > addition to what's already there than as a complete parallel API to very > similar functionality. And if this were added to the stdlib, I'm pretty > sure it would be a modification/extension of the abc module, not a > competing and similar but not identical module. > > Before reading your comments on ABC, I've released the python version of > "isducktype": https://github.com/apieum/ducktype > I've modified some behaviours within your comments and made it more stable. > > It would be interesting to have the version wich use ABCMeta, so I'm going > to implement it. > I've thought to name it SignedMeta, or ProtocolMeta, any opinion ? > > > First, it sounds like you're focusing on the wrong part of the > name--ABCMeta is a metaclass for implementing ABCs; leaving out the ABC > part sounds like you're designing a metaclass for something completely > unrelated. > > Less seriously, ProtocolMeta sounds like it's going to add more > PyProtocols-style functionality (in particular, adapters from one protocol > to another), but there's only so many synonyms available and they're all > taken, so I don't think that's a major problem. As for SignedMeta, I don't > even know what that's meant to imply. Is it to do with method signatures? > If so, I think "signature" is a better word; you don't really think about > anything being "signed" in a method signature; that's dragging in > associations from a different meaning of the word. > > But I don't have any better suggestions. Protocol, interface, prototype, > type, abstract type, abstract base... As I said, all of the names have > already been used up, repeatedly, and none of them have any particular > connotations that would explain how your module, metaclass, function, etc. > differ from abc. > > In fact, I'd probably just come up with a stupid package name like abc2 > and stick in modules named abc, collections.abc, and numbers and expose > names identical to the stdlib ones, so I could play around with it by just > using abc2.isinstance, abc2.abc.ABCMeta, abc2.collections.abc.Mapping, etc. > Or I could just toss some "from abc2 import"s around. But for any serious > use beyond playing around, that would be quite confusing. > > 2013/12/3 Nick Coghlan > >> On 3 December 2013 04:42, Andrew Barnert wrote: >> > Shouldn't this be tied to ABCs rather than redesigning and >> reimplementing >> > most of ABC to add a little bit on top? >> >> This seems like an apropos place for this link: >> http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ >> >> That's the existing "formalised ducktyping" hook, that already allows >> things like the following: >> >> >>> class MyClass: >> ... def __len__(self): >> ... return 0 >> ... >> >>> from collections.abc import Sized >> >>> isinstance(MyClass(), Sized) >> True >> >>> issubclass(MyClass, Sized) >> True >> >> The advantage ABCs have over ducktyping alone is that subclassing and >> explicit registration allow ambiguities like the one between the >> Sequence and Mapping interfaces to be resolved (you can't always just >> ducktype those, as they have the same methods - they differ only in >> how the __*item__ methods handle slice objects). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 5 10:46:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Dec 2013 19:46:54 +1000 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> Message-ID: On 5 December 2013 16:48, Gregory Salvan wrote: > Thank you for taking time to answer. > I trust you, but I've sometimes the feeling we're talking about different > things. > > I didn't know PEP 3124 and PEP 0362, they are great ressources. Signature is > particularly usefull, it's a pity it's not documented, It's covered in the inspect module docs: http://docs.python.org/3/library/inspect#introspecting-callables-with-the-signature-object > I'm ok with what you said (I talked effectively about signatures). > The matter I see, is that attributes and methods can be set dynamically > (that's why I talked about runtime), either I don't manage these cases, or I > have to extend ABCMeta and override __instancecheck__ and __subclasscheck__. > The second solution would make duplicate code unless I extract ABCMeta cache > and registry operations in a dedicated object. Use ABCMeta.__subclasshook__, that's what it is for. It's deliberate that ABCs don't check for instance attributes, and it's only the automatic caching behaviour (required for reasonable performance) that prevents the ABC machinery from recognising dynamic class updates by default. However, you *can* make it recognise updates by invalidating the caches (by doing an explicit registration on any ABC), and I'd be open to an "abc.invalidate_caches()" function as a more obvious way to force cache invalidation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From apieum at gmail.com Thu Dec 5 16:33:05 2013 From: apieum at gmail.com (Gregory Salvan) Date: Thu, 5 Dec 2013 16:33:05 +0100 Subject: [Python-ideas] check interfaces, isducktype(X, A) In-Reply-To: References: <096C000A-37EE-4D41-BB47-40B7A2EE0D56@yahoo.com> <487E1BD1-63D1-47E1-8CF8-8A12F2A49FEE@yahoo.com> Message-ID: > > It's covered in the inspect module docs: > > Oups, I've missed it was introduced in v3.3 > I'd be open to an "abc.invalidate_caches()" function as a > more obvious way to force cache invalidation. > Does this implementation make sense ? https://gist.github.com/apieum/7805431 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Dec 6 17:29:25 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 6 Dec 2013 09:29:25 -0700 Subject: [Python-ideas] Replacing the if __name__ == "__main__" idiom (was Re: making a module callable) In-Reply-To: References: <20131125071932.GA65531@cskk.homeip.net> <20131125141220.GE2085@ando> <20131125144244.1cb160f0@anarchist> <20131125171603.1812e9fd@anarchist> Message-ID: On Tue, Nov 26, 2013 at 3:01 AM, Nick Coghlan wrote: > > On 26 Nov 2013 10:00, "Mark Janssen" wrote: >> >> The only other possibility not mentioned thus far is to have a main.py >> file and force python programs to start from it. > > Looking for a __main__.py module (or submodule) is the way directory, > zipfile and package execution work, so this style is already possible today > for anyone that wants or needs it. It also doesn't work so well for non-packages. :) That said, I like how __main__.py makes a clear separation between the package and its script form. As I said in my opening email, I think that distinction is very fuzzy for modules in general and bites both beginners and advanced users. For me anything that proposed replacing the current "if __name__ == '__main__'" idiom should help make the script/module separation more distinct. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Dec 6 17:31:47 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 6 Dec 2013 09:31:47 -0700 Subject: [Python-ideas] Replacing the if __name__ == "__main__" idiom (was Re: making a module callable) In-Reply-To: References: <20131125071932.GA65531@cskk.homeip.net> <20131125141220.GE2085@ando> <20131125144244.1cb160f0@anarchist> <20131125171603.1812e9fd@anarchist> Message-ID: On Dec 6, 2013 9:29 AM, "Eric Snow" wrote: > That said, I like how __main__.py makes a clear separation between the package and its script form. As I said in my opening email, I think that distinction is very fuzzy for modules in general and bites both beginners and advanced users. For me anything that proposed replacing the current "if __name__ == '__main__'" idiom should help make the script/module separation more distinct. Regardless, it may be worth adding an additional approach to highlighting the script/module duality. We could raise a warning when a module (findable on sys.path) is run as script unless via the "-m" flag. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcepl at redhat.com Fri Dec 6 23:09:46 2013 From: mcepl at redhat.com (=?utf-8?B?TWF0xJtq?= Cepl) Date: Fri, 6 Dec 2013 23:09:46 +0100 Subject: [Python-ideas] why is not request in httplib pretty printed? Message-ID: <20131206220940.GA15554@wycliff.ceplovi.cz> Hi, looking at http://hg.python.org/cpython/file/543c76769c14/Lib/http/client.py#l847 (but this code has been same since like forever) I see that the HTTP request is NOT pretty printed: if self.debuglevel > 0: print("send:", repr(data)) whereas response in effect (because every header is printed separately) is. Wouldn't it be better to pretty print the request as well? Otherwise I get quite unreadable debugging logs like the following (notice how much response is more readable than request). Mat?j matej at wycliff: urllib2_kerberos (next *%)$ python test_trac.py send: 'GET /desktopqe-backlog/login HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: someserver.redhat.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 401 Authorization Required\r\n' header: Date: Fri, 06 Dec 2013 17:11:13 GMT header: Server: Apache header: WWW-Authenticate: Negotiate header: WWW-Authenticate: Basic realm="Kerberos Login" header: Content-Length: 483 header: Connection: close header: Content-Type: text/html; charset=iso-8859-1 2013-12-06 18:11:15,406 DEBUG inside http_error_401 2013-12-06 18:11:15,407 DEBUG retry count: 1 2013-12-06 18:11:15,407 DEBUG req.get_host() returned someserver.redhat.com 2013-12-06 18:11:15,407 DEBUG authGSSClientInit() succeeded 2013-12-06 18:11:15,486 DEBUG authGSSClientStep() succeeded 2013-12-06 18:11:15,486 DEBUG authGSSClientResponse() succeeded send: 'GET /desktopqe-backlog/login HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: someserver.redhat.com\r\nConnection: close\r\nAuthorization: Negotiate YIICdQYJKoZIhvcSA---some-junk--GZDggzt\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 302 Found\r\n' header: Date: Fri, 06 Dec 2013 17:11:14 GMT header: Server: Apache header: WWW-Authenticate: Negotiate YIGZBgkqhkiG9xIB--other-junk--NOxMdOR/5 header: Location: https://someserver.redhat.com/desktopqe-backlog header: Content-Length: 0 header: Pragma: no-cache header: Cache-Control: no-cache header: Expires: Fri, 01 Jan 1999 00:00:00 GMT header: Set-Cookie: trac_auth=148d029eb321e9a0782fd30ca3f9d17b; Path=/desktopqe-backlog; secure header: Connection: close header: Content-Type: text/plain; charset=UTF-8 send: 'GET /desktopqe-backlog HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: someserver.redhat.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Fri, 06 Dec 2013 17:11:15 GMT header: Server: Apache header: Cache-Control: must-revalidate header: Expires: Fri, 01 Jan 1999 00:00:00 GMT header: Content-Length: 9591 header: Set-Cookie: trac_form_token=a6eda80ba3f4ead82093a0af; Path=/desktopqe-backlog; secure header: Set-Cookie: trac_session=213441ab7934999739979a04; expires=Thu, 06-Mar-2014 17:11:15 GMT; Path=/desktopqe-backlog; secure header: Connection: close header: Content-Type: text/html;charset=utf-8 2013-12-06 18:11:17,190 DEBUG www-authenticate header not found 2013-12-06 18:11:17,190 CRITICAL mutual auth failed. No negotiate header matej at wycliff: urllib2_kerberos (next *%)$ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From tjreedy at udel.edu Sat Dec 7 00:07:42 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 06 Dec 2013 18:07:42 -0500 Subject: [Python-ideas] why is not request in httplib pretty printed? In-Reply-To: <20131206220940.GA15554@wycliff.ceplovi.cz> References: <20131206220940.GA15554@wycliff.ceplovi.cz> Message-ID: On 12/6/2013 5:09 PM, Mat?j Cepl wrote: > http://hg.python.org/cpython/file/543c76769c14/Lib/http/client.py#l847 > (but this code has been same since like forever) I see that the > HTTP request is NOT pretty printed: > > if self.debuglevel > 0: > print("send:", repr(data)) > > whereas response in effect (because every header is printed > separately) is. Some thing have just not been done yet. > Wouldn't it be better to pretty print the > request as well? > Otherwise I get quite unreadable debugging logs like the > following (notice how much response is more readable than > request). > matej at wycliff: urllib2_kerberos (next *%)$ python test_trac.py > send: 'GET /desktopqe-backlog/login HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: someserver.redhat.com\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n' I agree that the request above should be as nicely printed as the response below. If there is not already a tracker issue for this, I think you should open one. > reply: 'HTTP/1.1 401 Authorization Required\r\n' > header: Date: Fri, 06 Dec 2013 17:11:13 GMT > header: Server: Apache -- Terry Jan Reedy From stephen at xemacs.org Sat Dec 7 02:05:07 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 07 Dec 2013 10:05:07 +0900 Subject: [Python-ideas] Attribute docstrings [was: One-line abstractmethod function?] In-Reply-To: References: <52A0E469.7090904@mrabarnett.plus.com> <20131205211228.GB27767@avalon.amherst.edu> <20131206020629.GA15017@avalon.amherst.edu> <20131206020812.GB15017@avalon.amherst.edu> <52A136F4.2090609@stoneleaf.us> <20131206104621.GN2085@ando> Message-ID: <87vbz131z0.fsf@uwakimon.sk.tsukuba.ac.jp> Reply-To set to python-ideas at python.org. Terry Reedy writes: > For data attributes, which are usually mutable, it should be attached to > the attribute *concept*, which is represented by the name, rather than > the current but usually changeable value. Values are usually already > documented either by a value representation or a docstring. This could > be done with a string subclass that is used as needed. > > For methods, the value is nearly always constant. When multiple methods > share the same function, they usually also share the same name, and > represent the same concept. Aside: Properties are which? Data, or method? It's also not clear to me that "def first (self): return self.values[0]" is unlikely to be used for completely different purposes than getting the head of a list. I conclude the considerations above are mostly red herrings. The important thing, I suppose, is that the names of attributes defined in a class are not mutable. This means that their docstrings can be kept in a completely separate dict (or other string -> string mapping), which could even be stored in a separate file. (Emacs Lisp uses this to good effect. The DOC file for XEmacs is 1.6MB; for GNU Emacs it's 2.4MB.) Of course it has its problems, but they're pretty minor. From mcepl at redhat.com Sat Dec 7 12:04:21 2013 From: mcepl at redhat.com (=?utf-8?B?TWF0xJtq?= Cepl) Date: Sat, 7 Dec 2013 12:04:21 +0100 Subject: [Python-ideas] why is not request in httplib pretty printed? In-Reply-To: References: <20131206220940.GA15554@wycliff.ceplovi.cz> Message-ID: <20131207110414.GA23789@wycliff.ceplovi.cz> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > I agree that the request above should be as nicely printed as > the response below. If there is not already a tracker issue > for this, I think you should open one. I haven't found anything, so I have filed http://bugs.python.org/issue19917 Mat?j -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iD8DBQFSovx64J/vJdlkhKwRAlXjAJ42kfsxMbP+TIzwBjqAQFqhEQP08wCeM8Aa IzhfXXl+iYjyRKKk/mWTir0= =DdfB -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From mal at egenix.com Sat Dec 7 13:21:06 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 07 Dec 2013 13:21:06 +0100 Subject: [Python-ideas] Update the required C compiler for Windows to a supported version. In-Reply-To: <3dd00aa18f174c4e93a26f6e806b8439@BLUPR03MB293.namprd03.prod.outlook.com> References: <3dd00aa18f174c4e93a26f6e806b8439@BLUPR03MB293.namprd03.prod.outlook.com> Message-ID: <52A312B2.8080200@egenix.com> Hi Steve, On 27.11.2013 19:24, Steve Dower wrote: > Stephen J. Turnbull wrote: >> Vernon D. Cole writes: >> >>> I cannot compile a Python extension module with any Microsoft compiler >>> I can obtain. >> >> Your pain is understood, but it's not simple to address it. > > FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) > > The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. I would expect that Python is not the only OSS tool that requires to use the same compiler version for extensions as the one used to compile the main application. With Microsoft opening up towards OSS software, wouldn't it be better to keep the express versions available officially for a much longer period or allow 3rd parties to host the express ISOs ? If there's anything the Python Software Foundation could do to help with this, please let me know. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 07 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From g.rodola at gmail.com Sat Dec 7 14:03:12 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 7 Dec 2013 14:03:12 +0100 Subject: [Python-ideas] Easily reference a single unittest from cmdline Message-ID: During the development process it is very common to implement a new functionality, write a test for it and then run tests. I don't want to run *all* tests though, but only the new one which I just wrote. Currently unittest module lets you do this via cmdline with: python -m unittest test.test_module.TestClass.test_method This is not very practical though as you have to figure out the name of the module (easy) and the name of the test class (something which I *never* remember). Proposal is to allow a wildcard notation like this: python -m unittest test.test_module.*test_method* python -m unittest *test_method* What I expect from unittest is to execute only the test methods matching "*test_method*". This is related to an old proposal I raised 3 years ago: https://mail.python.org/pipermail/python-ideas/2010-August/007992.html After that dicussion I ended up coming up with a decorator for skipping tests: http://code.activestate.com/recipes/578234-unittestskip_others-decorator/?in=user-4178764 ...but I never find it really practical and I ended up abandoning it. In retrospective, I think the cmdline is the right place from where such a thing should be controlled. Thoughts? --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/ From g.rodola at gmail.com Sat Dec 7 14:27:58 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 7 Dec 2013 14:27:58 +0100 Subject: [Python-ideas] Easily reference a single unittest from cmdline In-Reply-To: References: Message-ID: I just took a look at nosetests. It provides a similar thing and lets you use a regular expression (which indeed makes a lot more sense than using wildcards) It works like this: nosetests test/test_module -m test_name --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/ On Sat, Dec 7, 2013 at 2:03 PM, Giampaolo Rodola' wrote: > During the development process it is very common to implement a new > functionality, write a test for it and then run tests. > I don't want to run *all* tests though, but only the new one which I just wrote. > Currently unittest module lets you do this via cmdline with: > > python -m unittest test.test_module.TestClass.test_method > > This is not very practical though as you have to figure out the name > of the module (easy) and the name of the test class (something which I > *never* remember). > > Proposal is to allow a wildcard notation like this: > > python -m unittest test.test_module.*test_method* > python -m unittest *test_method* > > What I expect from unittest is to execute only the test methods > matching "*test_method*". > > This is related to an old proposal I raised 3 years ago: > https://mail.python.org/pipermail/python-ideas/2010-August/007992.html > After that dicussion I ended up coming up with a decorator for skipping tests: > http://code.activestate.com/recipes/578234-unittestskip_others-decorator/?in=user-4178764 > ...but I never find it really practical and I ended up abandoning it. > > In retrospective, I think the cmdline is the right place from where > such a thing should be controlled. > > Thoughts? > > --- Giampaolo > https://code.google.com/p/pyftpdlib/ > https://code.google.com/p/psutil/ > https://code.google.com/p/pysendfile/ From tjreedy at udel.edu Sat Dec 7 23:03:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 07 Dec 2013 17:03:05 -0500 Subject: [Python-ideas] Easily reference a single unittest from cmdline In-Reply-To: References: Message-ID: On 12/7/2013 8:03 AM, Giampaolo Rodola' wrote: > During the development process it is very common to implement a new > functionality, write a test for it Or vice versa ;-) > and then run tests. > I don't want to run *all* tests though, but only the new one which I just wrote. I presume for time reasons. The idlelib test modules I have written so far run under a second, so this is not an issue for me. For me, a test module would have to run several seconds, at least, to be worth switching to a console window instead of hitting F5. > Currently unittest module lets you do this via cmdline with: > > python -m unittest test.test_module.TestClass.test_method > > This is not very practical though as you have to figure out the name > of the module (easy) and the name of the test class (something which I > *never* remember). I mostly use ClassnameTest or FunctionnameTest. > Proposal is to allow a wildcard notation like this: > > python -m unittest test.test_module.*test_method* I think this should better be python -m unittest "test.test_module.*.test_method" > python -m unittest *test_method* I think you left something out here. > What I expect from unittest is to execute only the test methods > matching "*test_method*". Inspired by you second post, how about python -m unittest test.test_module -t test_method1 test_method2 ... which gives more flexibility. Of course, test_x will match multiple tests if test_x appears in multiple testcase classes. I believe there are other proposals for more flexibility in picking which tests to run. There may be a tracker issue already. -- Terry Jan Reedy From amber.yust at gmail.com Sun Dec 8 05:44:36 2013 From: amber.yust at gmail.com (Amber Yust) Date: Sun, 08 Dec 2013 04:44:36 +0000 Subject: [Python-ideas] Batching/grouping function for itertools Message-ID: <5251239173677340489@gmail297201516> After seeing yet another person asking how to do this on #python (and having needed to do it in the past myself), I'm wondering why itertools doesn't have a function to break an iterator up into N-sized chunks. Existing possible solutions include both the "clever" but somewhat unreadable... batched_iter = zip(*[iter(input_iter)]*n) ...and the long-form... def batch(input_iter, n): input_iter = iter(input_iter) while True: yield [input_iter.next() for _ in range(n)] There doesn't seem, however, to be one clear "right" way to do this. Every time I come up against this task, I go back to itertools expecting one of the grouping functions there to cover it, but they don't. It seems like it would be a natural fit for itertools, and it would simplify things like processing of file formats that use a consistent number of lines per entry, et cetera. ~Amber -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Sun Dec 8 06:14:18 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 7 Dec 2013 21:14:18 -0800 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <5251239173677340489@gmail297201516> References: <5251239173677340489@gmail297201516> Message-ID: On Sat, Dec 7, 2013 at 8:44 PM, Amber Yust wrote: > After seeing yet another person asking how to do this on #python (and having > needed to do it in the past myself), I'm wondering why itertools doesn't > have a function to break an iterator up into N-sized chunks. +1. In my experience the grouper recipe in the docs serve less as a helpful example of how to use itertools and more as a thing to copy paste. That's what modules are for. -- Devin From ncoghlan at gmail.com Sun Dec 8 08:02:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Dec 2013 17:02:05 +1000 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> Message-ID: On 8 December 2013 15:14, Devin Jeanpierre wrote: > On Sat, Dec 7, 2013 at 8:44 PM, Amber Yust wrote: >> After seeing yet another person asking how to do this on #python (and having >> needed to do it in the past myself), I'm wondering why itertools doesn't >> have a function to break an iterator up into N-sized chunks. > > +1. In my experience the grouper recipe in the docs serve less as a > helpful example of how to use itertools and more as a thing to copy > paste. That's what modules are for. The windowing problem is too ill-defined - there are enough degrees of freedom that any API flexible enough to cover them all is harder to learn than just building out your own version that works the way you want it to, and a more restrictive API that *doesn't* cover all the variants introduces a sharp discontinuity between the "blessed" variant and the alternatives. For anyone that thinks the stdlib itertools is too minimalist (I'm not one of them), than "pip install more-itertools" provides the recipes from the stdlib docs, as well as a few other precomposed operations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From taleinat at gmail.com Sun Dec 8 09:14:42 2013 From: taleinat at gmail.com (Tal Einat) Date: Sun, 8 Dec 2013 10:14:42 +0200 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> Message-ID: Hello Amber, These issues -- a batching function in itertools and including the itertools recipes in the stdlib -- have both been discussed here recently. Specifically regarding the batching function, I couldn't find the most recent discussion via a quick search. IIRC the conclusion was what Nick said: different use-cases require slightly different behaviors, which can not be elegantly expressed as a single, simple and straight-forward function. Therefore, it is better to have a basic recipe in the docs, which everyone can modify according to their needs. With regard to including the other recipes in the stdlib, I recommend reading the most recent discussion on the archives [1]. The major argument against this is that these recipes are easily implemented based on the existing tools, but having all of them in the stdlib means having to support them all in the future, including maintaining backwards compatibility. Supporting stdlib code is considerably harder than having working examples in the docs. - Tal [1] https://mail.python.org/pipermail/python-ideas/2012-July/015714.html On Sun, Dec 8, 2013 at 9:02 AM, Nick Coghlan wrote: > On 8 December 2013 15:14, Devin Jeanpierre wrote: > > On Sat, Dec 7, 2013 at 8:44 PM, Amber Yust wrote: > >> After seeing yet another person asking how to do this on #python (and > having > >> needed to do it in the past myself), I'm wondering why itertools doesn't > >> have a function to break an iterator up into N-sized chunks. > > > > +1. In my experience the grouper recipe in the docs serve less as a > > helpful example of how to use itertools and more as a thing to copy > > paste. That's what modules are for. > > The windowing problem is too ill-defined - there are enough degrees of > freedom that any API flexible enough to cover them all is harder to > learn than just building out your own version that works the way you > want it to, and a more restrictive API that *doesn't* cover all the > variants introduces a sharp discontinuity between the "blessed" > variant and the alternatives. > > For anyone that thinks the stdlib itertools is too minimalist (I'm not > one of them), than "pip install more-itertools" provides the recipes > from the stdlib docs, as well as a few other precomposed operations. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Dec 8 10:25:17 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 Dec 2013 20:25:17 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> Message-ID: <20131208092517.GU2085@ando> On Sun, Dec 08, 2013 at 05:02:05PM +1000, Nick Coghlan wrote: > The windowing problem is too ill-defined - there are enough degrees of > freedom that any API flexible enough to cover them all is harder to > learn than just building out your own version that works the way you > want it to, and a more restrictive API that *doesn't* cover all the > variants introduces a sharp discontinuity between the "blessed" > variant and the alternatives. Playing Devil's Advocate here, I wonder if that is true though. It seems to me that there are two basic windowing variants: sliding windows, and discrete windows. That is, given a sequence [a, b, c, d, e, f, g, h, i] and a window size of 3, the two obvious, common results are: # sliding window (a,b,c), (b,c,d), (c,d,e), (d,e,f), (e,f,g), (f,g,h), (g,h,i) # discrete windows (a,b,c), (d,e,f), (g,h,i) Surely anything else is exotic enough that there is no question about leaving it up to the individual programmer. In the second case, there is a question about what to do with sequences that are not a multiple of the window size. Similar to zip(), there are two things one might do: - pad with some given object; - raise an exception If you want to just ignore extra items, just catch the exception and continue. So that's a maximum of three window functions: sliding(iterable, window_size) discrete(iterable, window_size, pad=None) strict_discrete(iterable, window_size) or just two, if you combine discrete and strict_discrete: discrete(iterable, window_size [, pad]) # raise if pad is not given What other varieties are there? Surely none that are common. Once, for a lark, I tried to come up with one that was fully general -- as well as a window size, you could specify how far to advance the window each step. The sliding variety would advance by 1 each step, the discrete variety would advance by the window size. But I never found any reason to use it with any other step sizes. Surely anything else is more useful in theory than in practice. (That's three times I've said something is "surely" true, always a sign my argument is weak *grin*) Given that this windowing problem keeps coming up, there's no doubt in my mind that it is a useful, if not fundamental, iterator operation. Ruby's Enumerable module includes both: http://ruby-doc.org/core-2.0.0/Enumerable.html each_cons is what I've been calling a sliding window, and each_slice is what I've been calling discrete chunks. -- Steven From ncoghlan at gmail.com Sun Dec 8 12:05:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Dec 2013 21:05:18 +1000 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208092517.GU2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On 8 December 2013 19:25, Steven D'Aprano wrote: > On Sun, Dec 08, 2013 at 05:02:05PM +1000, Nick Coghlan wrote: > >> The windowing problem is too ill-defined - there are enough degrees of >> freedom that any API flexible enough to cover them all is harder to >> learn than just building out your own version that works the way you >> want it to, and a more restrictive API that *doesn't* cover all the >> variants introduces a sharp discontinuity between the "blessed" >> variant and the alternatives. > > Playing Devil's Advocate here, I wonder if that is true though. It seems > to me that there are two basic windowing variants: sliding windows, and > discrete windows. That is, given a sequence [a, b, c, d, e, f, g, h, i] > and a window size of 3, the two obvious, common results are: > > # sliding window > (a,b,c), (b,c,d), (c,d,e), (d,e,f), (e,f,g), (f,g,h), (g,h,i) > > # discrete windows > (a,b,c), (d,e,f), (g,h,i) > > > Surely anything else is exotic enough that there is no question about > leaving it up to the individual programmer. > > In the second case, there is a question about what to do with sequences > that are not a multiple of the window size. Similar to zip(), there are > two things one might do: > > - pad with some given object; > - raise an exception > > If you want to just ignore extra items, just catch the exception and > continue. So that's a maximum of three window functions: > > sliding(iterable, window_size) > discrete(iterable, window_size, pad=None) > strict_discrete(iterable, window_size) > > or just two, if you combine discrete and strict_discrete: > > discrete(iterable, window_size [, pad]) > # raise if pad is not given > > What other varieties are there? Surely none that are common. Once, for a > lark, I tried to come up with one that was fully general -- as well as a > window size, you could specify how far to advance the window each step. > The sliding variety would advance by 1 each step, the discrete variety > would advance by the window size. But I never found any reason to use it > with any other step sizes. Surely anything else is more useful in theory > than in practice. > > (That's three times I've said something is "surely" true, always a sign > my argument is weak *grin*) I'm biased by a signal processing background where playing games with data windows and the amount of overlap between samples is a *really* common technique :) > Given that this windowing problem keeps coming up, there's no doubt in > my mind that it is a useful, if not fundamental, iterator operation. > Ruby's Enumerable module includes both: > > http://ruby-doc.org/core-2.0.0/Enumerable.html > > each_cons is what I've been calling a sliding window, and each_slice is > what I've been calling discrete chunks. The two examples in the itertools docs are currently just pairwise (sliding window of length 2) and grouper (distinct windows of arbitrary length, always padded) The general cases would be: def sliding_window(iterable, n): """Return a sliding window over the data, introducing one new item into each window""" iterables = tee(iterable, n) # Prime the iterables for i, itr in iterables: for __ in range(i): next(itr, None) return zip(*iterables) def discrete_window(iterable, n, fillvalue=None): """Return distinct windows of the data, padding the last window if necessary""" repeated_iterable = [iter(iterable)] * n return zip_longest(*repeated_iterable, fillvalue=fillvalue) Given the padding version of discrete_window, the exception raising version is just: def discrete_window_no_padding(iterable, n): sentinel = object() for x in discrete_window(iterable, n, sentinel): if x[-1] is sentinel: raise ValueError("Ragged final partition") yield x Given the "n-1" overlapping version of sliding window, the "selective overlap" version (ignoring any leftover data at the end) is just: def sliding_window_with_configurable_overlap(iterable, n, new_items=1): if new_items == 1: return sliding_window(iterable, n) return islice(sliding_window(iterable, n), 0, None, new_items) The main argument in favour of offering sliding_window and discrete_window as itertools is that they each rely on a sophisticated trick in iterator state manipulation: - the sliding window relies on using tee() and then priming the results to start at the appropriate place in the initial window - the discrete window relies on using *multiple* reference to a single iterator and exploiting the fact that the iterator advances each time a value is retrieved That's a deeper understanding of the object model than most people will have, so they're likely to just cargo cult the recipe from the docs anyway, without really trying to understand exactly how it works. I guess I'm +0 rather than -1 at this point, but it's really Raymond that needs to be convinced as module maintainer (other points of note: such a change wouldn't be possible until Python 3.5 anyway, and it would also require restructuring itertools to be a hybrid C/Python module, since writing these in C wouldn't offer any significant benefits - the inner loops in the pure Python versions are already using existing high speed iterators). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Dec 8 12:30:56 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 08 Dec 2013 13:30:56 +0200 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208092517.GU2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: 08.12.13 11:25, Steven D'Aprano ???????(??): > In the second case, there is a question about what to do with sequences > that are not a multiple of the window size. Similar to zip(), there are > two things one might do: > > - pad with some given object; > - raise an exception 3) emit last chunk incompleted; 4) skip incomplete chunk. There is also a question about result's type. Sometimes you need an iterator of subsequences (i.e. split string on equal string chunks), sometimes an iterator of iterators is enough. I.e. at least 8 variants are needed. From steve at pearwood.info Sun Dec 8 13:16:29 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 Dec 2013 23:16:29 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: <20131208121629.GV2085@ando> On Sun, Dec 08, 2013 at 01:30:56PM +0200, Serhiy Storchaka wrote: > 08.12.13 11:25, Steven D'Aprano ???????(??): > >In the second case, there is a question about what to do with sequences > >that are not a multiple of the window size. Similar to zip(), there are > >two things one might do: > > > >- pad with some given object; > >- raise an exception > > 3) emit last chunk incompleted; Given a window size of two, and input data [a, b, c], are you suggesting a variety that returns this? (a,b), (c,) There is no need for a separate function for that. Given a version that takes a pad object, if the pad argument is not given, return a partial chunk at the end. > 4) skip incomplete chunk. The very next sentence in my post references that: "If you want to just ignore extra items, just catch the exception and continue." There is no need for an extra function covering that case. > There is also a question about result's type. Sometimes you need an > iterator of subsequences (i.e. split string on equal string chunks), > sometimes an iterator of iterators is enough. None of the other itertools functions treat strings specially. Why should this one? If you want to re-join them into strings, you can do so with a trivial wrapper: (''.join(elements) for elements in group("some string", 3, pad=' ')) ought to do the trick, assuming group returns tuples or lists of characters. Re-combining the iterated-upon elements into the input type is not the responsibility of itertools. -- Steven From barry at python.org Sun Dec 8 14:13:10 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 8 Dec 2013 08:13:10 -0500 Subject: [Python-ideas] Easily reference a single unittest from cmdline References: Message-ID: <20131208081310.22aa55a3@anarchist.wooz.org> On Dec 07, 2013, at 02:27 PM, Giampaolo Rodola' wrote: >I just took a look at nosetests. >It provides a similar thing and lets you use a regular expression >(which indeed makes a lot more sense than using wildcards) >It works like this: > >nosetests test/test_module -m test_name I use nose2 in several projects and have a nice little plugin that reproduces the most useful (for me) bits of zope testrunner for specifying test patterns. I can usually just do something like $ nose2 -P test_this_one_thing $ nose2 -P TestThisWholeClass $ nose2 -P test_module $ nose2 -P test_things.rst http://tinyurl.com/l2trn4a -Barry P.S. Yes, nose2 *rocks* :) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ron3200 at gmail.com Sun Dec 8 16:13:06 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 08 Dec 2013 09:13:06 -0600 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On 12/08/2013 05:05 AM, Nick Coghlan wrote: > The main argument in favour of offering sliding_window and > discrete_window as itertools is that they each rely on a sophisticated > trick in iterator state manipulation: How about 2 lower level building blocks that would make these things easier to make and think about. Possibly function to take the next n items of an iterator without advancing it. Along with a function to advance an iterator n ahead without taking anything. These would be simpler and easier to maintain, and have a wider range of uses. Cheers, Ron From stephen at xemacs.org Sun Dec 8 16:21:35 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 09 Dec 2013 00:21:35 +0900 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208092517.GU2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: <87ob4r2wsg.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > What other varieties are there? Surely none that are common. Once, for a > lark, I tried to come up with one that was fully general -- as well as a > window size, you could specify how far to advance the window each step. > The sliding variety would advance by 1 each step, the discrete variety > would advance by the window size. But I never found any reason to use it > with any other step sizes. Surely anything else is more useful in theory > than in practice. Deseasonalization of serially correlated data where the seasonality is lower-frequency than the series, and more generally data-mining techniques that start with relatively coarse steps and refine as they go along are two that come immediately to mind. From rosuav at gmail.com Sun Dec 8 16:22:49 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Dec 2013 02:22:49 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On Mon, Dec 9, 2013 at 2:13 AM, Ron Adam wrote: > How about 2 lower level building blocks that would make these things easier > to make and think about. > > > Possibly function to take the next n items of an iterator without advancing > it. Fundamentally impossible. Here's a function that returns an iterator: def d20(): import random while True: yield random.randrange(1,21) dice_roller = d20() How are you going to take the next n items from dice_roller without advancing it? ChrisA From stephen at xemacs.org Sun Dec 8 16:34:28 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 09 Dec 2013 00:34:28 +0900 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: <87iouz2w6z.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > How are you going to take the next n items from dice_roller without > advancing it? Memoize. From rosuav at gmail.com Sun Dec 8 16:40:30 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Dec 2013 02:40:30 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <87iouz2w6z.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> <87iouz2w6z.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Dec 9, 2013 at 2:34 AM, Stephen J. Turnbull wrote: > Chris Angelico writes: > > > How are you going to take the next n items from dice_roller without > > advancing it? > > Memoize. That's not a building-block then, that's a quite separate feature. In this particular instance there's no way to distinguish between "predict the next three but don't advance the iterator" and "advance the iterator by three and then rewind it the same distance", but imagine an iterators that blocks for input, or something. You don't want a purportedly low-level function (from which you derive the more "usable" functions) doing memoization on that. ChrisA From ron3200 at gmail.com Sun Dec 8 16:40:27 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 08 Dec 2013 09:40:27 -0600 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On 12/08/2013 09:22 AM, Chris Angelico wrote: >> >How about 2 lower level building blocks that would make these things easier >> >to make and think about. >> > >> > >> >Possibly function to take the next n items of an iterator without advancing >> >it. > Fundamentally impossible. In some cases yes, but you would know if it could work before you chose to use these. > Here's a function that returns an iterator: > > def d20(): > import random > while True: > yield random.randrange(1,21) > > dice_roller = d20() > > How are you going to take the next n items from dice_roller without > advancing it? If it is to work with generators too... The function would need to be a wrapper that keeps a buffer. And the front of the buffer would always be the next to be yielded if there is anything in it. In the case of advancing an generator, without taking the values, it would still need to call the __next__ methods on it, but not actually return the values. Yes, it won't be as simple as it sounds. ;-) cheers, Ron From rosuav at gmail.com Sun Dec 8 16:46:53 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Dec 2013 02:46:53 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On Mon, Dec 9, 2013 at 2:40 AM, Ron Adam wrote: > The function would need to be a wrapper that keeps a buffer. And the front > of the buffer would always be the next to be yielded if there is anything in > it. Which is what Stephen said in his pithy message above. Yes, it's theoretically possible, but it's not something to do in the general case. Not something to depend on for a chunker/grouper, which should be able to pass once over the underlying iterator. ChrisA From ron3200 at gmail.com Sun Dec 8 17:07:07 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 08 Dec 2013 10:07:07 -0600 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On 12/08/2013 09:46 AM, Chris Angelico wrote: > On Mon, Dec 9, 2013 at 2:40 AM, Ron Adam wrote: >> >The function would need to be a wrapper that keeps a buffer. And the front >> >of the buffer would always be the next to be yielded if there is anything in >> >it. > Which is what Stephen said in his pithy message above. Yes, it's > theoretically possible, but it's not something to do in the general > case. Not something to depend on for a chunker/grouper, which should > be able to pass once over the underlying iterator. For the windowed chunker that was being discussed and uses that are similar, they need to hold references to some yielded items some place. This just packages that need in a convenient way. It almost seems to me that coroutines, which is the counter example you suggested, should maybe be a type of their own. That way, they can be tested. (And raise an error if used in cases like this.) But that's whole other topic. Best answer for now is don't do that. Cheers, Ron From steve at pearwood.info Sun Dec 8 17:38:55 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Dec 2013 03:38:55 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: <20131208163855.GW15615@ando> On Sun, Dec 08, 2013 at 09:13:06AM -0600, Ron Adam wrote: > Possibly function to take the next n items of an iterator without advancing > it. Fundamentally impossible. The best you can do it advance the iterator but store the results for later use. > Along with a function to advance an iterator n ahead without taking > anything. Too trivial to bother with. Just advance the iterator and throw the result away. def advance(it, n): for _ in range(n): next(it) You can even do it as a one-linear, at the expense of readability: {next(it) and None for _ in range(n)}.pop() (The pop isn't really necessary, I just like the fact that it means the expression evaluates as None.) > These would be simpler and easier to maintain, and have a wider range of > uses. Maybe, maybe not, but they don't solve the problem that people keep asking to be solved. -- Steven From steve at pearwood.info Sun Dec 8 17:45:19 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Dec 2013 03:45:19 +1100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <87iouz2w6z.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> <87iouz2w6z.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20131208164519.GX2085@ando> On Mon, Dec 09, 2013 at 12:34:28AM +0900, Stephen J. Turnbull wrote: > Chris Angelico writes: > > > How are you going to take the next n items from dice_roller without > > advancing it? > > Memoize. Er, I don't think so. How does the memoizing cache get those values if the underlying iterator isn't advanced? Obviously it can't. itertools.tee uses a cache, so we can demonstrate the issue: py> it = iter("abcde") py> wrapper = itertools.tee(it, 2)[0] py> _ = list(wrapper) If the iterator hasn't advanced, then next(it) should yield 'a'. But: py> next(it) Traceback (most recent call last): File "", line 1, in StopIteration Any sort of "iterator look-ahead" has a number of fundamental problems. Despite many requests, those problems are part of the reason why Python iterators don't provide a "peek" method to look ahead. Not even to look ahead a single value, let alone an arbitrary number of values. - The cache would require unbounded memory (unless you limit the look-ahead to N values); - iterators with side-effects would cause those side-effects at the wrong time; - iterators whose calculated values are time-dependent could be calculated at a different time from when they are returned, potentially giving the wrong result. For something like tee, it is difficult to see any other way other than memoisation to get the functionality needed, so we just have to live with the limitations. But offering dedicated look-ahead with caching as fundamental iterator tools, as Ron suggests, strikes me as completely the wrong thing to do if what we actually want is to group items. It doesn't solve the problem being asked, since it's still up to the caller to make their own grouper tool out of the memoising primitive. -- Steven From storchaka at gmail.com Sun Dec 8 18:32:40 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 08 Dec 2013 19:32:40 +0200 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208121629.GV2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> <20131208121629.GV2085@ando> Message-ID: 08.12.13 14:16, Steven D'Aprano ???????(??): > On Sun, Dec 08, 2013 at 01:30:56PM +0200, Serhiy Storchaka wrote: >> 08.12.13 11:25, Steven D'Aprano ???????(??): >>> In the second case, there is a question about what to do with sequences >>> that are not a multiple of the window size. Similar to zip(), there are >>> two things one might do: >>> >>> - pad with some given object; >>> - raise an exception >> >> 3) emit last chunk incompleted; > > Given a window size of two, and input data [a, b, c], are you suggesting > a variety that returns this? > > (a,b), (c,) > > There is no need for a separate function for that. Given a version that > takes a pad object, if the pad argument is not given, return a partial > chunk at the end. You had proposed raise an exception when the pad argument is not given in previous message. In any case these are three different cases, and you can combine only two of them in one function using "absent argument" trick. >> 4) skip incomplete chunk. > > The very next sentence in my post references that: > > "If you want to just ignore extra items, just catch the exception and > continue." > > There is no need for an extra function covering that case. You can't just use this generator in expression (e.g. as an argument to list). You need special wrapper which catches en exception. This is fourth variant. And if you need just this variant, why it is not in the stdlib? >> There is also a question about result's type. Sometimes you need an >> iterator of subsequences (i.e. split string on equal string chunks), >> sometimes an iterator of iterators is enough. > > None of the other itertools functions treat strings specially. Why > should this one? Because I relatively often need this idiom and almost never need general function for iterators. I'm sure a function which splits sequences are enough in at least 90% cases when you need grouping function. > If you want to re-join them into strings, you can do so > with a trivial wrapper: > > (''.join(elements) for elements in group("some string", 3, pad=' ')) > > ought to do the trick, assuming group returns tuples or lists of > characters. This is too slow and verbose and kill benefits of grouping function. From grosser.meister.morti at gmx.net Sun Dec 8 18:49:19 2013 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sun, 08 Dec 2013 18:49:19 +0100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <5251239173677340489@gmail297201516> References: <5251239173677340489@gmail297201516> Message-ID: <52A4B11F.7040200@gmx.net> On 12/08/2013 05:44 AM, Amber Yust wrote: > After seeing yet another person asking how to do this on #python (and having needed to do it in the past myself), I'm > wondering why itertools doesn't have a function to break an iterator up into N-sized chunks. > > Existing possible solutions include both the "clever" but somewhat unreadable... > > batched_iter = zip(*[iter(input_iter)]*n) > > ...and the long-form... > > def batch(input_iter, n): > input_iter = iter(input_iter) > while True: > yield [input_iter.next() for _ in range(n)] > This function drops items if the length of the input sequence is not a multiple of n. Fix: def batch(it, n): it = iter(it) while True: slice = [] for _ in range(n): try: slice.append(it.next()) except StopIteration: if slice: yield slice return yield slice > There doesn't seem, however, to be one clear "right" way to do this. Every time I come up against this task, I go back > to itertools expecting one of the grouping functions there to cover it, but they don't. > > It seems like it would be a natural fit for itertools, and it would simplify things like processing of file formats that > use a consistent number of lines per entry, et cetera. > > ~Amber > From amber.yust at gmail.com Sun Dec 8 19:06:05 2013 From: amber.yust at gmail.com (Amber Yust) Date: Sun, 08 Dec 2013 18:06:05 +0000 Subject: [Python-ideas] Batching/grouping function for itertools References: <5251239173677340489@gmail297201516> <52A4B11F.7040200@gmx.net> Message-ID: <-8796784632715378232@gmail297201516> So does zip if the items are of unequal length, and the two code examples I provided (the one using zip and the long-form one) are equivalent. On Sun Dec 08 2013 at 9:49:56 AM, Mathias Panzenb?ck < grosser.meister.morti at gmx.net> wrote: > On 12/08/2013 05:44 AM, Amber Yust wrote: > > After seeing yet another person asking how to do this on #python (and > having needed to do it in the past myself), I'm > > wondering why itertools doesn't have a function to break an iterator up > into N-sized chunks. > > > > Existing possible solutions include both the "clever" but somewhat > unreadable... > > > > batched_iter = zip(*[iter(input_iter)]*n) > > > > ...and the long-form... > > > > def batch(input_iter, n): > > input_iter = iter(input_iter) > > while True: > > yield [input_iter.next() for _ in range(n)] > > > > This function drops items if the length of the input sequence is not a > multiple of n. Fix: > > def batch(it, n): > it = iter(it) > while True: > slice = [] > for _ in range(n): > try: > slice.append(it.next()) > except StopIteration: > if slice: > yield slice > return > yield slice > > > There doesn't seem, however, to be one clear "right" way to do this. > Every time I come up against this task, I go back > > to itertools expecting one of the grouping functions there to cover it, > but they don't. > > > > It seems like it would be a natural fit for itertools, and it would > simplify things like processing of file formats that > > use a consistent number of lines per entry, et cetera. > > > > ~Amber > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sun Dec 8 19:24:28 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 08 Dec 2013 18:24:28 +0000 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <5251239173677340489@gmail297201516> References: <5251239173677340489@gmail297201516> Message-ID: On 08/12/2013 04:44, Amber Yust wrote: > After seeing yet another person asking how to do this on #python (and > having needed to do it in the past myself), I'm wondering why itertools > doesn't have a function to break an iterator up into N-sized chunks. > As discussed umpteen times previously, there is no way that we can agree on "a function" that can meet all of the variations that have been proposed on this theme. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From yoavglazner at gmail.com Sun Dec 8 19:57:32 2013 From: yoavglazner at gmail.com (yoav glazner) Date: Sun, 8 Dec 2013 20:57:32 +0200 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> Message-ID: Hi On Sun, Dec 8, 2013 at 8:24 PM, Mark Lawrence wrote: > On 08/12/2013 04:44, Amber Yust wrote: > >> After seeing yet another person asking how to do this on #python (and >> having needed to do it in the past myself), I'm wondering why itertools >> doesn't have a function to break an iterator up into N-sized chunks. >> >> > As discussed umpteen times previously, there is no way that we can agree > on "a function" that can meet all of the variations that have been proposed > on this theme. > > Maybe if we add this function: >>> def mod_pad(it, modulo, fillval): '"".join(mod_pad("hello", 3, fillval="!")) => hello!' for i, val in enumerate(iter(it)): yield val for _ in range(i%modulo): yield fillval >>> "".join(mod_pad("hello", 3, fillval="!")) 'hello!' Then we can make grouper/batcher throw an exception in the case of iter_len % modulo != 0 grouper(mod_pad("hello", 3, fillval="!"), 3) => hel, lo! (in a iterator...) grouper('hello', 3) => BOOM -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 8 22:56:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Dec 2013 07:56:45 +1000 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> Message-ID: On 9 Dec 2013 01:47, "Chris Angelico" wrote: > > On Mon, Dec 9, 2013 at 2:40 AM, Ron Adam wrote: > > The function would need to be a wrapper that keeps a buffer. And the front > > of the buffer would always be the next to be yielded if there is anything in > > it. > > Which is what Stephen said in his pithy message above. Yes, it's > theoretically possible, but it's not something to do in the general > case. Not something to depend on for a chunker/grouper, which should > be able to pass once over the underlying iterator. tee() is this building block. Cheers, Nick. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Dec 8 23:09:14 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 08 Dec 2013 17:09:14 -0500 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208121629.GV2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> <20131208121629.GV2085@ando> Message-ID: On 12/8/2013 7:16 AM, Steven D'Aprano wrote: > On Sun, Dec 08, 2013 at 01:30:56PM +0200, Serhiy Storchaka wrote: >> There is also a question about result's type. Sometimes you need an >> iterator of subsequences (i.e. split string on equal string chunks), >> sometimes an iterator of iterators is enough. > > None of the other itertools functions treat strings specially. Why > should this one? If you want to re-join them into strings, you can do so > with a trivial wrapper: > > (''.join(elements) for elements in group("some string", 3, pad=' ')) A large fraction, perhaps over half, of the multiple requests for a chunker or grouper function are for sequences, not general iterables, as input, with the desired output type being the input type. For this, an iterator of *slices* is *far* more efficient. The same function could easily handle overlaps. (There are still the possible varieties of short slice handling). *Untested*: def window(seq, size, advance=None, extra='skip'): '''Yield successive slices of len size of sequence seq. Move window advance items (default = size). Extra determines the handling of extra items. The options are 'skip' (default), 'keep', and 'raise'. ''' if overlap == None: advance = size i, j, n = 0, size, len(seq) while j <= n: yield seq[i:j] i += advance j += advance if j < n + advance: if extra == 'keep': yield seq[i:j] elif extra == 'raise' raise ValueError('extra items') else: raise ValueError('bad extra') Having gotten this far, it would be possible to treat the above as a fast path for sequences and wrap it in try:except and if len or slice fail, fall back to a general iterator version. The result could be a builtin rather than itertool. -- Terry Jan Reedy From grosser.meister.morti at gmx.net Sun Dec 8 23:26:21 2013 From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=) Date: Sun, 08 Dec 2013 23:26:21 +0100 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <-8796784632715378232@gmail297201516> References: <5251239173677340489@gmail297201516> <52A4B11F.7040200@gmx.net> <-8796784632715378232@gmail297201516> Message-ID: <52A4F20D.2020704@gmx.net> I see. Well, I woudln't expect that behaviour from such a function. On 12/08/2013 07:06 PM, Amber Yust wrote: > So does zip if the items are of unequal length, and the two code examples I provided (the one using zip and the > long-form one) are equivalent. > > On Sun Dec 08 2013 at 9:49:56 AM, Mathias Panzenb?ck > wrote: > > On 12/08/2013 05:44 AM, Amber Yust wrote: > > After seeing yet another person asking how to do this on #python (and having needed to do it in the past myself), I'm > > wondering why itertools doesn't have a function to break an iterator up into N-sized chunks. > > > > Existing possible solutions include both the "clever" but somewhat unreadable... > > > > batched_iter = zip(*[iter(input_iter)]*n) > > > > ...and the long-form... > > > > def batch(input_iter, n): > > input_iter = iter(input_iter) > > while True: > > yield [input_iter.next() for _ in range(n)] > > > > This function drops items if the length of the input sequence is not a multiple of n. Fix: > > def batch(it, n): > it = iter(it) > while True: > slice = [] > for _ in range(n): > try: > slice.append(it.next()) > except StopIteration: > if slice: > yield slice > return > yield slice > > > There doesn't seem, however, to be one clear "right" way to do this. Every time I come up against this task, I go > back > > to itertools expecting one of the grouping functions there to cover it, but they don't. > > > > It seems like it would be a natural fit for itertools, and it would simplify things like processing of file > formats that > > use a consistent number of lines per entry, et cetera. > > > > ~Amber > > > > _________________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/__mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/__codeofconduct/ > From antony.lee at berkeley.edu Mon Dec 9 08:11:08 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Sun, 8 Dec 2013 23:11:08 -0800 Subject: [Python-ideas] Automatically turning __prepare__ into a classmethod. Message-ID: Just a small idea... __new__ is automagically turned into a staticmethod, because other interpretations don't make sense. Likewise, perhaps __prepare__ could be automagically turned into a classmethod? Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From piotr.skamruk at gmail.com Mon Dec 9 10:07:35 2013 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Mon, 9 Dec 2013 10:07:35 +0100 Subject: [Python-ideas] Easily reference a single unittest from cmdline In-Reply-To: References: Message-ID: 2013/12/7 Giampaolo Rodola' : > I just took a look at nosetests. > It provides a similar thing and lets you use a regular expression > (which indeed makes a lot more sense than using wildcards) > It works like this: > > nosetests test/test_module -m test_name Check http://pytest.org/latest/usage.html#usage Probably: pytest -k part_of_test_name is what You are seeking. If You have a while - please look deeper on pytest - it's really powerful/user friendly tool. From musicdenotation at gmail.com Mon Dec 9 11:07:00 2013 From: musicdenotation at gmail.com (musicdenotation at gmail.com) Date: Mon, 09 Dec 2013 17:07:00 +0700 Subject: [Python-ideas] Pipe indentation Message-ID: <52a5966d.41c4440a.6bde.31d4@mx.google.com> if condition: | do_this | if cond2: | |do_that else: | do_this_instead This has the advantage of you always knowing which nesting level you are. From rosuav at gmail.com Mon Dec 9 11:11:09 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Dec 2013 21:11:09 +1100 Subject: [Python-ideas] Pipe indentation In-Reply-To: <52a5966d.41c4440a.6bde.31d4@mx.google.com> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: On Mon, Dec 9, 2013 at 9:07 PM, wrote: > if condition: > | do_this > | if cond2: > | |do_that > else: > | do_this_instead > > This has the advantage of you always knowing which nesting level you are. Alternative suggestion: Tab indentation. Do exactly the same thing, but replace every pipe with a tab. Then you can set your editor to show indentation visibly (SciTE does this already). ChrisA From daniel at daniel-watkins.co.uk Mon Dec 9 11:29:29 2013 From: daniel at daniel-watkins.co.uk (Daniel Watkins) Date: Mon, 9 Dec 2013 10:29:29 +0000 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: <20131209102929.GX22882@daniel-watkins.co.uk> On Mon, Dec 09, 2013 at 09:11:09PM +1100, Chris Angelico wrote: > On Mon, Dec 9, 2013 at 9:07 PM, wrote: > > if condition: > > | do_this > > | if cond2: > > | |do_that > > else: > > | do_this_instead > > > > This has the advantage of you always knowing which nesting level you are. > > Alternative suggestion: Tab indentation. Do exactly the same thing, > but replace every pipe with a tab. Then you can set your editor to > show indentation visibly (SciTE does this already). Alternative suggestion: indent with 4 spaces. Do exactly the same thing, but replace every pipe with 4 spaces. Then you can set your eyes to see the visible indentation. ;) More seriously, if you are at the point where you're getting confused by levels of indentation, you're probably either (a) nesting too much, or (b) including too much in indented blocks. Either of these can be solved by splitting your code up in to more smaller functions/methods (which, if you name them well, will have the side-effect of making your code clearer anyway!). Cheers, Dan From oscar.j.benjamin at gmail.com Mon Dec 9 11:54:40 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 9 Dec 2013 10:54:40 +0000 Subject: [Python-ideas] Batching/grouping function for itertools In-Reply-To: <20131208121629.GV2085@ando> References: <5251239173677340489@gmail297201516> <20131208092517.GU2085@ando> <20131208121629.GV2085@ando> Message-ID: On 8 December 2013 12:16, Steven D'Aprano wrote: > On Sun, Dec 08, 2013 at 01:30:56PM +0200, Serhiy Storchaka wrote: >> 08.12.13 11:25, Steven D'Aprano ???????(??): >> >In the second case, there is a question about what to do with sequences >> >that are not a multiple of the window size. Similar to zip(), there are >> >two things one might do: >> > >> >- pad with some given object; >> >- raise an exception >> >> 3) emit last chunk incompleted; > > Given a window size of two, and input data [a, b, c], are you suggesting > a variety that returns this? > > (a,b), (c,) This is the only variant I have ever used. You can see an example use case here: https://mail.python.org/pipermail//python-ideas/2013-August/022767.html And this is the implementation I use: def chunks(iterable, chunksize=100): islices = map(islice, repeat(iter(iterable)), repeat(chunksize)) return takewhile(bool, map(list, islices)) Oscar From denis.spir at gmail.com Mon Dec 9 15:00:33 2013 From: denis.spir at gmail.com (spir) Date: Mon, 09 Dec 2013 15:00:33 +0100 Subject: [Python-ideas] Pipe indentation In-Reply-To: <52a5966d.41c4440a.6bde.31d4@mx.google.com> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: <52A5CD01.7010605@gmail.com> On 12/09/2013 11:07 AM, musicdenotation at gmail.com wrote: > if condition: > | do_this > | if cond2: > | |do_that > else: > | do_this_instead > > This has the advantage of you always knowing which nesting level you are. Some programming editors, like Geany, do that for you :-) if you set the right setting (albeit with a "thinissim" greay dotted line). Denis From flying-sheep at web.de Mon Dec 9 15:19:29 2013 From: flying-sheep at web.de (Philipp A.) Date: Mon, 9 Dec 2013 15:19:29 +0100 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: 2013/12/9 Chris Angelico > Alternative suggestion: Tab indentation. Do exactly the same thing, > but replace every pipe with a tab. Then you can set your editor to > show indentation visibly (SciTE does this already). > > ChrisA > that?s also my favourite way to do it. some editors can show tabs as an slim unobtrusive slim line and leading/trailing spaces as middle dots (?) that way you can not only instantly see all misplaced whitespace characters, but also indentation levels. (one tab per indentation level is pretty much as semantically unambiguous as it can get) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Mon Dec 9 15:45:58 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 9 Dec 2013 08:45:58 -0600 Subject: [Python-ideas] Pipe indentation In-Reply-To: <20131209102929.GX22882@daniel-watkins.co.uk> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <20131209102929.GX22882@daniel-watkins.co.uk> Message-ID: > More seriously, if you are at the point where you're getting confused by > levels of indentation, you're probably either (a) nesting too much, or > (b) including too much in indented blocks. Or (c), apparently only using one-space indentation. :-) Skip From breamoreboy at yahoo.co.uk Mon Dec 9 16:17:36 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 09 Dec 2013 15:17:36 +0000 Subject: [Python-ideas] Pipe indentation In-Reply-To: <20131209102929.GX22882@daniel-watkins.co.uk> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <20131209102929.GX22882@daniel-watkins.co.uk> Message-ID: On 09/12/2013 10:29, Daniel Watkins wrote: > On Mon, Dec 09, 2013 at 09:11:09PM +1100, Chris Angelico wrote: >> On Mon, Dec 9, 2013 at 9:07 PM, wrote: >>> if condition: >>> | do_this >>> | if cond2: >>> | |do_that >>> else: >>> | do_this_instead >>> >>> This has the advantage of you always knowing which nesting level you are. >> >> Alternative suggestion: Tab indentation. Do exactly the same thing, >> but replace every pipe with a tab. Then you can set your editor to >> show indentation visibly (SciTE does this already). > > Alternative suggestion: indent with 4 spaces. Do exactly the same > thing, but replace every pipe with 4 spaces. Then you can set your eyes > to see the visible indentation. > > ;) > > > More seriously, if you are at the point where you're getting confused by > levels of indentation, you're probably either (a) nesting too much, or > (b) including too much in indented blocks. Either of these can be > solved by splitting your code up in to more smaller functions/methods > (which, if you name them well, will have the side-effect of making your > code clearer anyway!). > > > Cheers, > > Dan Alternative alternative alternative suggestion don't post items like this as I've just barfed all over my screen :( -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From flying-sheep at web.de Mon Dec 9 20:22:05 2013 From: flying-sheep at web.de (Philipp A.) Date: Mon, 9 Dec 2013 20:22:05 +0100 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <20131209102929.GX22882@daniel-watkins.co.uk> Message-ID: bad joke, that makes it even less clear. but @OP: yeah, syntax highlighting is the way to go as mainainung pipes and spaces/tabs would be too much hassle even with editor support. 2013/12/9 Skip Montanaro > > More seriously, if you are at the point where you're getting confused by > > levels of indentation, you're probably either (a) nesting too much, or > > (b) including too much in indented blocks. > > Or (c), apparently only using one-space indentation. :-) > > Skip > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjwelborn at live.com Tue Dec 10 04:13:20 2013 From: cjwelborn at live.com (Christopher Welborn) Date: Mon, 09 Dec 2013 21:13:20 -0600 Subject: [Python-ideas] Pipe indentation In-Reply-To: <52a5966d.41c4440a.6bde.31d4@mx.google.com> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: On 12/09/2013 04:07 AM, musicdenotation at gmail.com wrote: > if condition: > | do_this > | if cond2: > | |do_that > else: > | do_this_instead > > This has the advantage of you always knowing which nesting level you are. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Here's a link to a screen shot that shows Sublime Text's nesting level lines: http://imgur.com/yRGNCKu ...a lot of editors do this, tab or space. p.s. - it's not my code in the screen shot, its from the powerline module for vim (just grabbed the closest thing with nesting). -- - Christopher Welborn http://welbornprod.com From stephen at xemacs.org Tue Dec 10 06:21:07 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Dec 2013 14:21:07 +0900 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> Message-ID: <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Christopher Welborn writes: > Here's a link to a screen shot that shows Sublime Text's nesting level > lines: > http://imgur.com/yRGNCKu > ...a lot of editors do this, tab or space. But do they get it Pythonically correct? Ie, in counting indentation, 1 TAB = 1 SPC, regardless of what it looks like on screen. From rosuav at gmail.com Tue Dec 10 06:48:16 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 Dec 2013 16:48:16 +1100 Subject: [Python-ideas] Pipe indentation In-Reply-To: <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 10, 2013 at 4:21 PM, Stephen J. Turnbull wrote: > Christopher Welborn writes: > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > lines: > > > http://imgur.com/yRGNCKu > > > ...a lot of editors do this, tab or space. > > But do they get it Pythonically correct? Ie, in counting indentation, > 1 TAB = 1 SPC, regardless of what it looks like on screen. SciTE gets it Python3ically correct: tab != space regardless of number. ChrisA From guido at python.org Tue Dec 10 07:31:24 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Dec 2013 22:31:24 -0800 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Dec 9, 2013 at 9:48 PM, Chris Angelico wrote: > On Tue, Dec 10, 2013 at 4:21 PM, Stephen J. Turnbull wrote: >> Christopher Welborn writes: >> >> > Here's a link to a screen shot that shows Sublime Text's nesting level >> > lines: >> >> > http://imgur.com/yRGNCKu >> >> > ...a lot of editors do this, tab or space. >> >> But do they get it Pythonically correct? Ie, in counting indentation, >> 1 TAB = 1 SPC, regardless of what it looks like on screen. The default Python 2 rule is actually that 1 TAB == 8 SPC. > SciTE gets it Python3ically correct: tab != space regardless of number. Right. And in Python 2 you can get the same effect with python -tt. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Tue Dec 10 08:02:47 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Dec 2013 16:02:47 +0900 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > On Tue, Dec 10, 2013 at 4:21 PM, Stephen J. Turnbull wrote: > > Christopher Welborn writes: > > > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > > lines: > > > > > http://imgur.com/yRGNCKu > > > > > ...a lot of editors do this, tab or space. > > > > But do they get it Pythonically correct? Ie, in counting indentation, > > 1 TAB = 1 SPC, regardless of what it looks like on screen. Oops, that's TAB = SPC*8 in Python 2. > SciTE gets it Python3ically correct: tab != space regardless of > number. But what does that mean? How does SciTE distinguish between "SPC TAB" at the beginning of line and just "TAB"? Where does it place guidelines given def f(x): SPC if x: SPC SPC print("gotcha") SPC else: SPC TAB print("gotcha again") which is acceptable to Python 3.3? I suppose the right answer, as usual, is "don't use both spaces and tabs for indentation in the same program", but I guess then it really doesn't matter how the editor handles tabs and spaces. Oh, BTW, I'm definitely -1 on "pipe indentation" in the source program, since editors can clearly do it if you want it (I don't) whether it's present in the source, or not. From rosuav at gmail.com Tue Dec 10 08:10:58 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 Dec 2013 18:10:58 +1100 Subject: [Python-ideas] Pipe indentation In-Reply-To: <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 10, 2013 at 6:02 PM, Stephen J. Turnbull wrote: > But what does that mean? How does SciTE distinguish between "SPC TAB" > at the beginning of line and just "TAB"? Where does it place guidelines > given > > def f(x): > SPC if x: > SPC SPC print("gotcha") > SPC else: > SPC TAB print("gotcha again") Hmm, actually I have no idea, so I just checked. Seems I was slightly wrong in my description; the guides seem to be at fixed column positions (configurable, but not dynamic). The differentiation between tab and space is only for its signalling highlight - if you mismatch indentation in any way, the wrong one is highlighted with a very visible marker. And that highlight correctly accepts your above example, though it's a little finicky to type (putting a space, then a tab, on an otherwise-blank line deletes the space, but one can be explicitly inserted after the tab's already there). The upshot is that you get instant feedback if you ever get tabs/spaces wrong, and you get nice guides if you always indent consistently, and in between is shooting yourself in the foot anyway. ChrisA From stephen at xemacs.org Tue Dec 10 09:35:43 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 10 Dec 2013 17:35:43 +0900 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87wqjd14tc.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > The upshot is that you get instant feedback if you ever get > tabs/spaces wrong, and you get nice guides if you always indent > consistently, and in between is shooting yourself in the foot > anyway. Sounds like somebody's thought this through pretty carefully! I'm an emacs guy, so unlikely to switch, but that description sounds like a good specification to start with if we add that kind of functionality to our python-mode. From solipsis at pitrou.net Tue Dec 10 14:35:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Dec 2013 14:35:32 +0100 Subject: [Python-ideas] Pipe indentation References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20131210143532.4fb0bda8@fsol> On Tue, 10 Dec 2013 14:21:07 +0900 "Stephen J. Turnbull" wrote: > Christopher Welborn writes: > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > lines: > > > http://imgur.com/yRGNCKu > > > ...a lot of editors do this, tab or space. > > But do they get it Pythonically correct? Ie, in counting indentation, > 1 TAB = 1 SPC, regardless of what it looks like on screen. Generally they're purely graphical, i.e. they don't care about the *logical nesting* of your code, only about how many spaces are drawn on screen. (no, I don't know what it does with a non-monospaced font) Regards Antoine. From cjwelborn at live.com Tue Dec 10 14:55:01 2013 From: cjwelborn at live.com (Christopher Welborn) Date: Tue, 10 Dec 2013 07:55:01 -0600 Subject: [Python-ideas] Pipe indentation In-Reply-To: <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 12/09/2013 11:21 PM, Stephen J. Turnbull wrote: > Christopher Welborn writes: > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > lines: > > > http://imgur.com/yRGNCKu > > > ...a lot of editors do this, tab or space. > > But do they get it Pythonically correct? Ie, in counting indentation, > 1 TAB = 1 SPC, regardless of what it looks like on screen. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Whatever you set the tab-spacing to is what it shows up as. I set mine to 4 so the pic shows 4. If you set it to 1 or 2 you will see a lot of vertical lines. -- - Christopher Welborn http://welbornprod.com From guido at python.org Tue Dec 10 16:47:40 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Dec 2013 07:47:40 -0800 Subject: [Python-ideas] Pipe indentation In-Reply-To: <87wqjd14tc.fsf@uwakimon.sk.tsukuba.ac.jp> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqjd14tc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Dec 10, 2013 at 12:35 AM, Stephen J. Turnbull wrote: > Chris Angelico writes: > > > The upshot is that you get instant feedback if you ever get > > tabs/spaces wrong, and you get nice guides if you always indent > > consistently, and in between is shooting yourself in the foot > > anyway. > > Sounds like somebody's thought this through pretty carefully! I'm an > emacs guy, so unlikely to switch, but that description sounds like a > good specification to start with if we add that kind of functionality > to our python-mode. Probably the first time significant thought was put into this was when I designed the -t/-tt option for Python 2. I don't recall when that was but it was in a very early Python 2 release. Everything else seems just derivative to me -- either the same solution, or buggy. :-) -- --Guido van Rossum (python.org/~guido) From tinchester at gmail.com Tue Dec 10 19:29:57 2013 From: tinchester at gmail.com (=?UTF-8?Q?Tin_Tvrtkovi=C4=87?=) Date: Tue, 10 Dec 2013 19:29:57 +0100 Subject: [Python-ideas] Automagically set logger name Message-ID: Hello, usually logger instances are retrieved and initialized with the module name, using the well-known pattern: logger = logging.getLogger(__name__) In Java's very popular log4j library (which is cited as an influence by PEP 282), loggers are usually retrieved and initialized in basically an identical way: private final static Logger LOG = Logger.getLogger(.class); However, in the upcoming log4j 2 library, a new way is available: private final static Logger LOG = LogManager.getLogger(); // Returns a Logger with the name of the calling class. [1] Basically the method throws an exception, catches it and fishes out the class name from the stack trace. This is a little less explicit, but still a more convenient (and less annoying) way of accomplishing an extremely common pattern. I was wondering what the core devs make of it: - is it a good idea? (In general, and in Python) Is it worth it? - is it feasible in Python? (taking into account other implementations too) - are there any gotchas that would make it worse than the current standard? [1]: https://svn.apache.org/repos/asf/logging/log4j/log4j2/trunk/log4j-api/src/main/java/org/apache/logging/log4j/LogManager.java -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Dec 10 19:32:34 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 10 Dec 2013 12:32:34 -0600 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqjd14tc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: I now feel dumbfounded because I never knew the -t/-tt option was there until now... On Tue, Dec 10, 2013 at 9:47 AM, Guido van Rossum wrote: > On Tue, Dec 10, 2013 at 12:35 AM, Stephen J. Turnbull > wrote: > > Chris Angelico writes: > > > > > The upshot is that you get instant feedback if you ever get > > > tabs/spaces wrong, and you get nice guides if you always indent > > > consistently, and in between is shooting yourself in the foot > > > anyway. > > > > Sounds like somebody's thought this through pretty carefully! I'm an > > emacs guy, so unlikely to switch, but that description sounds like a > > good specification to start with if we add that kind of functionality > > to our python-mode. > > Probably the first time significant thought was put into this was when > I designed the -t/-tt option for Python 2. I don't recall when that > was but it was in a very early Python 2 release. Everything else seems > just derivative to me -- either the same solution, or buggy. :-) > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From boehm.matthew at gmail.com Tue Dec 10 20:51:43 2013 From: boehm.matthew at gmail.com (Matthew Boehm) Date: Tue, 10 Dec 2013 14:51:43 -0500 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <87y53t1948.fsf@uwakimon.sk.tsukuba.ac.jp> <87wqjd14tc.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Just to throw another editor's hat in the ring, the best way to achieve this in vim is with the indent-guides plugin https://github.com/nathanaelkane/vim-indent-guides however in stock vim, you can use 'listchars' to render a tab as a pipe followed by spaces. -Matt Boehm On Tue, Dec 10, 2013 at 1:32 PM, Ryan Gonzalez wrote: > I now feel dumbfounded because I never knew the -t/-tt option was there > until now... > > > On Tue, Dec 10, 2013 at 9:47 AM, Guido van Rossum wrote: > >> On Tue, Dec 10, 2013 at 12:35 AM, Stephen J. Turnbull >> wrote: >> > Chris Angelico writes: >> > >> > > The upshot is that you get instant feedback if you ever get >> > > tabs/spaces wrong, and you get nice guides if you always indent >> > > consistently, and in between is shooting yourself in the foot >> > > anyway. >> > >> > Sounds like somebody's thought this through pretty carefully! I'm an >> > emacs guy, so unlikely to switch, but that description sounds like a >> > good specification to start with if we add that kind of functionality >> > to our python-mode. >> >> Probably the first time significant thought was put into this was when >> I designed the -t/-tt option for Python 2. I don't recall when that >> was but it was in a very early Python 2 release. Everything else seems >> just derivative to me -- either the same solution, or buggy. :-) >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Ryan > When your hammer is C++, everything begins to look like a thumb. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Wed Dec 11 01:39:45 2013 From: antony.lee at berkeley.edu (Antony Lee) Date: Tue, 10 Dec 2013 16:39:45 -0800 Subject: [Python-ideas] Slightly changing the signature of the inspect.Parameter constructor Message-ID: It seems to me that POSITIONAL_OR_KEYWORD is the most often used kind of Parameter (after all, this is the "default" kind of Parameter), so perhaps the constructor for Parameter could be changed from def __init__(self, name, kind, *, default=_empty, annotation=_empty, _partial_kwarg=False): to def __init__(self, name, kind=Parameter.POSITIONAL_OR_KEYWORD, *, default=_empty, annotation=_empty, _partial_kwarg=False): Any thoughts on that? Best, Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Thu Dec 12 00:45:45 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 11 Dec 2013 23:45:45 +0000 Subject: [Python-ideas] Pipe indentation In-Reply-To: <20131210143532.4fb0bda8@fsol> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp>,<20131210143532.4fb0bda8@fsol> Message-ID: A space is always the same width, and tabs are normally rendered as multiple spaces in code editors, so it works just fine. Word wrap is where things get fun... (though I've only had one person complain that my Visual Studio extension doesn't handle it properly... who wraps code?) Top posted from my Windows Phone ________________________________ From: Antoine Pitrou Sent: ?12/?11/?2013 0:36 To: python-ideas at python.org Subject: Re: [Python-ideas] Pipe indentation On Tue, 10 Dec 2013 14:21:07 +0900 "Stephen J. Turnbull" wrote: > Christopher Welborn writes: > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > lines: > > > http://imgur.com/yRGNCKu > > > ...a lot of editors do this, tab or space. > > But do they get it Pythonically correct? Ie, in counting indentation, > 1 TAB = 1 SPC, regardless of what it looks like on screen. Generally they're purely graphical, i.e. they don't care about the *logical nesting* of your code, only about how many spaces are drawn on screen. (no, I don't know what it does with a non-monospaced font) Regards Antoine. _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Dec 12 02:11:13 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Dec 2013 17:11:13 -0800 Subject: [Python-ideas] Pipe indentation In-Reply-To: References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <20131210143532.4fb0bda8@fsol> Message-ID: <7BC2F430-08C5-46A2-A2EB-FFF24DB15464@yahoo.com> On Dec 11, 2013, at 15:45, Steve Dower wrote: > A space is always the same width, and tabs are normally rendered as multiple spaces in code editors, so it works just fine. Word wrap is where things get fun... (though I've only had one person complain that my Visual Studio extension doesn't handle it properly... who wraps code?) You mean viewing soft-wrapped if it goes beyond the window width? I do that (when I can't just reject or edit the code to fit properly in the first place). Much better than horizontally scrolling, especially in a terminal session. It's ugly no matter what you do, but at least you can see it. > > Top posted from my Windows Phone > From: Antoine Pitrou > Sent: ?12/?11/?2013 0:36 > To: python-ideas at python.org > Subject: Re: [Python-ideas] Pipe indentation > > On Tue, 10 Dec 2013 14:21:07 +0900 > "Stephen J. Turnbull" > wrote: > > Christopher Welborn writes: > > > > > Here's a link to a screen shot that shows Sublime Text's nesting level > > > lines: > > > > > http://imgur.com/yRGNCKu > > > > > ...a lot of editors do this, tab or space. > > > > But do they get it Pythonically correct? Ie, in counting indentation, > > 1 TAB = 1 SPC, regardless of what it looks like on screen. > > Generally they're purely graphical, i.e. they don't care about the > *logical nesting* of your code, only about how many spaces are > drawn on screen. > (no, I don't know what it does with a non-monospaced font) > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Dec 12 23:05:58 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 12 Dec 2013 16:05:58 -0600 Subject: [Python-ideas] Pipe indentation In-Reply-To: <7BC2F430-08C5-46A2-A2EB-FFF24DB15464@yahoo.com> References: <52a5966d.41c4440a.6bde.31d4@mx.google.com> <871u1l2se4.fsf@uwakimon.sk.tsukuba.ac.jp> <20131210143532.4fb0bda8@fsol> <7BC2F430-08C5-46A2-A2EB-FFF24DB15464@yahoo.com> Message-ID: I usually span the code across multiple lines if it doesn't fit in the IDLE window. I save gedit's Word Wrap for when I'm lazy. On Wed, Dec 11, 2013 at 7:11 PM, Andrew Barnert wrote: > On Dec 11, 2013, at 15:45, Steve Dower wrote: > > A space is always the same width, and tabs are normally rendered as > multiple spaces in code editors, so it works just fine. Word wrap is where > things get fun... (though I've only had one person complain that my Visual > Studio extension doesn't handle it properly... who wraps code?) > > > You mean viewing soft-wrapped if it goes beyond the window width? I do > that (when I can't just reject or edit the code to fit properly in the > first place). Much better than horizontally scrolling, especially in a > terminal session. It's ugly no matter what you do, but at least you can see > it. > > > Top posted from my Windows Phone > ------------------------------ > From: Antoine Pitrou > Sent: 12/11/2013 0:36 > To: python-ideas at python.org > Subject: Re: [Python-ideas] Pipe indentation > > On Tue, 10 Dec 2013 14:21:07 +0900 > "Stephen J. Turnbull" > wrote: > > Christopher Welborn writes: > > > > > Here's a link to a screen shot that shows Sublime Text's nesting > level > > > lines: > > > > > http://imgur.com/yRGNCKu > > > > > ...a lot of editors do this, tab or space. > > > > But do they get it Pythonically correct? Ie, in counting indentation, > > 1 TAB = 1 SPC, regardless of what it looks like on screen. > > Generally they're purely graphical, i.e. they don't care about the > *logical nesting* of your code, only about how many spaces are > drawn on screen. > (no, I don't know what it does with a non-monospaced font) > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fcoaps at yahoo.com Tue Dec 17 14:07:43 2013 From: fcoaps at yahoo.com (f p) Date: Tue, 17 Dec 2013 05:07:43 -0800 (PST) Subject: [Python-ideas] Warning on conditionals Message-ID: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> Hi, I'm new here, There are cases of bad use of int or str constants without comparison operators in the if statement as shown in this example: http://stackoverflow.com/questions/20633818/python-if-elif-always-not-working. Maybe the case in which the if statement is defeated should rate a warning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From __peter__ at web.de Tue Dec 17 14:46:58 2013 From: __peter__ at web.de (Peter Otten) Date: Tue, 17 Dec 2013 14:46:58 +0100 Subject: [Python-ideas] Warning on conditionals References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> Message-ID: f p wrote: > Hi, I'm new here, > > There are cases of bad use of int or str constants without comparison > operators in the if statement as shown in this example: > > http://stackoverflow.com/questions/20633818/python-if-elif-always-not- working. > > Maybe the case in which the if statement is defeated should rate a > warning. When a newbie writes if animal == "cat" or "dog": ... it is clear that he has the wrong idea about how this is evaluated, but in the general case if animal == cat or animal_barks: ... this is not an error. A check would be of very limited use. That said, there is an external tool that issues a warning: $ cat tmp.py import random a = random.choice("ab") if a == "b" or "c": print "yes" else: assert False, "unreachable" $ pychecker tmp.py Processing module tmp (tmp.py)... yes Warnings... tmp.py:3: Using a conditional statement with a constant value (c) From python at mrabarnett.plus.com Tue Dec 17 16:14:29 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 Dec 2013 15:14:29 +0000 Subject: [Python-ideas] Warning on conditionals In-Reply-To: References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> Message-ID: <52B06A55.4080001@mrabarnett.plus.com> On 17/12/2013 13:46, Peter Otten wrote: > f p wrote: > >> Hi, I'm new here, >> >> There are cases of bad use of int or str constants without comparison >> operators in the if statement as shown in this example: >> >> http://stackoverflow.com/questions/20633818/python-if-elif-always-not- > working. >> >> Maybe the case in which the if statement is defeated should rate a >> warning. > > When a newbie writes > > if animal == "cat" or "dog": > ... > > it is clear that he has the wrong idea about how this is evaluated, but in > the general case > > if animal == cat or animal_barks: > ... > > this is not an error. A check would be of very limited use. > That said, there is an external tool that issues a warning: > > $ cat tmp.py > import random > a = random.choice("ab") > if a == "b" or "c": > print "yes" > else: > assert False, "unreachable" > $ pychecker tmp.py > Processing module tmp (tmp.py)... > yes > > Warnings... > > tmp.py:3: Using a conditional statement with a constant value (c) > It's the kind of thing that you could have in IDLE. From amber.yust at gmail.com Tue Dec 17 16:34:38 2013 From: amber.yust at gmail.com (Amber Yust) Date: Tue, 17 Dec 2013 15:34:38 +0000 Subject: [Python-ideas] Warning on conditionals References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> <52B06A55.4080001@mrabarnett.plus.com> Message-ID: <8131640268748924455@gmail297201516> A better check might be to look for cases where both (a) and/or are used in a condition and (b) the right-hand side of the and/or is a constant. On Tue Dec 17 2013 at 7:15:06 AM, MRAB wrote: > On 17/12/2013 13:46, Peter Otten wrote: > > f p wrote: > > > >> Hi, I'm new here, > >> > >> There are cases of bad use of int or str constants without comparison > >> operators in the if statement as shown in this example: > >> > >> http://stackoverflow.com/questions/20633818/python-if-elif-always-not- > > working. > >> > >> Maybe the case in which the if statement is defeated should rate a > >> warning. > > > > When a newbie writes > > > > if animal == "cat" or "dog": > > ... > > > > it is clear that he has the wrong idea about how this is evaluated, but > in > > the general case > > > > if animal == cat or animal_barks: > > ... > > > > this is not an error. A check would be of very limited use. > > That said, there is an external tool that issues a warning: > > > > $ cat tmp.py > > import random > > a = random.choice("ab") > > if a == "b" or "c": > > print "yes" > > else: > > assert False, "unreachable" > > $ pychecker tmp.py > > Processing module tmp (tmp.py)... > > yes > > > > Warnings... > > > > tmp.py:3: Using a conditional statement with a constant value (c) > > > It's the kind of thing that you could have in IDLE. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis.spir at gmail.com Tue Dec 17 17:15:42 2013 From: denis.spir at gmail.com (spir) Date: Tue, 17 Dec 2013 17:15:42 +0100 Subject: [Python-ideas] Warning on conditionals In-Reply-To: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> Message-ID: <52B078AE.6060806@gmail.com> On 12/17/2013 02:07 PM, f p wrote: > Hi, I'm new here, > > There are cases of bad use of int or str constants without comparison operators in the if statement as shown in this example: > > http://stackoverflow.com/questions/20633818/python-if-elif-always-not-working. > > Maybe the case in which the if statement is defeated should rate a warning. Original code (from stackoverflow) is: if taxon == "Bracelets": catId = "178785" elif taxon == "Kids Earrings" or "Earrings": catId = "177591" [...more elif's...] The actual design issue, in my view, is reproducing in programming operator priority from math syntax. If there were no such priority (at least, for logical operators), then people would have to write: elif (taxon == "Kids Earrings") or "Earrings": catId = "177591" ...which makes semantic error obvious, or at least easy to understand, since in: elif taxon == ("Kids Earrings" or "Earrings"): catId = "177591" the second operand is a logical value, compared to a 'taxon'. Without parenthesis grouping, however, a "fuzzy logic" (lol!), natural-language-like meaning is well possible, indeed it is the original coder's intent. Related, but distinct, it is well known that priority of logical and/or is quite an effective source of bugs (also, the closer binding of not). And I remember a study about expressions with operators [*] showing how common bugs about plain arithmetic operator priority are (+-*/), as surprising as is. Evident conclusion, i guess is that operator priority in programming is bad. (For additional reasons, I guess the right choice in programming is prefix syntax operations.) [**] Denis [*] The purpose of the study, IIRC, was about syntactic complexity of operations, but it produced such data about error sources as a side-product. [**] Actually, the same applies to math syntax, but there a revolution is even less probable -- see how reluctant mathematicians are to abandon the funny idea of PI beeing "the circle constant". http://tauday.com/ From abarnert at yahoo.com Tue Dec 17 18:02:20 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 17 Dec 2013 09:02:20 -0800 Subject: [Python-ideas] Warning on conditionals In-Reply-To: <52B078AE.6060806@gmail.com> References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> <52B078AE.6060806@gmail.com> Message-ID: On Dec 17, 2013, at 8:15, spir wrote: > On 12/17/2013 02:07 PM, f p wrote: >> Hi, I'm new here, >> >> There are cases of bad use of int or str constants without comparison operators in the if statement as shown in this example: >> >> http://stackoverflow.com/questions/20633818/python-if-elif-always-not-working. >> >> Maybe the case in which the if statement is defeated should rate a warning. > > Original code (from stackoverflow) is: > > if taxon == "Bracelets": > catId = "178785" > elif taxon == "Kids Earrings" or "Earrings": > catId = "177591" > [...more elif's...] > > The actual design issue, in my view, is reproducing in programming operator priority from math syntax. If there were no such priority (at least, for logical operators), then people would have to write: > > elif (taxon == "Kids Earrings") or "Earrings": > catId = "177591" > > ...which makes semantic error obvious, or at least easy to understand, since in: > > elif taxon == ("Kids Earrings" or "Earrings"): > catId = "177591" > > the second operand is a logical value, compared to a 'taxon'. No it isn't. The second operand is the string "Kids Earrings". Which would be harder to debug, not easier. > > Without parenthesis grouping, however, a "fuzzy logic" (lol!), natural-language-like meaning is well possible, indeed it is the original coder's intent. > > Related, but distinct, it is well known that priority of logical and/or is quite an effective source of bugs (also, the closer binding of not). And I remember a study about expressions with operators [*] showing how common bugs about plain arithmetic operator priority are (+-*/), as surprising as is. > Evident conclusion, i guess is that operator priority in programming is bad. (For additional reasons, I guess the right choice in programming is prefix syntax operations.) [**] Operator precedence is surprising once, but makes your life easier once you learn it. Prefix notation is unreadable once, and remains unreadable even after you learn it. It's as high a barrier as postfix (RPN) notation, but without the advantages. (Plenty of engineers use their trusty HP 48 happily. But no one uses prefix notation except programmers in languages like Java and Lisp dialects without infix, and all of them complain about it.) I assume no one is going to suggest turning Python into Forth, so why suggest turning it into a form that has all the disadvantages of both with the advantages of neither? Anyway, it's easy to write a linter that enforces simple rules like not relying on precedence (forcing explicit parens), and then people (and teams, and teachers) who want to follow those rules can do so. > > Denis > > [*] The purpose of the study, IIRC, was about syntactic complexity of operations, but it produced such data about error sources as a side-product. > > [**] Actually, the same applies to math syntax, but there a revolution is even less probable -- see how reluctant mathematicians are to abandon the funny idea of PI beeing "the circle constant". http://tauday.com/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Tue Dec 17 18:07:36 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 17 Dec 2013 09:07:36 -0800 Subject: [Python-ideas] Warning on conditionals In-Reply-To: <52B06A55.4080001@mrabarnett.plus.com> References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> <52B06A55.4080001@mrabarnett.plus.com> Message-ID: <9016F381-0762-41F7-A2B4-E08C7B2E8385@yahoo.com> On Dec 17, 2013, at 7:14, MRAB wrote: > On 17/12/2013 13:46, Peter Otten wrote: >> When a newbie writes >> >> if animal == "cat" or "dog": >> ... >> >> it is clear that he has the wrong idea about how this is evaluated, but in >> the general case >> >> if animal == cat or animal_barks: >> ... >> >> this is not an error. A check would be of very limited use. >> That said, there is an external tool that issues a warning: >> >> $ cat tmp.py >> import random >> a = random.choice("ab") >> if a == "b" or "c": >> print "yes" >> else: >> assert False, "unreachable" >> $ pychecker tmp.py >> Processing module tmp (tmp.py)... >> yes >> >> Warnings... >> >> tmp.py:3: Using a conditional statement with a constant value (c) > It's the kind of thing that you could have in IDLE. We have multiple different linters and style checkers for a reason. Until one of them is so clearly perfect that there's no need for the others, I think it would be more useful to make it easier to integrate checkers into IDLE than to try to duplicate their functionality. Also, a pretty high percentage of newbies don't use IDLE, so this wouldn't solve the problem for most people. From tjreedy at udel.edu Tue Dec 17 20:55:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 17 Dec 2013 14:55:05 -0500 Subject: [Python-ideas] Warning on conditionals In-Reply-To: <9016F381-0762-41F7-A2B4-E08C7B2E8385@yahoo.com> References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> <52B06A55.4080001@mrabarnett.plus.com> <9016F381-0762-41F7-A2B4-E08C7B2E8385@yahoo.com> Message-ID: On 12/17/2013 12:07 PM, Andrew Barnert wrote: > On Dec 17, 2013, at 7:14, MRAB wrote: >>> tmp.py:3: Using a conditional statement with a constant value (c) >> It's the kind of thing that you could have in IDLE. > > We have multiple different linters and style checkers for a reason. Until one of them is so clearly perfect that there's no need for the others, I think it would be more useful to make it easier to integrate checkers into IDLE than to try to duplicate their functionality. There is an issue on the tracker to make it easy to integrate code checkers into Idle. -- Terry Jan Reedy From haoyi.sg at gmail.com Tue Dec 17 21:01:28 2013 From: haoyi.sg at gmail.com (Haoyi Li) Date: Tue, 17 Dec 2013 12:01:28 -0800 Subject: [Python-ideas] Warning on conditionals In-Reply-To: References: <1387285663.29639.YahooMailNeo@web161701.mail.bf1.yahoo.com> <52B06A55.4080001@mrabarnett.plus.com> <9016F381-0762-41F7-A2B4-E08C7B2E8385@yahoo.com> Message-ID: > But no one uses prefix notation except programmers in languages like Java and Lisp dialects without infix And I must say, even in Java and Javascript, the libraries with APIs people love often are the ones who use heavy method-chaining, like Guava and JQuery and Underscore. Other than the fact that the method names are alphanumeric instead of symbolic, method chaining is basically infix notation too. On Tue, Dec 17, 2013 at 11:55 AM, Terry Reedy wrote: > On 12/17/2013 12:07 PM, Andrew Barnert wrote: > >> On Dec 17, 2013, at 7:14, MRAB wrote: >> > > tmp.py:3: Using a conditional statement with a constant value (c) >>>> >>> It's the kind of thing that you could have in IDLE. >>> >> >> We have multiple different linters and style checkers for a reason. Until >> one of them is so clearly perfect that there's no need for the others, I >> think it would be more useful to make it easier to integrate checkers into >> IDLE than to try to duplicate their functionality. >> > > There is an issue on the tracker to make it easy to integrate code > checkers into Idle. > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhtonyuk at gmail.com Sat Dec 21 14:14:05 2013 From: mikhtonyuk at gmail.com (Sergii Mikhtoniuk) Date: Sat, 21 Dec 2013 15:14:05 +0200 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming Message-ID: Hi all, I would very much appreciate your opinion on my proposal for improvement of *concurrent.futures* package. Comparing to other languages such as Scala and C#, Python?s futures significantly fall behind in functionality especially in ability to chain computations and compose different futures without blocking and waiting for result. New packages continue to emerge (*asyncio*) which provide their own futures implementation, making composition even more difficult. Proposed improvement implements Scala-like Future as a monadic construct. It allows performing multiple kinds of operations on Future?s result without blocking, enabling reactive programming in Python. It implements common pattern separating *Future* and *Promise* interface, making it very easy for 3rd party systems to use futures in their API. Please have a look at this PEP draft, and reference implementation (as separate library). I?m very interested in: - How PEPable is this? - What are your thoughts on backward compatibility (current implementation does not sacrifice any design points for it, but better compatibility can be achieved)? - Thoughts on Future-based APIs in other packages? Thanks, Sergii -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Sat Dec 21 14:36:21 2013 From: phd at phdru.name (Oleg Broytman) Date: Sat, 21 Dec 2013 14:36:21 +0100 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: <20131221133621.GA23931@phdru.name> Hi! On Sat, Dec 21, 2013 at 03:14:05PM +0200, Sergii Mikhtoniuk wrote: > Please have a look at this PEP > draft, It would be nice to have a text version of the PEP posted to the mailing list. > and reference implementation (as > separate library). May I advice to add it to the References section of the PEP? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ncoghlan at gmail.com Sat Dec 21 15:12:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Dec 2013 00:12:46 +1000 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On 21 December 2013 23:14, Sergii Mikhtoniuk wrote: > Please have a look at this PEP draft, and reference implementation (as > separate library). > > I?m very interested in: > - How PEPable is this? This looks like a really interesting idea, and well worth pursuing as a PEP for 3.5 (the failure to make the Future/Promise split was actually noted as a flaw in the concurrent.futures design at the time, but wasn't seen as serious enough to prevent inclusion of the module. However, this draft PEP does a decent job of showing the kinds of operations that the current combined class design can make difficult) > - What are your thoughts on backward compatibility (current implementation > does not sacrifice any design points for it, but better compatibility can be > achieved)? Yes, I think better compatibility is needed. One possible option would be to take the path of defining a "concurrent.futures.abc" module that provides this higher level API, and largely leave the existing concrete classes alone (aside from adjusting them to fit the ABC). The main possible issue I see is with the result setting APIs on the existing Future objects. There are also a lot of staticmethod declarations that look like they should probably be classmethod declarations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Dec 21 18:59:08 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Dec 2013 09:59:08 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: I'm pretty worried about this proposal. It seems to encourage callback-style programming, whereas I would like to get away from it (that's the main thrust of asyncio/PEP 3156). The proposal reminds me of Twisted Deferred, which is too complicated for my taste. Note that asyncio has a composition operation of its own, gather(), but encourages using standard try/except/else/finally syntax for error handling, rather than adding Twisted-style errbacks. Nevertheless it is possible that I am misunderstanding the proposal (e.g. the distinction between Future and Promise wasn't clear from reading the draft PEP, nor how it would interact with asyncio). I recommend that the author work find a way to improve the English grammar of the proposal, e.g. by working with a native speaker. On Sat, Dec 21, 2013 at 6:12 AM, Nick Coghlan wrote: > On 21 December 2013 23:14, Sergii Mikhtoniuk wrote: >> Please have a look at this PEP draft, and reference implementation (as >> separate library). >> >> I?m very interested in: >> - How PEPable is this? > > This looks like a really interesting idea, and well worth pursuing as > a PEP for 3.5 (the failure to make the Future/Promise split was > actually noted as a flaw in the concurrent.futures design at the time, > but wasn't seen as serious enough to prevent inclusion of the module. > However, this draft PEP does a decent job of showing the kinds of > operations that the current combined class design can make difficult) > >> - What are your thoughts on backward compatibility (current implementation >> does not sacrifice any design points for it, but better compatibility can be >> achieved)? > > Yes, I think better compatibility is needed. One possible option would > be to take the path of defining a "concurrent.futures.abc" module that > provides this higher level API, and largely leave the existing > concrete classes alone (aside from adjusting them to fit the ABC). > > The main possible issue I see is with the result setting APIs on the > existing Future objects. > > There are also a lot of staticmethod declarations that look like they > should probably be classmethod declarations. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From mikhtonyuk at gmail.com Sat Dec 21 23:48:29 2013 From: mikhtonyuk at gmail.com (Sergii Mikhtoniuk) Date: Sun, 22 Dec 2013 00:48:29 +0200 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: Hello Guido, Nick, Thanks a lot for your valuable feedback. Indeed there is a lot of overlap between *asyncio* and *concurrent.futures*packages, so it would be very interesting to hear your overall thoughts on role/future of *concurrent* package. Do you consider it rudimentary and replaceable by *asyncio.Future* completely? I think the question is even not much about which is your preferred implementation, but rather do we want having futures as stand-alone package or not. Do you see all these implementations converging in future? To me *Future* is a very simple and self-contained primitive, independent of thread pools, processes, IO, and networking. One thing *concurrent.futures* package does a very good job at is defining an isolated namespace for futures, stressing out this clear boundary (let?s disregard here *ThreadPoolExecutor* and *ProcessPoolExecutor* classes which for some reason ended up in it too). So when I think of *schedulers* and *event loops* implementations I see them as ones that build on top of the *Future* primitive, not providing it as part of their implementation. What I think is important is that having unified *Future* class simplifies interoperability between different kinds of schedulers, not necessarily associated with *asyncio* event loops (process pools for example). *Futures* are definitely not the final solution for concurrency problem but rather a well-established utility for representing async operations. It is a job of higher layer systems to provide more convenient ways for dealing with asyncs (*yield from* coroutines, *async/await* rewrites etc.), so I would not say that futures encourage callback-style programming, it?s simply a lower-layer functionality. On the contrary, monadic methods for futures composition (e.g. *map()*, *all()*, *first()* etc.) ensure that no errors would be lost in the process, so I think they would complement* yield from* model quite nicely by hiding complexity of state maintenance from user and reducing the number of back-and-forth communications between event loop and coroutines. Besides *Futures*, reactive programming has more utilities to offer, such as *Observables* (representing asynchronous streams of values). It is also a very useful abstraction with a rich set of composition strategies (merging, concatenation, grouping), and may deserve its place in separate package. Hope to hear back from you to get better picture on overall design direction here before jumping to implementation details. Brief follow-up to your questions: - Idea behind the Future/Promise separation is to draw a clean line between client-facing and scheduler-side APIs respectively. Futures should not expose any completion methods, which clients should not call anyway. - Completely agree with you that Twisted-style callbacks are evil and it is better to have single code path for getting result or raising exception - Sorry for bad grammar in the proposal, it?s an early draft written in 3 AM, so I will definitely improve on it if we decide to move forward. Thanks, Sergii On Sat, Dec 21, 2013 at 7:59 PM, Guido van Rossum wrote: > I'm pretty worried about this proposal. It seems to encourage > callback-style programming, whereas I would like to get away from it > (that's the main thrust of asyncio/PEP 3156). The proposal reminds me > of Twisted Deferred, which is too complicated for my taste. > > Note that asyncio has a composition operation of its own, gather(), > but encourages using standard try/except/else/finally syntax for error > handling, rather than adding Twisted-style errbacks. > > Nevertheless it is possible that I am misunderstanding the proposal > (e.g. the distinction between Future and Promise wasn't clear from > reading the draft PEP, nor how it would interact with asyncio). I > recommend that the author work find a way to improve the English > grammar of the proposal, e.g. by working with a native speaker. > > On Sat, Dec 21, 2013 at 6:12 AM, Nick Coghlan wrote: > > On 21 December 2013 23:14, Sergii Mikhtoniuk > wrote: > >> Please have a look at this PEP draft, and reference implementation (as > >> separate library). > >> > >> I?m very interested in: > >> - How PEPable is this? > > > > This looks like a really interesting idea, and well worth pursuing as > > a PEP for 3.5 (the failure to make the Future/Promise split was > > actually noted as a flaw in the concurrent.futures design at the time, > > but wasn't seen as serious enough to prevent inclusion of the module. > > However, this draft PEP does a decent job of showing the kinds of > > operations that the current combined class design can make difficult) > > > >> - What are your thoughts on backward compatibility (current > implementation > >> does not sacrifice any design points for it, but better compatibility > can be > >> achieved)? > > > > Yes, I think better compatibility is needed. One possible option would > > be to take the path of defining a "concurrent.futures.abc" module that > > provides this higher level API, and largely leave the existing > > concrete classes alone (aside from adjusting them to fit the ABC). > > > > The main possible issue I see is with the result setting APIs on the > > existing Future objects. > > > > There are also a lot of staticmethod declarations that look like they > > should probably be classmethod declarations. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 22 01:26:51 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Dec 2013 16:26:51 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk wrote: > Hello Guido, Nick, > > Thanks a lot for your valuable feedback. You're welcome. > Indeed there is a lot of overlap between asyncio and concurrent.futures > packages, so it would be very interesting to hear your overall thoughts on > role/future of concurrent package. Do you consider it rudimentary and > replaceable by asyncio.Future completely? They don't really compare. concurrent.futures is about *threads*. asyncio.Future is about *avoiding* threads in favor of more lightweight "tasks" and "coroutines", which in turn are built on top of lower-level callbacks. (While I want to get away from callbacks as a programming paradigm, asyncio uses them at the lower levels both because they are a logical low-level building block and for interoperability with other frameworks like Tornado and Twisted.) > I think the question is even not much about which is your preferred > implementation, but rather do we want having futures as stand-alone package > or not. Do you see all these implementations converging in future? I see them as not even competing. They use different paradigms and apply to different use cases. > To me Future is a very simple and self-contained primitive, independent of > thread pools, processes, IO, and networking. (Agreed on the I/O and networking part only.) > One thing concurrent.futures > package does a very good job at is defining an isolated namespace for > futures, stressing out this clear boundary (let?s disregard here > ThreadPoolExecutor and ProcessPoolExecutor classes which for some reason > ended up in it too). Actually the executor API is an important and integral part of that package, and threads underlie everything you can do with its Futures. > So when I think of schedulers and event loops implementations I see them as > ones that build on top of the Future primitive, not providing it as part of > their implementation. What I think is important is that having unified > Future class simplifies interoperability between different kinds of > schedulers, not necessarily associated with asyncio event loops (process > pools for example). The interoperability is completely missing. I can't tell if you've used asyncio at all, but the important operation of *waiting* for a result is fundamentally different there than in concurrent.futures. In the latter, you just write "x = f.result()" and your thread blocks until the result is available. In asyncio, you have to write "x = yield from f.result()" which is a coroutine block that lets other tasks run in the same thread. ("yield from" in this case is how Python spells the operation that C# calls "await"). > Futures are definitely not the final solution for concurrency problem but > rather a well-established utility for representing async operations. It is a > job of higher layer systems to provide more convenient ways for dealing with > asyncs (yield from coroutines, async/await rewrites etc.), But unless you are proposing some kind of radical change to add compile-time type checking/inference to Python, the rewrite option is unavailable in Python. > so I would not > say that futures encourage callback-style programming, The concurrent.futures.Future class does not. But unless I misread your proposal, your extensions do. > it?s simply a > lower-layer functionality. On the contrary, monadic (Say that word one more time and everyone tunes out. :-) > methods for futures > composition (e.g. map(), all(), first() etc.) ensure that no errors would be > lost in the process, so I think they would complement yield from model quite > nicely by hiding complexity of state maintenance from user and reducing the > number of back-and-forth communications between event loop and coroutines. I'm not sure I follow. Again, I'm not sure if you've actually written any code using asyncio. TBH I've written a fair number of example programs for asyncio and I've very rarely felt the need for these composition functions. The main composition primitive I tend to use is "yield from". > Besides Futures, reactive programming What *is* reactive programming? If you're talking about http://en.wikipedia.org/wiki/Reactive_programming, I'm not sure that it maps well to Python. > has more utilities to offer, such as > Observables (representing asynchronous streams of values). It is also a very > useful abstraction with a rich set of composition strategies (merging, > concatenation, grouping), and may deserve its place in separate package. It all sounds very abstract and academic. :-) > Hope to hear back from you to get better picture on overall design direction > here before jumping to implementation details. > > Brief follow-up to your questions: > - Idea behind the Future/Promise separation is to draw a clean line between > client-facing and scheduler-side APIs respectively. Futures should not > expose any completion methods, which clients should not call anyway. Given that Future and Promise are often used synonymously, using them to make this distinction sounds confusing. I agree that Futures have two different APIs, one for the consumer and another for the producer. But I'm not sure that it's necessary to separate them more strictly -- convention seems good enough to me here. (It's the same with many communication primitives, like queues and even threads.) (The one thing that trips people up frequently is that, while set_result() and set_exception() are producer APIs, cancel() is a consumer API, and the cancellation signal travels from the consumer to the producer.) > - Completely agree with you that Twisted-style callbacks are evil and it is > better to have single code path for getting result or raising exception > - Sorry for bad grammar in the proposal, it?s an early draft written in 3 > AM, so I will definitely improve on it if we decide to move forward. No problem! > Thanks, > Sergii -- --Guido van Rossum (python.org/~guido) From ben at bendarnell.com Sun Dec 22 02:37:26 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 21 Dec 2013 20:37:26 -0500 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum wrote: > On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk > wrote: > > Indeed there is a lot of overlap between asyncio and concurrent.futures > > packages, so it would be very interesting to hear your overall thoughts > on > > role/future of concurrent package. Do you consider it rudimentary and > > replaceable by asyncio.Future completely? > > They don't really compare. concurrent.futures is about *threads*. > asyncio.Future is about *avoiding* threads in favor of more > lightweight "tasks" and "coroutines", which in turn are built on top > of lower-level callbacks. > concurrent.futures.ThreadPoolExecutor is about threads; the Future class itself is broader. When I integrated Futures into Tornado I used concurrent.futures.Future directly (when available). asyncio.Future is just an optimized version of c.f.Future (the optimization comes from assuming single-threaded usage). There should at least be a common ABC between them. > > (While I want to get away from callbacks as a programming paradigm, > asyncio uses them at the lower levels both because they are a logical > low-level building block and for interoperability with other > frameworks like Tornado and Twisted.) > > > I think the question is even not much about which is your preferred > > implementation, but rather do we want having futures as stand-alone > package > > or not. Do you see all these implementations converging in future? > > I see them as not even competing. They use different paradigms and > apply to different use cases. > > > To me Future is a very simple and self-contained primitive, independent > of > > thread pools, processes, IO, and networking. > > (Agreed on the I/O and networking part only.) > > > One thing concurrent.futures > > package does a very good job at is defining an isolated namespace for > > futures, stressing out this clear boundary (let?s disregard here > > ThreadPoolExecutor and ProcessPoolExecutor classes which for some reason > > ended up in it too). > > Actually the executor API is an important and integral part of that > package, and threads underlie everything you can do with its Futures. > > > So when I think of schedulers and event loops implementations I see them > as > > ones that build on top of the Future primitive, not providing it as part > of > > their implementation. What I think is important is that having unified > > Future class simplifies interoperability between different kinds of > > schedulers, not necessarily associated with asyncio event loops (process > > pools for example). > > The interoperability is completely missing. I can't tell if you've > used asyncio at all, but the important operation of *waiting* for a > result is fundamentally different there than in concurrent.futures. In > the latter, you just write "x = f.result()" and your thread blocks > until the result is available. In asyncio, you have to write "x = > yield from f.result()" which is a coroutine block that lets other > tasks run in the same thread. ("yield from" in this case is how Python > spells the operation that C# calls "await"). > The way I see it, the fundamental operation on Futures is add_done_callback. We then have various higher-level operations that let us get away from using callbacks directly. One of these happens to be a method on Future: the blocking mode of Future.result(). Another is implemented in asyncio and Tornado, in the ability to "yield" a Future. The blocking mode of result() is just a convenience; if asyncio-style futures had existed first then we could instead have a function like "x = concurrent.futures.wait_for(f)". In fact, you could write this wait_for function today in a way that works for both concurrent and asyncio futures. This is already interoperable: Tornado's Resolver interface ( https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184) returns a Future, which may be generated by a ThreadPoolExecutor or an asynchronous wrapper around pycares or twisted. In the other direction I've worked on hybrid apps that have one Tornado thread alongside a bunch of Django threads; in these apps it would work to have a Django thread block on f.result() for a Future returned by Tornado's AsyncHTTPClient. -Ben > > Futures are definitely not the final solution for concurrency problem but > > rather a well-established utility for representing async operations. It > is a > > job of higher layer systems to provide more convenient ways for dealing > with > > asyncs (yield from coroutines, async/await rewrites etc.), > > But unless you are proposing some kind of radical change to add > compile-time type checking/inference to Python, the rewrite option is > unavailable in Python. > > > so I would not > > say that futures encourage callback-style programming, > > The concurrent.futures.Future class does not. But unless I misread > your proposal, your extensions do. > > > it?s simply a > > lower-layer functionality. On the contrary, monadic > > (Say that word one more time and everyone tunes out. :-) > > > methods for futures > > composition (e.g. map(), all(), first() etc.) ensure that no errors > would be > > lost in the process, so I think they would complement yield from model > quite > > nicely by hiding complexity of state maintenance from user and reducing > the > > number of back-and-forth communications between event loop and > coroutines. > > I'm not sure I follow. Again, I'm not sure if you've actually written > any code using asyncio. > > TBH I've written a fair number of example programs for asyncio and > I've very rarely felt the need for these composition functions. The > main composition primitive I tend to use is "yield from". > > > Besides Futures, reactive programming > > What *is* reactive programming? If you're talking about > http://en.wikipedia.org/wiki/Reactive_programming, > I'm not sure that it maps well to Python. > > > has more utilities to offer, such as > > Observables (representing asynchronous streams of values). It is also a > very > > useful abstraction with a rich set of composition strategies (merging, > > concatenation, grouping), and may deserve its place in separate package. > > It all sounds very abstract and academic. :-) > > > Hope to hear back from you to get better picture on overall design > direction > > here before jumping to implementation details. > > > > Brief follow-up to your questions: > > - Idea behind the Future/Promise separation is to draw a clean line > between > > client-facing and scheduler-side APIs respectively. Futures should not > > expose any completion methods, which clients should not call anyway. > > Given that Future and Promise are often used synonymously, using them > to make this distinction sounds confusing. I agree that Futures have > two different APIs, one for the consumer and another for the producer. > But I'm not sure that it's necessary to separate them more strictly -- > convention seems good enough to me here. (It's the same with many > communication primitives, like queues and even threads.) > > (The one thing that trips people up frequently is that, while > set_result() and set_exception() are producer APIs, cancel() is a > consumer API, and the cancellation signal travels from the consumer to > the producer.) > > > - Completely agree with you that Twisted-style callbacks are evil and > it is > > better to have single code path for getting result or raising exception > > - Sorry for bad grammar in the proposal, it?s an early draft written in > 3 > > AM, so I will definitely improve on it if we decide to move forward. > > No problem! > > > Thanks, > > Sergii > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 22 04:45:06 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Dec 2013 19:45:06 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: There's still the issue that in the threading version, you wait for a Future by blocking the current thread, while in the asyncio version, you must use "yield from" to block. For interoperability you would have to refrain from *any* blocking operations (including "yield from") so you would only be able to use callbacks. But whether you had to write "x = f.result()" or "x = concurrent.futures.wait_for(f)", either way you'd implicitly be blocking the current thread. Yes, a clever scheduler could run other callbacks while blocking, but that's not a complete solution, because another callback might do a similar blocking operation, and whatever that waited for could hold up the earlier blocking operation, causing an ever-deeper recursion and of event loop invocations that might never complete. (I've had to debug this in production code.) To cut through that you'd have to have some kind of stack-swapping coroutine implementation like gevent, or a syntactic preprocessor that inserts yield or yield-from operations (I've heard from people who do this), or you'd need a clairvoyant scheduler that would know which callbacks won't block. I like the C# solution, but it depends on static typing so a compiler can know when to emit the coroutine interactions. That wouldn't work in Python, unless you made the compiler recognizing the wait_for() operation by name, which feels unsavory (although we do it for super() :-). I guess for extreme interop, callbacks that never block is your only option anyway, but I'd be sad if we had to to recommend this as the preferred paradigm, or claim that it is all you need. --Guido (if I don't respond to this thread for the next two weeks, it's because I'm on vacation :-) On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell wrote: > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum wrote: >> >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk >> wrote: >> > Indeed there is a lot of overlap between asyncio and concurrent.futures >> > packages, so it would be very interesting to hear your overall thoughts >> > on >> > role/future of concurrent package. Do you consider it rudimentary and >> > replaceable by asyncio.Future completely? >> >> They don't really compare. concurrent.futures is about *threads*. >> asyncio.Future is about *avoiding* threads in favor of more >> lightweight "tasks" and "coroutines", which in turn are built on top >> of lower-level callbacks. > > > concurrent.futures.ThreadPoolExecutor is about threads; the Future class > itself is broader. When I integrated Futures into Tornado I used > concurrent.futures.Future directly (when available). asyncio.Future is just > an optimized version of c.f.Future (the optimization comes from assuming > single-threaded usage). There should at least be a common ABC between them. > >> >> >> (While I want to get away from callbacks as a programming paradigm, >> asyncio uses them at the lower levels both because they are a logical >> low-level building block and for interoperability with other >> frameworks like Tornado and Twisted.) >> >> > I think the question is even not much about which is your preferred >> > implementation, but rather do we want having futures as stand-alone >> > package >> > or not. Do you see all these implementations converging in future? >> >> I see them as not even competing. They use different paradigms and >> apply to different use cases. >> >> > To me Future is a very simple and self-contained primitive, independent >> > of >> > thread pools, processes, IO, and networking. >> >> (Agreed on the I/O and networking part only.) >> >> > One thing concurrent.futures >> > package does a very good job at is defining an isolated namespace for >> > futures, stressing out this clear boundary (let?s disregard here >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some reason >> > ended up in it too). >> >> Actually the executor API is an important and integral part of that >> package, and threads underlie everything you can do with its Futures. >> >> > So when I think of schedulers and event loops implementations I see them >> > as >> > ones that build on top of the Future primitive, not providing it as part >> > of >> > their implementation. What I think is important is that having unified >> > Future class simplifies interoperability between different kinds of >> > schedulers, not necessarily associated with asyncio event loops (process >> > pools for example). >> >> The interoperability is completely missing. I can't tell if you've >> used asyncio at all, but the important operation of *waiting* for a >> result is fundamentally different there than in concurrent.futures. In >> the latter, you just write "x = f.result()" and your thread blocks >> until the result is available. In asyncio, you have to write "x = >> yield from f.result()" which is a coroutine block that lets other >> tasks run in the same thread. ("yield from" in this case is how Python >> spells the operation that C# calls "await"). > > > The way I see it, the fundamental operation on Futures is add_done_callback. > We then have various higher-level operations that let us get away from using > callbacks directly. One of these happens to be a method on Future: the > blocking mode of Future.result(). Another is implemented in asyncio and > Tornado, in the ability to "yield" a Future. The blocking mode of result() > is just a convenience; if asyncio-style futures had existed first then we > could instead have a function like "x = concurrent.futures.wait_for(f)". In > fact, you could write this wait_for function today in a way that works for > both concurrent and asyncio futures. > > This is already interoperable: Tornado's Resolver interface > (https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184) > returns a Future, which may be generated by a ThreadPoolExecutor or an > asynchronous wrapper around pycares or twisted. In the other direction I've > worked on hybrid apps that have one Tornado thread alongside a bunch of > Django threads; in these apps it would work to have a Django thread block on > f.result() for a Future returned by Tornado's AsyncHTTPClient. > > -Ben > >> >> > Futures are definitely not the final solution for concurrency problem >> > but >> > rather a well-established utility for representing async operations. It >> > is a >> > job of higher layer systems to provide more convenient ways for dealing >> > with >> > asyncs (yield from coroutines, async/await rewrites etc.), >> >> But unless you are proposing some kind of radical change to add >> compile-time type checking/inference to Python, the rewrite option is >> unavailable in Python. >> >> > so I would not >> > say that futures encourage callback-style programming, >> >> The concurrent.futures.Future class does not. But unless I misread >> your proposal, your extensions do. >> >> > it?s simply a >> > lower-layer functionality. On the contrary, monadic >> >> (Say that word one more time and everyone tunes out. :-) >> >> > methods for futures >> > composition (e.g. map(), all(), first() etc.) ensure that no errors >> > would be >> > lost in the process, so I think they would complement yield from model >> > quite >> > nicely by hiding complexity of state maintenance from user and reducing >> > the >> > number of back-and-forth communications between event loop and >> > coroutines. >> >> I'm not sure I follow. Again, I'm not sure if you've actually written >> any code using asyncio. >> >> TBH I've written a fair number of example programs for asyncio and >> I've very rarely felt the need for these composition functions. The >> main composition primitive I tend to use is "yield from". >> >> > Besides Futures, reactive programming >> >> What *is* reactive programming? If you're talking about >> http://en.wikipedia.org/wiki/Reactive_programming, >> I'm not sure that it maps well to Python. >> >> > has more utilities to offer, such as >> > Observables (representing asynchronous streams of values). It is also a >> > very >> > useful abstraction with a rich set of composition strategies (merging, >> > concatenation, grouping), and may deserve its place in separate package. >> >> It all sounds very abstract and academic. :-) >> >> > Hope to hear back from you to get better picture on overall design >> > direction >> > here before jumping to implementation details. >> > >> > Brief follow-up to your questions: >> > - Idea behind the Future/Promise separation is to draw a clean line >> > between >> > client-facing and scheduler-side APIs respectively. Futures should not >> > expose any completion methods, which clients should not call anyway. >> >> Given that Future and Promise are often used synonymously, using them >> to make this distinction sounds confusing. I agree that Futures have >> two different APIs, one for the consumer and another for the producer. >> But I'm not sure that it's necessary to separate them more strictly -- >> convention seems good enough to me here. (It's the same with many >> communication primitives, like queues and even threads.) >> >> (The one thing that trips people up frequently is that, while >> set_result() and set_exception() are producer APIs, cancel() is a >> consumer API, and the cancellation signal travels from the consumer to >> the producer.) >> >> > - Completely agree with you that Twisted-style callbacks are evil and >> > it is >> > better to have single code path for getting result or raising exception >> > - Sorry for bad grammar in the proposal, it?s an early draft written in >> > 3 >> > AM, so I will definitely improve on it if we decide to move forward. >> >> No problem! >> >> > Thanks, >> > Sergii >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) From rosuav at gmail.com Sun Dec 22 04:53:58 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 Dec 2013 14:53:58 +1100 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sun, Dec 22, 2013 at 2:45 PM, Guido van Rossum wrote: > To cut through that you'd have to have > some kind of stack-swapping coroutine implementation like gevent... Forgive the stupid question, but how is stack-swapping during blocking calls materially different from threads? ChrisA From guido at python.org Sun Dec 22 05:01:26 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Dec 2013 20:01:26 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 7:53 PM, Chris Angelico wrote: > On Sun, Dec 22, 2013 at 2:45 PM, Guido van Rossum wrote: >> To cut through that you'd have to have >> some kind of stack-swapping coroutine implementation like gevent... > > Forgive the stupid question, but how is stack-swapping during blocking > calls materially different from threads? It's also known as "green threads". The gevent folks and the Stackless folks (and a few others) do this and claim it is vastly superior to OS threads. I believe the main difference is that an OS thread takes up a relatively large amount of resources in the kernel as well as in user space (for the stack) while a green thread takes up a comparatively much smaller amount of space, all in user space -- with the result that you can have many more green threads than you could have OS threads, and switching between them will be much faster. The price you pay is that the kernel doesn't know what you're doing and you have to intercept all system-call-level I/O to make it non-blocking -- if you accidentally make a blocking syscall, no other green thread will run. This may sound like a pure implementation-level distinction, but implementation is what makes things practical (otherwise we'd all be using Turing machines or lambda calculus :-). -- --Guido van Rossum (python.org/~guido) From rosuav at gmail.com Sun Dec 22 05:25:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 Dec 2013 15:25:07 +1100 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sun, Dec 22, 2013 at 3:01 PM, Guido van Rossum wrote: > On Sat, Dec 21, 2013 at 7:53 PM, Chris Angelico wrote: >> On Sun, Dec 22, 2013 at 2:45 PM, Guido van Rossum wrote: >>> To cut through that you'd have to have >>> some kind of stack-swapping coroutine implementation like gevent... >> >> Forgive the stupid question, but how is stack-swapping during blocking >> calls materially different from threads? > > It's also known as "green threads". The gevent folks and the Stackless > folks (and a few others) do this and claim it is vastly superior to OS > threads. Gotcha. I grew up on OS/2 where the threading was lean and mean, so I just used it. I'd be mildly curious to know how different implementations of threads compare, and how many of them actually warrant a "lighter-weight thread" feature like this, but for something that aims to be cross-platform, I can see the value in doing it. ChrisA From abarnert at yahoo.com Sun Dec 22 06:41:07 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 21 Dec 2013 21:41:07 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: <45EAF7BB-9895-472B-8E6F-5437DB05A474@yahoo.com> A few things I'd like to see. Your Future uses a condition variable. That could be a big hit for single threaded uses, and since one of your goals is to make asyncio use the same futures as threaded executors, that might not be acceptable. How hard would it be to allow passing a class (or other factory callable) in place of the default condition when constructing a Promise, or just a "threaded=True" flag. (I realize this is more complicated than it sounds--an event loop can always push callbacks onto a thread pool, or just use a thread pool to implement something that's hard to do in a single thread, like DNS lookup--so you probably also need a way to upgrade a single threaded promise to a thread safe one.) There's a lot of rationale about why separate futures and promises are important, but no rationale for the specific design. If you explained where you deviated from twisted Deferred, JS Promises/A, etc., and why, it would be a lot easier to evaluate your design. It's not documented how chaining multiple callbacks in sequence works. Does the second callback get the return value of the first one, or the original one? Can you return a new future (possibly with callbacks already attached, possibly already completed)? Sent from a random iPhone On Dec 21, 2013, at 5:14, Sergii Mikhtoniuk wrote: > Hi all, > > I would very much appreciate your opinion on my proposal for improvement of concurrent.futures package. > > Comparing to other languages such as Scala and C#, Python?s futures significantly fall behind in functionality especially in ability to chain computations and compose different futures without blocking and waiting for result. New packages continue to emerge (asyncio) which provide their own futures implementation, making composition even more difficult. > > Proposed improvement implements Scala-like Future as a monadic construct. It allows performing multiple kinds of operations on Future?s result without blocking, enabling reactive programming in Python. It implements common pattern separating Future and Promise interface, making it very easy for 3rd party systems to use futures in their API. > > Please have a look at this PEP draft, and reference implementation (as separate library). > > I?m very interested in: > - How PEPable is this? > - What are your thoughts on backward compatibility (current implementation does not sacrifice any design points for it, but better compatibility can be achieved)? > - Thoughts on Future-based APIs in other packages? > > Thanks, > Sergii > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Dec 22 06:50:36 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 21 Dec 2013 21:50:36 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: <39F3FE04-2E72-43DC-83E4-D67805AE442A@yahoo.com> Even OS/2 can't do thousands of threads, so if you want to write a server with one thread per client (or two) you'd still need green threads. Meanwhile, Windows, Linux, and OS X all have pretty fast thread startup and decent schedulers, but the need for a static-sized stack still means you can't do thousands even on today's computers--especially in 32 bit land (which is depressingly still common on Windows). Sent from a random iPhone On Dec 21, 2013, at 20:25, Chris Angelico wrote: > On Sun, Dec 22, 2013 at 3:01 PM, Guido van Rossum wrote: >> On Sat, Dec 21, 2013 at 7:53 PM, Chris Angelico wrote: >>> On Sun, Dec 22, 2013 at 2:45 PM, Guido van Rossum wrote: >>>> To cut through that you'd have to have >>>> some kind of stack-swapping coroutine implementation like gevent... >>> >>> Forgive the stupid question, but how is stack-swapping during blocking >>> calls materially different from threads? >> >> It's also known as "green threads". The gevent folks and the Stackless >> folks (and a few others) do this and claim it is vastly superior to OS >> threads. > > Gotcha. I grew up on OS/2 where the threading was lean and mean, so I > just used it. I'd be mildly curious to know how different > implementations of threads compare, and how many of them actually > warrant a "lighter-weight thread" feature like this, but for something > that aims to be cross-platform, I can see the value in doing it. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From solipsis at pitrou.net Sun Dec 22 15:54:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Dec 2013 15:54:15 +0100 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming References: Message-ID: <20131222155415.5b8d417e@fsol> On Sat, 21 Dec 2013 20:01:26 -0800 Guido van Rossum wrote: > On Sat, Dec 21, 2013 at 7:53 PM, Chris Angelico wrote: > > On Sun, Dec 22, 2013 at 2:45 PM, Guido van Rossum wrote: > >> To cut through that you'd have to have > >> some kind of stack-swapping coroutine implementation like gevent... > > > > Forgive the stupid question, but how is stack-swapping during blocking > > calls materially different from threads? > > It's also known as "green threads". The gevent folks and the Stackless > folks (and a few others) do this and claim it is vastly superior to OS > threads. I believe the main difference is that an OS thread takes up a > relatively large amount of resources in the kernel as well as in user > space (for the stack) while a green thread takes up a comparatively > much smaller amount of space, all in user space -- with the result > that you can have many more green threads than you could have OS > threads, and switching between them will be much faster. Also, in Python 2.x, it avoids the contention cost of the GIL with many OS threads (cf. the Dave Beazley benchmark :-)). Regards Antoine. From ben at bendarnell.com Sun Dec 22 23:29:50 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sun, 22 Dec 2013 17:29:50 -0500 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 10:45 PM, Guido van Rossum wrote: > There's still the issue that in the threading version, you wait for a > Future by blocking the current thread, while in the asyncio version, > you must use "yield from" to block. For interoperability you would > have to refrain from *any* blocking operations (including "yield > from") so you would only be able to use callbacks. But whether you had > to write "x = f.result()" or "x = concurrent.futures.wait_for(f)", > either way you'd implicitly be blocking the current thread. > Threaded *consumers* of Futures wait for them by blocking, while asynchronous consumers wait for them by yielding. It doesn't matter whether the *producer* of the Future is threaded or asynchronous (except that if you know you won't be using threads you can use a faster thread-unsafe Future implementation). -Ben > > Yes, a clever scheduler could run other callbacks while blocking, but > that's not a complete solution, because another callback might do a > similar blocking operation, and whatever that waited for could hold up > the earlier blocking operation, causing an ever-deeper recursion and > of event loop invocations that might never complete. (I've had to > debug this in production code.) To cut through that you'd have to have > some kind of stack-swapping coroutine implementation like gevent, or a > syntactic preprocessor that inserts yield or yield-from operations > (I've heard from people who do this), or you'd need a clairvoyant > scheduler that would know which callbacks won't block. > > I like the C# solution, but it depends on static typing so a compiler > can know when to emit the coroutine interactions. That wouldn't work > in Python, unless you made the compiler recognizing the wait_for() > operation by name, which feels unsavory (although we do it for super() > :-). > > I guess for extreme interop, callbacks that never block is your only > option anyway, but I'd be sad if we had to to recommend this as the > preferred paradigm, or claim that it is all you need. > > --Guido (if I don't respond to this thread for the next two weeks, > it's because I'm on vacation :-) > > > On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell wrote: > > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum > wrote: > >> > >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk < > mikhtonyuk at gmail.com> > >> wrote: > >> > Indeed there is a lot of overlap between asyncio and > concurrent.futures > >> > packages, so it would be very interesting to hear your overall > thoughts > >> > on > >> > role/future of concurrent package. Do you consider it rudimentary and > >> > replaceable by asyncio.Future completely? > >> > >> They don't really compare. concurrent.futures is about *threads*. > >> asyncio.Future is about *avoiding* threads in favor of more > >> lightweight "tasks" and "coroutines", which in turn are built on top > >> of lower-level callbacks. > > > > > > concurrent.futures.ThreadPoolExecutor is about threads; the Future class > > itself is broader. When I integrated Futures into Tornado I used > > concurrent.futures.Future directly (when available). asyncio.Future is > just > > an optimized version of c.f.Future (the optimization comes from assuming > > single-threaded usage). There should at least be a common ABC between > them. > > > >> > >> > >> (While I want to get away from callbacks as a programming paradigm, > >> asyncio uses them at the lower levels both because they are a logical > >> low-level building block and for interoperability with other > >> frameworks like Tornado and Twisted.) > >> > >> > I think the question is even not much about which is your preferred > >> > implementation, but rather do we want having futures as stand-alone > >> > package > >> > or not. Do you see all these implementations converging in future? > >> > >> I see them as not even competing. They use different paradigms and > >> apply to different use cases. > >> > >> > To me Future is a very simple and self-contained primitive, > independent > >> > of > >> > thread pools, processes, IO, and networking. > >> > >> (Agreed on the I/O and networking part only.) > >> > >> > One thing concurrent.futures > >> > package does a very good job at is defining an isolated namespace for > >> > futures, stressing out this clear boundary (let?s disregard here > >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some > reason > >> > ended up in it too). > >> > >> Actually the executor API is an important and integral part of that > >> package, and threads underlie everything you can do with its Futures. > >> > >> > So when I think of schedulers and event loops implementations I see > them > >> > as > >> > ones that build on top of the Future primitive, not providing it as > part > >> > of > >> > their implementation. What I think is important is that having unified > >> > Future class simplifies interoperability between different kinds of > >> > schedulers, not necessarily associated with asyncio event loops > (process > >> > pools for example). > >> > >> The interoperability is completely missing. I can't tell if you've > >> used asyncio at all, but the important operation of *waiting* for a > >> result is fundamentally different there than in concurrent.futures. In > >> the latter, you just write "x = f.result()" and your thread blocks > >> until the result is available. In asyncio, you have to write "x = > >> yield from f.result()" which is a coroutine block that lets other > >> tasks run in the same thread. ("yield from" in this case is how Python > >> spells the operation that C# calls "await"). > > > > > > The way I see it, the fundamental operation on Futures is > add_done_callback. > > We then have various higher-level operations that let us get away from > using > > callbacks directly. One of these happens to be a method on Future: the > > blocking mode of Future.result(). Another is implemented in asyncio and > > Tornado, in the ability to "yield" a Future. The blocking mode of > result() > > is just a convenience; if asyncio-style futures had existed first then we > > could instead have a function like "x = concurrent.futures.wait_for(f)". > In > > fact, you could write this wait_for function today in a way that works > for > > both concurrent and asyncio futures. > > > > This is already interoperable: Tornado's Resolver interface > > (https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184 > ) > > returns a Future, which may be generated by a ThreadPoolExecutor or an > > asynchronous wrapper around pycares or twisted. In the other direction > I've > > worked on hybrid apps that have one Tornado thread alongside a bunch of > > Django threads; in these apps it would work to have a Django thread > block on > > f.result() for a Future returned by Tornado's AsyncHTTPClient. > > > > -Ben > > > >> > >> > Futures are definitely not the final solution for concurrency problem > >> > but > >> > rather a well-established utility for representing async operations. > It > >> > is a > >> > job of higher layer systems to provide more convenient ways for > dealing > >> > with > >> > asyncs (yield from coroutines, async/await rewrites etc.), > >> > >> But unless you are proposing some kind of radical change to add > >> compile-time type checking/inference to Python, the rewrite option is > >> unavailable in Python. > >> > >> > so I would not > >> > say that futures encourage callback-style programming, > >> > >> The concurrent.futures.Future class does not. But unless I misread > >> your proposal, your extensions do. > >> > >> > it?s simply a > >> > lower-layer functionality. On the contrary, monadic > >> > >> (Say that word one more time and everyone tunes out. :-) > >> > >> > methods for futures > >> > composition (e.g. map(), all(), first() etc.) ensure that no errors > >> > would be > >> > lost in the process, so I think they would complement yield from model > >> > quite > >> > nicely by hiding complexity of state maintenance from user and > reducing > >> > the > >> > number of back-and-forth communications between event loop and > >> > coroutines. > >> > >> I'm not sure I follow. Again, I'm not sure if you've actually written > >> any code using asyncio. > >> > >> TBH I've written a fair number of example programs for asyncio and > >> I've very rarely felt the need for these composition functions. The > >> main composition primitive I tend to use is "yield from". > >> > >> > Besides Futures, reactive programming > >> > >> What *is* reactive programming? If you're talking about > >> http://en.wikipedia.org/wiki/Reactive_programming, > >> I'm not sure that it maps well to Python. > >> > >> > has more utilities to offer, such as > >> > Observables (representing asynchronous streams of values). It is also > a > >> > very > >> > useful abstraction with a rich set of composition strategies (merging, > >> > concatenation, grouping), and may deserve its place in separate > package. > >> > >> It all sounds very abstract and academic. :-) > >> > >> > Hope to hear back from you to get better picture on overall design > >> > direction > >> > here before jumping to implementation details. > >> > > >> > Brief follow-up to your questions: > >> > - Idea behind the Future/Promise separation is to draw a clean line > >> > between > >> > client-facing and scheduler-side APIs respectively. Futures should not > >> > expose any completion methods, which clients should not call anyway. > >> > >> Given that Future and Promise are often used synonymously, using them > >> to make this distinction sounds confusing. I agree that Futures have > >> two different APIs, one for the consumer and another for the producer. > >> But I'm not sure that it's necessary to separate them more strictly -- > >> convention seems good enough to me here. (It's the same with many > >> communication primitives, like queues and even threads.) > >> > >> (The one thing that trips people up frequently is that, while > >> set_result() and set_exception() are producer APIs, cancel() is a > >> consumer API, and the cancellation signal travels from the consumer to > >> the producer.) > >> > >> > - Completely agree with you that Twisted-style callbacks are evil and > >> > it is > >> > better to have single code path for getting result or raising > exception > >> > - Sorry for bad grammar in the proposal, it?s an early draft written > in > >> > 3 > >> > AM, so I will definitely improve on it if we decide to move forward. > >> > >> No problem! > >> > >> > Thanks, > >> > Sergii > >> > >> -- > >> --Guido van Rossum (python.org/~guido) > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Dec 22 23:42:15 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Dec 2013 14:42:15 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: Aha. That is clever. I will have to look into the details more, but the idea is promising. Sorry I didn't see tty his before. On Dec 22, 2013 12:30 PM, "Ben Darnell" wrote: > On Sat, Dec 21, 2013 at 10:45 PM, Guido van Rossum wrote: > >> There's still the issue that in the threading version, you wait for a >> Future by blocking the current thread, while in the asyncio version, >> you must use "yield from" to block. For interoperability you would >> have to refrain from *any* blocking operations (including "yield >> from") so you would only be able to use callbacks. But whether you had >> to write "x = f.result()" or "x = concurrent.futures.wait_for(f)", >> either way you'd implicitly be blocking the current thread. >> > > Threaded *consumers* of Futures wait for them by blocking, while > asynchronous consumers wait for them by yielding. It doesn't matter > whether the *producer* of the Future is threaded or asynchronous (except > that if you know you won't be using threads you can use a faster > thread-unsafe Future implementation). > > -Ben > > >> >> Yes, a clever scheduler could run other callbacks while blocking, but >> that's not a complete solution, because another callback might do a >> similar blocking operation, and whatever that waited for could hold up >> the earlier blocking operation, causing an ever-deeper recursion and >> of event loop invocations that might never complete. (I've had to >> debug this in production code.) To cut through that you'd have to have >> some kind of stack-swapping coroutine implementation like gevent, or a >> syntactic preprocessor that inserts yield or yield-from operations >> (I've heard from people who do this), or you'd need a clairvoyant >> scheduler that would know which callbacks won't block. >> >> I like the C# solution, but it depends on static typing so a compiler >> can know when to emit the coroutine interactions. That wouldn't work >> in Python, unless you made the compiler recognizing the wait_for() >> operation by name, which feels unsavory (although we do it for super() >> :-). >> >> I guess for extreme interop, callbacks that never block is your only >> option anyway, but I'd be sad if we had to to recommend this as the >> preferred paradigm, or claim that it is all you need. >> >> --Guido (if I don't respond to this thread for the next two weeks, >> it's because I'm on vacation :-) >> >> >> On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell wrote: >> > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum >> wrote: >> >> >> >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk < >> mikhtonyuk at gmail.com> >> >> wrote: >> >> > Indeed there is a lot of overlap between asyncio and >> concurrent.futures >> >> > packages, so it would be very interesting to hear your overall >> thoughts >> >> > on >> >> > role/future of concurrent package. Do you consider it rudimentary and >> >> > replaceable by asyncio.Future completely? >> >> >> >> They don't really compare. concurrent.futures is about *threads*. >> >> asyncio.Future is about *avoiding* threads in favor of more >> >> lightweight "tasks" and "coroutines", which in turn are built on top >> >> of lower-level callbacks. >> > >> > >> > concurrent.futures.ThreadPoolExecutor is about threads; the Future class >> > itself is broader. When I integrated Futures into Tornado I used >> > concurrent.futures.Future directly (when available). asyncio.Future is >> just >> > an optimized version of c.f.Future (the optimization comes from assuming >> > single-threaded usage). There should at least be a common ABC between >> them. >> > >> >> >> >> >> >> (While I want to get away from callbacks as a programming paradigm, >> >> asyncio uses them at the lower levels both because they are a logical >> >> low-level building block and for interoperability with other >> >> frameworks like Tornado and Twisted.) >> >> >> >> > I think the question is even not much about which is your preferred >> >> > implementation, but rather do we want having futures as stand-alone >> >> > package >> >> > or not. Do you see all these implementations converging in future? >> >> >> >> I see them as not even competing. They use different paradigms and >> >> apply to different use cases. >> >> >> >> > To me Future is a very simple and self-contained primitive, >> independent >> >> > of >> >> > thread pools, processes, IO, and networking. >> >> >> >> (Agreed on the I/O and networking part only.) >> >> >> >> > One thing concurrent.futures >> >> > package does a very good job at is defining an isolated namespace for >> >> > futures, stressing out this clear boundary (let?s disregard here >> >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some >> reason >> >> > ended up in it too). >> >> >> >> Actually the executor API is an important and integral part of that >> >> package, and threads underlie everything you can do with its Futures. >> >> >> >> > So when I think of schedulers and event loops implementations I see >> them >> >> > as >> >> > ones that build on top of the Future primitive, not providing it as >> part >> >> > of >> >> > their implementation. What I think is important is that having >> unified >> >> > Future class simplifies interoperability between different kinds of >> >> > schedulers, not necessarily associated with asyncio event loops >> (process >> >> > pools for example). >> >> >> >> The interoperability is completely missing. I can't tell if you've >> >> used asyncio at all, but the important operation of *waiting* for a >> >> result is fundamentally different there than in concurrent.futures. In >> >> the latter, you just write "x = f.result()" and your thread blocks >> >> until the result is available. In asyncio, you have to write "x = >> >> yield from f.result()" which is a coroutine block that lets other >> >> tasks run in the same thread. ("yield from" in this case is how Python >> >> spells the operation that C# calls "await"). >> > >> > >> > The way I see it, the fundamental operation on Futures is >> add_done_callback. >> > We then have various higher-level operations that let us get away from >> using >> > callbacks directly. One of these happens to be a method on Future: the >> > blocking mode of Future.result(). Another is implemented in asyncio and >> > Tornado, in the ability to "yield" a Future. The blocking mode of >> result() >> > is just a convenience; if asyncio-style futures had existed first then >> we >> > could instead have a function like "x = >> concurrent.futures.wait_for(f)". In >> > fact, you could write this wait_for function today in a way that works >> for >> > both concurrent and asyncio futures. >> > >> > This is already interoperable: Tornado's Resolver interface >> > ( >> https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184) >> > returns a Future, which may be generated by a ThreadPoolExecutor or an >> > asynchronous wrapper around pycares or twisted. In the other direction >> I've >> > worked on hybrid apps that have one Tornado thread alongside a bunch of >> > Django threads; in these apps it would work to have a Django thread >> block on >> > f.result() for a Future returned by Tornado's AsyncHTTPClient. >> > >> > -Ben >> > >> >> >> >> > Futures are definitely not the final solution for concurrency problem >> >> > but >> >> > rather a well-established utility for representing async operations. >> It >> >> > is a >> >> > job of higher layer systems to provide more convenient ways for >> dealing >> >> > with >> >> > asyncs (yield from coroutines, async/await rewrites etc.), >> >> >> >> But unless you are proposing some kind of radical change to add >> >> compile-time type checking/inference to Python, the rewrite option is >> >> unavailable in Python. >> >> >> >> > so I would not >> >> > say that futures encourage callback-style programming, >> >> >> >> The concurrent.futures.Future class does not. But unless I misread >> >> your proposal, your extensions do. >> >> >> >> > it?s simply a >> >> > lower-layer functionality. On the contrary, monadic >> >> >> >> (Say that word one more time and everyone tunes out. :-) >> >> >> >> > methods for futures >> >> > composition (e.g. map(), all(), first() etc.) ensure that no errors >> >> > would be >> >> > lost in the process, so I think they would complement yield from >> model >> >> > quite >> >> > nicely by hiding complexity of state maintenance from user and >> reducing >> >> > the >> >> > number of back-and-forth communications between event loop and >> >> > coroutines. >> >> >> >> I'm not sure I follow. Again, I'm not sure if you've actually written >> >> any code using asyncio. >> >> >> >> TBH I've written a fair number of example programs for asyncio and >> >> I've very rarely felt the need for these composition functions. The >> >> main composition primitive I tend to use is "yield from". >> >> >> >> > Besides Futures, reactive programming >> >> >> >> What *is* reactive programming? If you're talking about >> >> http://en.wikipedia.org/wiki/Reactive_programming, >> >> I'm not sure that it maps well to Python. >> >> >> >> > has more utilities to offer, such as >> >> > Observables (representing asynchronous streams of values). It is >> also a >> >> > very >> >> > useful abstraction with a rich set of composition strategies >> (merging, >> >> > concatenation, grouping), and may deserve its place in separate >> package. >> >> >> >> It all sounds very abstract and academic. :-) >> >> >> >> > Hope to hear back from you to get better picture on overall design >> >> > direction >> >> > here before jumping to implementation details. >> >> > >> >> > Brief follow-up to your questions: >> >> > - Idea behind the Future/Promise separation is to draw a clean line >> >> > between >> >> > client-facing and scheduler-side APIs respectively. Futures should >> not >> >> > expose any completion methods, which clients should not call anyway. >> >> >> >> Given that Future and Promise are often used synonymously, using them >> >> to make this distinction sounds confusing. I agree that Futures have >> >> two different APIs, one for the consumer and another for the producer. >> >> But I'm not sure that it's necessary to separate them more strictly -- >> >> convention seems good enough to me here. (It's the same with many >> >> communication primitives, like queues and even threads.) >> >> >> >> (The one thing that trips people up frequently is that, while >> >> set_result() and set_exception() are producer APIs, cancel() is a >> >> consumer API, and the cancellation signal travels from the consumer to >> >> the producer.) >> >> >> >> > - Completely agree with you that Twisted-style callbacks are evil >> and >> >> > it is >> >> > better to have single code path for getting result or raising >> exception >> >> > - Sorry for bad grammar in the proposal, it?s an early draft >> written in >> >> > 3 >> >> > AM, so I will definitely improve on it if we decide to move forward. >> >> >> >> No problem! >> >> >> >> > Thanks, >> >> > Sergii >> >> >> >> -- >> >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> >> Python-ideas mailing list >> >> Python-ideas at python.org >> >> https://mail.python.org/mailman/listinfo/python-ideas >> >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> > >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhtonyuk at gmail.com Tue Dec 24 17:56:23 2013 From: mikhtonyuk at gmail.com (Sergii Mikhtoniuk) Date: Tue, 24 Dec 2013 18:56:23 +0200 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: Thanks everyone for your feedback. Taking all your suggestions into account I have revised my proposal . In short, it?s now: - defines separate Future classes for cooperative and multithreaded cases in concurrent.futures package - multithreaded implementation adds thread-safety to basic implementation, so in cooperative concurrency case there is absolutely no overhead - cooperative future?s interface is identical to asyncio.future - asyncio.Future inherits from concurrent.futures.cooperative.Future adding only methods specific to `yield from` - adds common composition methods for futures (intended to replace and enhance asyncio.wait/gather and concurrent.futures.wait) There?s still some work to be done for backward compatibility of concurrent.futures.Future, but implementation is almost ready . Would really appreciate if you could take a look. Thanks, Sergii On Mon, Dec 23, 2013 at 12:42 AM, Guido van Rossum wrote: > Aha. That is clever. I will have to look into the details more, but the > idea is promising. Sorry I didn't see tty his before. > On Dec 22, 2013 12:30 PM, "Ben Darnell" wrote: > >> On Sat, Dec 21, 2013 at 10:45 PM, Guido van Rossum wrote: >> >>> There's still the issue that in the threading version, you wait for a >>> Future by blocking the current thread, while in the asyncio version, >>> you must use "yield from" to block. For interoperability you would >>> have to refrain from *any* blocking operations (including "yield >>> from") so you would only be able to use callbacks. But whether you had >>> to write "x = f.result()" or "x = concurrent.futures.wait_for(f)", >>> either way you'd implicitly be blocking the current thread. >>> >> >> Threaded *consumers* of Futures wait for them by blocking, while >> asynchronous consumers wait for them by yielding. It doesn't matter >> whether the *producer* of the Future is threaded or asynchronous (except >> that if you know you won't be using threads you can use a faster >> thread-unsafe Future implementation). >> >> -Ben >> >> >>> >>> Yes, a clever scheduler could run other callbacks while blocking, but >>> that's not a complete solution, because another callback might do a >>> similar blocking operation, and whatever that waited for could hold up >>> the earlier blocking operation, causing an ever-deeper recursion and >>> of event loop invocations that might never complete. (I've had to >>> debug this in production code.) To cut through that you'd have to have >>> some kind of stack-swapping coroutine implementation like gevent, or a >>> syntactic preprocessor that inserts yield or yield-from operations >>> (I've heard from people who do this), or you'd need a clairvoyant >>> scheduler that would know which callbacks won't block. >>> >>> I like the C# solution, but it depends on static typing so a compiler >>> can know when to emit the coroutine interactions. That wouldn't work >>> in Python, unless you made the compiler recognizing the wait_for() >>> operation by name, which feels unsavory (although we do it for super() >>> :-). >>> >>> I guess for extreme interop, callbacks that never block is your only >>> option anyway, but I'd be sad if we had to to recommend this as the >>> preferred paradigm, or claim that it is all you need. >>> >>> --Guido (if I don't respond to this thread for the next two weeks, >>> it's because I'm on vacation :-) >>> >>> >>> On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell wrote: >>> > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum >>> wrote: >>> >> >>> >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk < >>> mikhtonyuk at gmail.com> >>> >> wrote: >>> >> > Indeed there is a lot of overlap between asyncio and >>> concurrent.futures >>> >> > packages, so it would be very interesting to hear your overall >>> thoughts >>> >> > on >>> >> > role/future of concurrent package. Do you consider it rudimentary >>> and >>> >> > replaceable by asyncio.Future completely? >>> >> >>> >> They don't really compare. concurrent.futures is about *threads*. >>> >> asyncio.Future is about *avoiding* threads in favor of more >>> >> lightweight "tasks" and "coroutines", which in turn are built on top >>> >> of lower-level callbacks. >>> > >>> > >>> > concurrent.futures.ThreadPoolExecutor is about threads; the Future >>> class >>> > itself is broader. When I integrated Futures into Tornado I used >>> > concurrent.futures.Future directly (when available). asyncio.Future >>> is just >>> > an optimized version of c.f.Future (the optimization comes from >>> assuming >>> > single-threaded usage). There should at least be a common ABC between >>> them. >>> > >>> >> >>> >> >>> >> (While I want to get away from callbacks as a programming paradigm, >>> >> asyncio uses them at the lower levels both because they are a logical >>> >> low-level building block and for interoperability with other >>> >> frameworks like Tornado and Twisted.) >>> >> >>> >> > I think the question is even not much about which is your preferred >>> >> > implementation, but rather do we want having futures as stand-alone >>> >> > package >>> >> > or not. Do you see all these implementations converging in future? >>> >> >>> >> I see them as not even competing. They use different paradigms and >>> >> apply to different use cases. >>> >> >>> >> > To me Future is a very simple and self-contained primitive, >>> independent >>> >> > of >>> >> > thread pools, processes, IO, and networking. >>> >> >>> >> (Agreed on the I/O and networking part only.) >>> >> >>> >> > One thing concurrent.futures >>> >> > package does a very good job at is defining an isolated namespace >>> for >>> >> > futures, stressing out this clear boundary (let?s disregard here >>> >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some >>> reason >>> >> > ended up in it too). >>> >> >>> >> Actually the executor API is an important and integral part of that >>> >> package, and threads underlie everything you can do with its Futures. >>> >> >>> >> > So when I think of schedulers and event loops implementations I see >>> them >>> >> > as >>> >> > ones that build on top of the Future primitive, not providing it as >>> part >>> >> > of >>> >> > their implementation. What I think is important is that having >>> unified >>> >> > Future class simplifies interoperability between different kinds of >>> >> > schedulers, not necessarily associated with asyncio event loops >>> (process >>> >> > pools for example). >>> >> >>> >> The interoperability is completely missing. I can't tell if you've >>> >> used asyncio at all, but the important operation of *waiting* for a >>> >> result is fundamentally different there than in concurrent.futures. In >>> >> the latter, you just write "x = f.result()" and your thread blocks >>> >> until the result is available. In asyncio, you have to write "x = >>> >> yield from f.result()" which is a coroutine block that lets other >>> >> tasks run in the same thread. ("yield from" in this case is how Python >>> >> spells the operation that C# calls "await"). >>> > >>> > >>> > The way I see it, the fundamental operation on Futures is >>> add_done_callback. >>> > We then have various higher-level operations that let us get away from >>> using >>> > callbacks directly. One of these happens to be a method on Future: the >>> > blocking mode of Future.result(). Another is implemented in asyncio >>> and >>> > Tornado, in the ability to "yield" a Future. The blocking mode of >>> result() >>> > is just a convenience; if asyncio-style futures had existed first then >>> we >>> > could instead have a function like "x = >>> concurrent.futures.wait_for(f)". In >>> > fact, you could write this wait_for function today in a way that works >>> for >>> > both concurrent and asyncio futures. >>> > >>> > This is already interoperable: Tornado's Resolver interface >>> > ( >>> https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184) >>> > returns a Future, which may be generated by a ThreadPoolExecutor or an >>> > asynchronous wrapper around pycares or twisted. In the other >>> direction I've >>> > worked on hybrid apps that have one Tornado thread alongside a bunch of >>> > Django threads; in these apps it would work to have a Django thread >>> block on >>> > f.result() for a Future returned by Tornado's AsyncHTTPClient. >>> > >>> > -Ben >>> > >>> >> >>> >> > Futures are definitely not the final solution for concurrency >>> problem >>> >> > but >>> >> > rather a well-established utility for representing async >>> operations. It >>> >> > is a >>> >> > job of higher layer systems to provide more convenient ways for >>> dealing >>> >> > with >>> >> > asyncs (yield from coroutines, async/await rewrites etc.), >>> >> >>> >> But unless you are proposing some kind of radical change to add >>> >> compile-time type checking/inference to Python, the rewrite option is >>> >> unavailable in Python. >>> >> >>> >> > so I would not >>> >> > say that futures encourage callback-style programming, >>> >> >>> >> The concurrent.futures.Future class does not. But unless I misread >>> >> your proposal, your extensions do. >>> >> >>> >> > it?s simply a >>> >> > lower-layer functionality. On the contrary, monadic >>> >> >>> >> (Say that word one more time and everyone tunes out. :-) >>> >> >>> >> > methods for futures >>> >> > composition (e.g. map(), all(), first() etc.) ensure that no errors >>> >> > would be >>> >> > lost in the process, so I think they would complement yield from >>> model >>> >> > quite >>> >> > nicely by hiding complexity of state maintenance from user and >>> reducing >>> >> > the >>> >> > number of back-and-forth communications between event loop and >>> >> > coroutines. >>> >> >>> >> I'm not sure I follow. Again, I'm not sure if you've actually written >>> >> any code using asyncio. >>> >> >>> >> TBH I've written a fair number of example programs for asyncio and >>> >> I've very rarely felt the need for these composition functions. The >>> >> main composition primitive I tend to use is "yield from". >>> >> >>> >> > Besides Futures, reactive programming >>> >> >>> >> What *is* reactive programming? If you're talking about >>> >> http://en.wikipedia.org/wiki/Reactive_programming, >>> >> I'm not sure that it maps well to Python. >>> >> >>> >> > has more utilities to offer, such as >>> >> > Observables (representing asynchronous streams of values). It is >>> also a >>> >> > very >>> >> > useful abstraction with a rich set of composition strategies >>> (merging, >>> >> > concatenation, grouping), and may deserve its place in separate >>> package. >>> >> >>> >> It all sounds very abstract and academic. :-) >>> >> >>> >> > Hope to hear back from you to get better picture on overall design >>> >> > direction >>> >> > here before jumping to implementation details. >>> >> > >>> >> > Brief follow-up to your questions: >>> >> > - Idea behind the Future/Promise separation is to draw a clean line >>> >> > between >>> >> > client-facing and scheduler-side APIs respectively. Futures should >>> not >>> >> > expose any completion methods, which clients should not call anyway. >>> >> >>> >> Given that Future and Promise are often used synonymously, using them >>> >> to make this distinction sounds confusing. I agree that Futures have >>> >> two different APIs, one for the consumer and another for the producer. >>> >> But I'm not sure that it's necessary to separate them more strictly -- >>> >> convention seems good enough to me here. (It's the same with many >>> >> communication primitives, like queues and even threads.) >>> >> >>> >> (The one thing that trips people up frequently is that, while >>> >> set_result() and set_exception() are producer APIs, cancel() is a >>> >> consumer API, and the cancellation signal travels from the consumer to >>> >> the producer.) >>> >> >>> >> > - Completely agree with you that Twisted-style callbacks are evil >>> and >>> >> > it is >>> >> > better to have single code path for getting result or raising >>> exception >>> >> > - Sorry for bad grammar in the proposal, it?s an early draft >>> written in >>> >> > 3 >>> >> > AM, so I will definitely improve on it if we decide to move forward. >>> >> >>> >> No problem! >>> >> >>> >> > Thanks, >>> >> > Sergii >>> >> >>> >> -- >>> >> --Guido van Rossum (python.org/~guido) >>> >> _______________________________________________ >>> >> Python-ideas mailing list >>> >> Python-ideas at python.org >>> >> https://mail.python.org/mailman/listinfo/python-ideas >>> >> Code of Conduct: http://python.org/psf/codeofconduct/ >>> > >>> > >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Sat Dec 21 23:29:14 2013 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang) Date: Sat, 21 Dec 2013 14:29:14 -0800 (PST) Subject: [Python-ideas] thoughts on the new 3.4 statistics module Message-ID: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> First of all: thank you, Steven and everyone else involved, for taking on the task of starting to implement this long-missed (at least by me) feature ! I really hope the module will be a success and grow over time. I have two thoughts at the moment about the implementation that I think may be worth discussing, if it hasn't happened yet (I have to admit I did not go through all previous posts on this topic, only read the PEP): First: I am not entirely convinced by when the module raises Errors. In some places its undoubtedly justified to raise StatisticsError (like when empty sequences are passed to mean()). On the other hand, should there really be an error, when for example no unique value for the mode can be found? Effectively, that would force users to guard every (!) call to the function with try/except. In my opinion, a better choice would be to return float('nan') or even better a module-specific object (call it Undefined or something) that one can check for. This behavior could, in general, be implemented for cases, where input can actually be handled and a result be calculated (like a list of values in the mode example), but this result is considered "undefined" by the algorithm. Second: I am not entirely happy with the three different flavors of the median function. I *do* know that this has been discussed before, but I'm not sure whether *all* alternatives have been considered (the PEP only talks about the median.low, median.high syntax, which, in fact, I wouldn't like that much either. My suggestion would be to have a resolve parameter, by which the behavior of a single median function can be modified. My main argument here is that as the module will grow in the future there will be many more such situations, in which different ways of calculating statistics are all totally acceptable and you would want to leave the choice to the user (the mode function can already be considered as an example: maybe the user would want to have the list of "modes" returned in case that no unambiguous value can be calculated; actually the current code seems to be prepared for later implementation of this feature because it does generate the list, just is not returning it). Now if, in all such situations, the solution is to have extra functions the module will soon end up completely cluttered with them. If, on the other hand, every function that will foreseeably have to handle ambiguous situations had a resolve parameter the module structure would be much clearer. In the median example you would then call median(data) for the default behavior, arguably the interpolation, but median(data, resolve='low') or median(data, resolve='high') for the alternative calculations. Statistically educated users could then guess, relatively easily, which functions have the resolve parameter and a quick look at the function's help could tell them, which arguments are accepted. Finally, let me just point out that these are really just first thoughts and I do understand that these are design decisions about which different people will have different opinions, but I think now is still a good time to discuss them, while with an established and (hopefully :) ) much larger module you will not be able to change things that easily anymore. Hoping for a lively discussion, Wolfgang -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Dec 24 20:13:09 2013 From: phd at phdru.name (Oleg Broytman) Date: Tue, 24 Dec 2013 20:13:09 +0100 Subject: [Python-ideas] thoughts on the new 3.4 statistics module In-Reply-To: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> References: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> Message-ID: <20131224191309.GA28949@phdru.name> Hi! On Sat, Dec 21, 2013 at 02:29:14PM -0800, Wolfgang wrote: > First: I am not entirely convinced by when the module raises Errors. In > some places its undoubtedly justified to raise StatisticsError (like when > empty sequences are passed to mean()). > On the other hand, should there really be an error, when for example no > unique value for the mode can be found? > Effectively, that would force users to guard every (!) call to the function > with try/except. Not necessary. The user of the library can combine a few calls in a function/method and catch one exception for the entire calculation. Or catch it even higher up the stack. > In my opinion, a better choice would be to return > float('nan') or even better a module-specific object (call it Undefined or > something) that one can check for. With such special values the user must check every return value. What is the advantage over catching exceptions? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From steve at pearwood.info Wed Dec 25 01:47:25 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 25 Dec 2013 11:47:25 +1100 Subject: [Python-ideas] thoughts on the new 3.4 statistics module In-Reply-To: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> References: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> Message-ID: <20131225004724.GI29356@ando> Hi Wolfgang, and thanks for the feedback! My responses below. On Sat, Dec 21, 2013 at 02:29:14PM -0800, Wolfgang wrote: > First: I am not entirely convinced by when the module raises Errors. In > some places its undoubtedly justified to raise StatisticsError (like when > empty sequences are passed to mean()). > On the other hand, should there really be an error, when for example no > unique value for the mode can be found? There was no agreement on the best way to handle data with multiple modes, so we went with the simplest version that could work. It's easier to add functionality to the standard library than to take it away: better to delay putting something in for a release or two, than to put it in and then be stuck with the consequences of a poor decision for years. An earlier version of statistics.py included a mode function that let you specify the maximum number of modes. That function may eventually be added to the module, or made available on PyPI. The version included in the standard library implements the basic, school-book version of mode: it returns the one unique mode, as calculated by counting distinct values, or it fails, and the most Pythonic way to implement failure is with an exception. > Effectively, that would force users to guard every (!) call to the function > with try/except. No different from any other function. If you think a function might fail, then you guard it with try...except. > In my opinion, a better choice would be to return > float('nan') or even better a module-specific object (call it Undefined or > something) that one can check for. This behavior could, in general, be > implemented for cases, where input can actually be handled and a result be > calculated (like a list of values in the mode example), but this result is > considered "undefined" by the algorithm. You can easily get that behaviour with a simple wrapper function: def my_mode(values): try: return mode(values) except StatisticsError: return float('nan') But I'm not convinced that this is appropriate for nominal data. Would you expect that the mode of ['red', 'blue', 'green'] should be a floating point NAN? I know I wouldn't. > Second: I am not entirely happy with the three different flavors of the > median function. I *do* know that this has been discussed before, but I'm > not sure whether *all* alternatives have been considered (the PEP only > talks about the median.low, median.high syntax, which, in fact, I wouldn't > like that much either. My suggestion would be to have a resolve parameter, > by which the behavior of a single median function can be modified. For median, I don't believe this is appropriate. As a general rule, if a function has a parameter which is usually called with a constant known when you write the source code: median(data, resolve='middle') # resolve is known at edit-time especially if that parameter takes only two or three values, then the function probably should be split into two or three separate functions. I don't think that there are any common use-cases for selecting the type of median at *runtime*: kind = get_median_kind() median(data, resolve=kind) but if you can think of any, I'd like to hear them. However, your general suggestion isn't entirely inappropriate. In my research, I learned that there are at least fifteen different definitions of quartiles in common use, although some are mathematically equivalent. See here: http://www.amstat.org/publications/jse/v14n3/langford.html I find six distinct definitions for quartiles, and ten for quantiles/ fractiles. R supports nine different quantile versions, Haskell six, and SAS also supports multiple versions. (I don't remember how many.) Mathematica provides a four-argument parameterized version of Quantile. With six distinct versions of quartile, and ten of quantile, it's too many to provide separate functions for each: too much duplication, too much clutter. Most people won't care which quantile they get, so there ought to be a sensible default. For those who care about matching some particular version (say, that used by Excel, or that used by Texas Instruments calculators), there ought to be a parameter that allows you to select which version is used. R calls this parameter "type". I don't remember what SAS and Haskell call it, but the term I prefer is "scheme". I don't know if statistics.py will ever gain a function for calculating quantiles other than the median. I will probably put quantiles and quartiles on PyPI first, and if I do, I will follow your suggestion to provide a parameter to select the version used (although I'll probably call it "scheme" rather than "resolve"). -- Steven From python at mrabarnett.plus.com Wed Dec 25 04:15:04 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 25 Dec 2013 03:15:04 +0000 Subject: [Python-ideas] thoughts on the new 3.4 statistics module In-Reply-To: <20131225004724.GI29356@ando> References: <45123489-39fd-4bdb-aeab-47443d6f0939@googlegroups.com> <20131225004724.GI29356@ando> Message-ID: <52BA4DB8.7040502@mrabarnett.plus.com> On 25/12/2013 00:47, Steven D'Aprano wrote: > Hi Wolfgang, and thanks for the feedback! My responses below. > > On Sat, Dec 21, 2013 at 02:29:14PM -0800, Wolfgang wrote: > >> First: I am not entirely convinced by when the module raises Errors. In >> some places its undoubtedly justified to raise StatisticsError (like when >> empty sequences are passed to mean()). >> On the other hand, should there really be an error, when for example no >> unique value for the mode can be found? > > There was no agreement on the best way to handle data with multiple > modes, so we went with the simplest version that could work. It's easier > to add functionality to the standard library than to take it away: > better to delay putting something in for a release or two, than to put > it in and then be stuck with the consequences of a poor decision for > years. > > An earlier version of statistics.py included a mode function that let > you specify the maximum number of modes. That function may eventually be > added to the module, or made available on PyPI. The version included in > the standard library implements the basic, school-book version of mode: > it returns the one unique mode, as calculated by counting distinct > values, or it fails, and the most Pythonic way to implement failure is > with an exception. > [snip] Data that has multiple modes is "multimodal" (who said stats was difficult? :-)), so perhaps there could be a "multimode" function that returns a list of modes. From techtonik at gmail.com Wed Dec 25 08:47:11 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 25 Dec 2013 10:47:11 +0300 Subject: [Python-ideas] Reducing language complexity Message-ID: One of the tools to reduce language complexity is "explicitness" or the direct link to help/tutorial/documentation from the concept. The problem with most concepts in computer languages that they don't have distinct markers by which you can recognize one feature or another. For example, you can't recognize that code is generator based or uses metaclass magic without searching for yield or some references to metaclass through the source file. One of the ways to reduce language complexity for new people who read you code, is to prepare them for advanced concepts that your code uses beforehand. For example, with the following section: using generators as yield ^^^ name of this language feature and also help reference ^^^ distinct keywords and feature markers that you enable -- anatoly t. From rosuav at gmail.com Wed Dec 25 09:53:14 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Dec 2013 19:53:14 +1100 Subject: [Python-ideas] Reducing language complexity In-Reply-To: References: Message-ID: On Wed, Dec 25, 2013 at 6:47 PM, anatoly techtonik wrote: > One of the ways to reduce language complexity for new people who read > you code, is to prepare them for advanced concepts that your code uses > beforehand. For example, with the following section: > > using generators as yield > > ^^^ name of this language feature and also help reference > ^^^ distinct keywords and feature > markers that you enable Yes, and it's great that you can choose what the keyword is! We should announce the usage of this thus: using using as as using as as using and only then use using and as as as and using. And of course, it's then critical to declare the fact that you're using built-in functions, in case they confuse people too. And their parameters might be confusing, too; best to acknowledge the exact parameter lists that each built-in function uses. Something like this: using print(*values, sep=' ', end='\n', file=sys.stdout, flush=False) using open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Of course, this could get rather long, so we could break things out into other files, and put all those 'using' declarations out of the way. Rather than create new syntax for all this, I recommend we use specially-formatted comments; that way, older versions of Python won't be bothered by it. So let's put those two declarations into a file called stdio.h, and put this at the top of your Python script: #include I'm sure everyone will agree that this is the best thing for Python. ChrisA From techtonik at gmail.com Wed Dec 25 10:03:37 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 25 Dec 2013 12:03:37 +0300 Subject: [Python-ideas] Reducing language complexity In-Reply-To: References: Message-ID: On Wed, Dec 25, 2013 at 11:53 AM, Chris Angelico wrote: > On Wed, Dec 25, 2013 at 6:47 PM, anatoly techtonik wrote: >> One of the ways to reduce language complexity for new people who read >> you code, is to prepare them for advanced concepts that your code uses >> beforehand. For example, with the following section: >> >> using generators as yield >> >> ^^^ name of this language feature and also help reference >> ^^^ distinct keywords and feature >> markers that you enable > > Yes, and it's great that you can choose what the keyword is! We should > announce the usage of this thus: > > using using as as > using as as using > > and only then use using and as as as and using. > > And of course, it's then critical to declare the fact that you're > using built-in functions, in case they confuse people too. And their > parameters might be confusing, too; best to acknowledge the exact > parameter lists that each built-in function uses. Something like this: > > using print(*values, sep=' ', end='\n', file=sys.stdout, flush=False) > using open(file, mode='r', buffering=-1, encoding=None, > errors=None, newline=None, closefd=True, opener=None) > > Of course, this could get rather long, so we could break things out > into other files, and put all those 'using' declarations out of the > way. Rather than create new syntax for all this, I recommend we use > specially-formatted comments; that way, older versions of Python won't > be bothered by it. So let's put those two declarations into a file > called stdio.h, and put this at the top of your Python script: > > #include > > I'm sure everyone will agree that this is the best thing for Python. Unfortunately, the first message with "Language complexity formula" is deemed to be "not python related" by moderators, so this mail landed without necessary context. Without understanding what adds to the language complexity, this discussion looks useless. -- anatoly t. From taleinat at gmail.com Wed Dec 25 10:23:12 2013 From: taleinat at gmail.com (Tal Einat) Date: Wed, 25 Dec 2013 11:23:12 +0200 Subject: [Python-ideas] Reducing language complexity In-Reply-To: References: Message-ID: On Wed, Dec 25, 2013 at 11:03 AM, anatoly techtonik wrote: > > Unfortunately, the first message with "Language complexity formula" is > deemed to be "not python related" by moderators, so this mail landed > without necessary context. Without understanding what adds to the > language complexity, this discussion looks useless. > -- > anatoly t. Hi Anatoly, I'm the moderator who rejected your other message titled "Language complexity formula". To be clear, I did not write that the rejected message was "not Python related". I wrote that it was not relevant specifically to the Python-Ideas mailing list. As I'm sure you know, this list is for discussing ideas for the Python language. Your message only suggested a formula for evaluating the complexity of programming languages, while containing no ideas or suggestions for the language itself. If that message was intended as context for this message (titled "Reducing language complexity"), then you should have included the context as well as the suggestion in a single message. To be clear, please do not post several messages which discuss the same subject, or very similar subjects, in parallel. Of course, you are free to add context and explanations to the discussion as you see fit. Regards, - Tal Einat From abarnert at yahoo.com Wed Dec 25 10:35:17 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 25 Dec 2013 01:35:17 -0800 Subject: [Python-ideas] Reducing language complexity In-Reply-To: References: Message-ID: <179F68B2-33D5-41B3-A7E7-6FCD2E90A802@yahoo.com> When I search for "python generators" instead of "python yield", I get the same docs pages, blog posts, and StackOverflow questions making up 90% of the results, in the same order. When I search for "generators" instead of "yield" at docs.python.org, I get _worse_ results--a bunch of stuff about the C API and then a slew of random modules--the yield statement, the relevant tutorial sections, etc. are nowhere to be seen. When I use the built-in help, "yield" gives me a nice overview; "generators" tells me there's no documentation. So, in what way is "generators" a better term for finding help/tutorial/documentation than "yield"? And I'm not sure what term for the metaclass feature would be better than "metaclass". (If you were going to suggest "using metaclass as __metaclass__" I might agree that was a small improvement, if Python hadn't fixed that 5 years ago.) Sent from a random iPhone On Dec 24, 2013, at 23:47, anatoly techtonik wrote: > One of the tools to reduce language complexity is "explicitness" or > the direct link to help/tutorial/documentation from the concept. The > problem with most concepts in computer languages that they don't have > distinct markers by which you can recognize one feature or another. > For example, you can't recognize that code is generator based or uses > metaclass magic without searching for yield or some references to > metaclass through the source file. > > One of the ways to reduce language complexity for new people who read > you code, is to prepare them for advanced concepts that your code uses > beforehand. For example, with the following section: > > using generators as yield > > ^^^ name of this language feature and also help reference > ^^^ distinct keywords and feature > markers that you enable > -- > anatoly t. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Wed Dec 25 11:19:32 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 25 Dec 2013 21:19:32 +1100 Subject: [Python-ideas] Reducing language complexity In-Reply-To: References: Message-ID: <20131225101931.GK29356@ando> On Wed, Dec 25, 2013 at 12:03:37PM +0300, anatoly techtonik wrote: > Unfortunately, the first message with "Language complexity formula" is > deemed to be "not python related" by moderators, so this mail landed > without necessary context. Without understanding what adds to the > language complexity, this discussion looks useless. You should write a blog post about the language complexity formula. That way people who are motivated by it can read it, while those who don't won't need to be bothered by an off-topic discussion on something which isn't specific to Python. As for your suggestion to "prepare" readers for "advanced" concepts: using generators as yield who defines what counts as "advanced"? Python for-loops iterating over sequences may seen advanced to a C or Pascal programmer expecting to write code like "for i := 1 to 50 do process(seq[i]);". Exceptions may seem advanced to Java programmers. Even functions may seem terribly advanced to those who have never programmed before. (And I've seen them on the tutor list.) How do you expect this "using" statement to actually help the reader? I have never, ever, not even as a total newbie knowing next to nothing about Python (or any other language) been in a position where I was reading code, came up on a keyword, function or idiom I didn't understand, and thought "Oh, if only this was declared at the beginning of the file, I would have understood it better!". On the other hand, I would feel pretty annoyed if I came across a file containing using flibbit as smudgeon at the start, spent a lot of time reading up on flibbit and smudgeon, then came back to the file and discovered that the hours I had spent was completely unnecessary because the part of the file I cared about didn't actually use flibbit or smudgeon. If a reader comes across "advanced" code they don't understand: def increment(values): for value in values: yield value+1 isn't it pretty trivial for them to ask "what does yield do?". You don't need a "using" statement to lead them to the right search terms. If you think the search terms aren't obvious enough and want to write code aimed at the least-experienced reader, just use a comment, and put it right where it is needed: # Keywords: yield, generator, Python. # http://link-to-your-blog-explaining-this def increment(values): for value in values: yield value+1 -- Steven From mcepl at redhat.com Thu Dec 26 23:17:34 2013 From: mcepl at redhat.com (Matej Cepl) Date: Thu, 26 Dec 2013 23:17:34 +0100 Subject: [Python-ideas] textwrap.TextWrapper width=None (or Inf) Message-ID: <20131226221733.GA21940@wycliff.ceplovi.cz> Hi, I wonder why nobody asked on bugs.python.org for rather obvious functionality of being able to reflow a paragraph to one line? Meaning, that any paragraph would be stripped of all whitespace (etc. ... whatever is configured by the additional parameters of the TextWrapper class) and then joined into long line. I know that ''.join(text.splitlines()) does something similar, but a) it doesn't handle all whitespace munging, b) it just seems like an obvious functionality for TextWrapper to have. Any thoughts on it? Should I just file a bug? Best, Mat?j -- http://www.ceplovi.cz/matej/, Jabber: mceplceplovi.cz GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC [...] a superior pilot uses his superior judgment to avoid having to exercise his superior skill. -- http://www.jwz.org/blog/2009/09/that-duct-tape-silliness/#comment-10653 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From tjreedy at udel.edu Thu Dec 26 23:51:42 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 26 Dec 2013 17:51:42 -0500 Subject: [Python-ideas] textwrap.TextWrapper width=None (or Inf) In-Reply-To: <20131226221733.GA21940@wycliff.ceplovi.cz> References: <20131226221733.GA21940@wycliff.ceplovi.cz> Message-ID: On 12/26/2013 5:17 PM, Matej Cepl wrote: > I wonder why nobody asked on bugs.python.org for rather obvious > functionality of being able to reflow a paragraph to one line? Because width=4000000000 should do just that? > Any thoughts on it? Should I just file a bug? It would be an enhancement request, not a bug report, but it seems not needed. -- Terry Jan Reedy From steve at pearwood.info Fri Dec 27 00:17:44 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 27 Dec 2013 10:17:44 +1100 Subject: [Python-ideas] textwrap.TextWrapper width=None (or Inf) In-Reply-To: <20131226221733.GA21940@wycliff.ceplovi.cz> References: <20131226221733.GA21940@wycliff.ceplovi.cz> Message-ID: <20131226231743.GO29356@ando> On Thu, Dec 26, 2013 at 11:17:34PM +0100, Matej Cepl wrote: > Hi, > > I wonder why nobody asked on bugs.python.org for rather obvious > functionality of being able to reflow a paragraph to one line? Possibly because nobody needed the functionality? Or because they didn't think that *wrapping* a long line into a paragraph and *unwrapping* a paragraph into a single line should be handled by the same function? > Meaning, that any paragraph would be stripped of all whitespace > (etc. ... whatever is configured by the additional parameters of > the TextWrapper class) and then joined into long line. I know > that > > ''.join(text.splitlines()) > > does something similar, but I would expect that you should use ' '.join, rather than the empty string. Otherwises lines will be incorrectly concatenated: """the cat in the hat""" => "the cat inthe hat" > a) it doesn't handle all whitespace munging, Can you given an example of what whitespace munging it fails to handle? > b) it just seems like an obvious functionality for > TextWrapper to have. py> text = """the cat in ... the hat""" py> textwrap.wrap(text, width=len(text)) ['the cat in the hat'] Is there a case that this does not handle? Perhaps this is not obvious enough. A simple helper function may increase discoverability: def unwrap(text): return wrap(text, width=len(text)) -- Steven From amber.yust at gmail.com Fri Dec 27 02:57:02 2013 From: amber.yust at gmail.com (Amber Yust) Date: Fri, 27 Dec 2013 01:57:02 +0000 Subject: [Python-ideas] "maybe import"? Message-ID: <-7047415593530210756@gmail297201516> It's a fairly standard pattern to see things like this: try: import foo except ImportError: foo = None (and of course, variants with from...import et cetera). These can potentially add a lot of clutter to the imports section of a file, given that it requires 4 lines to do a conditional import. It seems like it'd be useful and clean to have a syntax that looked like this: maybe import foo from bar maybe import baz from qux maybe import quy as quz Where the behavior would essentially be as above - attempt to run the import normally, and in cases where the import fails, map the name to a value of None instead. Users who want a different behavior are still free to use the long-form syntax. A possibly variant might be to also only run the import if the name isn't already bound, so that you could do something like... from frobber_a maybe import frob as frobber from frobbler_b maybe import frobble as frobber from frobber_c maybe import frobit as frobber ...to potentially try different fallback options if the first choice for an interface provider isn't available. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Dec 27 05:08:09 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 27 Dec 2013 15:08:09 +1100 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-7047415593530210756@gmail297201516> References: <-7047415593530210756@gmail297201516> Message-ID: <20131227040808.GP29356@ando> On Fri, Dec 27, 2013 at 01:57:02AM +0000, Amber Yust wrote: > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > (and of course, variants with from...import et cetera). These can > potentially add a lot of clutter to the imports section of a file, given > that it requires 4 lines to do a conditional import. You can reduce that down to two lines: try: import foo except ImportError: foo = None but your point is taken. A more common pattern in my experience is: try: import this except ImportError: import that as this > It seems like it'd be useful and clean to have a syntax that looked like > this: > > maybe import foo > from bar maybe import baz > from qux maybe import quy as quz > > Where the behavior would essentially be as above - attempt to run the > import normally, and in cases where the import fails, map the name to a > value of None instead. Users who want a different behavior are still free > to use the long-form syntax. Hmmm. The basic idea makes a certain level of sense to me, but I'm not sure it makes enough sense to overcome the barrier required before adding a new keyword. I'm not (yet) convinced of the need for this functionality, but if Python did gain this, I think I would prefer the colour of this bike-shed to be "perhaps import" rather than "maybe import". A couple of reasons: - it seems to me that "maybe" is more likely to already be used in code than "perhaps", e.g. in three-value logics (true, false, maybe); - to me, "maybe" feels somewhat random, arbitrary or indeterminant, whereas "perhaps" feels subtly more determinant. I can't justify this claim by dictionary definitions, perhaps it's just me :-) If we're entertaining changes to imports, another possibility would be to allow fallback module names: import this or that or another as this Each of "this", "that", "another" will be attempted, the first successful import being bound to the name "this". The "as this" part would be mandatory, so as to require a consistent name regardless of which module was imported. This would be a syntax error, since it isn't clear what name would be bound at the end: import this or that or another This would also be allowed: from this or that or another import spam With this syntax, we could add None as a special case: import this or that or another or None as this would be equivalent to: module_names = ("this", "that", "another", "None") for name in module_names: if name == "None": spam = None else: try: this = __import__(name) except ImportError: continue break else: raise ImportError and the "from...import" case could be written as: from this or that or None import spam roughly equivalent to: module_names = ("this", "that", "another", "None") for name in module_names: if name == "None": spam = None else: try: temp = __import__(name) spam = temp.spam except ImportError: continue break else: raise ImportError Advantages: - covers both use-cases where you want to try a series of modules, and the one where you fall back to None; - "or" is already a keyword, no new keywords needed; - reads more like English; - "import None" currently gives SyntaxError, so this can't interfere with modules actually called "None". Disadvantages: - more complexity to imports; - only saves a few lines; - this usage of "or" is not quite the same as the usage as a boolean operator, e.g. different from "x in a or b". > A possibly variant might be to also only run > the import if the name isn't already bound, so that you could do something > like... > > from frobber_a maybe import frob as frobber > from frobbler_b maybe import frobble as frobber > from frobber_c maybe import frobit as frobber > > ...to potentially try different fallback options if the first choice for an > interface provider isn't available. I dislike this form, because it requires short-circuiting execution of separate lines. What would you expect this to do? from frobber_a maybe import frob as frobber frobber = 23 from frobbler_b maybe import frobble as frobber Is the second maybe import attempted or not? I have no idea whether it should be or shouldn't be. -- Steven From amber.yust at gmail.com Fri Dec 27 05:36:21 2013 From: amber.yust at gmail.com (Amber Yust) Date: Fri, 27 Dec 2013 04:36:21 +0000 Subject: [Python-ideas] "maybe import"? References: <-7047415593530210756@gmail297201516> Message-ID: <2119483115886762605@gmail297201516> On Thu Dec 26 2013 at 7:40:19 PM, Bruce Leban wrote: > I think you mean do the import if the name is unbound *or bound to None.* Otherwise, > it doesn't work in the example you gave. > Yes, that is what I meant, sorry. Another option would be "not bound to something that is not a module" - but I think "unbound or None" is probably the most versatile option. On Thu Dec 26 2013 at 8:08:45 PM, Steven D'Aprano wrote: > I'm not (yet) convinced of the need for this functionality, but if > Python did gain this, I think I would prefer the colour of this > bike-shed to be "perhaps import" rather than "maybe import". Another option would be to re-use the 'or' keyword: from foo import bar or None Where the bit after the 'or' simply specifies a default value to assign if an ImportError occurs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amber.yust at gmail.com Fri Dec 27 05:38:29 2013 From: amber.yust at gmail.com (Amber Yust) Date: Fri, 27 Dec 2013 04:38:29 +0000 Subject: [Python-ideas] "maybe import"? References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> Message-ID: <-9004133628104978885@gmail297201516> In fact, reuse of the 'or' keyword also leads to these nice options: import foo or None as bar from foo import bar or None as baz On Thu Dec 26 2013 at 8:36:24 PM, Amber Yust wrote: > On Thu Dec 26 2013 at 7:40:19 PM, Bruce Leban wrote: > > I think you mean do the import if the name is unbound *or bound to None.* Otherwise, > it doesn't work in the example you gave. > > > Yes, that is what I meant, sorry. Another option would be "not bound to > something that is not a module" - but I think "unbound or None" is probably > the most versatile option. > > On Thu Dec 26 2013 at 8:08:45 PM, Steven D'Aprano > wrote: > > I'm not (yet) convinced of the need for this functionality, but if > Python did gain this, I think I would prefer the colour of this > bike-shed to be "perhaps import" rather than "maybe import". > > > Another option would be to re-use the 'or' keyword: > > from foo import bar or None > > Where the bit after the 'or' simply specifies a default value to assign if > an ImportError occurs. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Dec 27 05:57:20 2013 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 26 Dec 2013 20:57:20 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-7047415593530210756@gmail297201516> References: <-7047415593530210756@gmail297201516> Message-ID: On Thu, Dec 26, 2013 at 5:57 PM, Amber Yust wrote: > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > (and of course, variants with from...import et cetera). These can > potentially add a lot of clutter to the imports section of a file, given > that it requires 4 lines to do a conditional import. > > It seems like it'd be useful and clean to have a syntax that looked like > this: > > maybe import foo > from bar maybe import baz > from qux maybe import quy as quz > > Where the behavior would essentially be as above - attempt to run the > import normally, and in cases where the import fails, map the name to a > value of None instead. Users who want a different behavior are still free > to use the long-form syntax. A possibly variant might be to also only run > the import if the name isn't already bound, so that you could do something > like... > > from frobber_a maybe import frob as frobber > from frobbler_b maybe import frobble as frobber > from frobber_c maybe import frobit as frobber > > ...to potentially try different fallback options if the first choice for > an interface provider isn't available. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Such idioms are common. Though I don't think we should encourage their use. That said, if you want to have something like this added it should not use a new keyword ("maybe") but should use what we have. These look odd to me but are possible ideas: import foo else foo = None from foo import bar else bar = None that reuse of else on import statements would also enable the other idiom of importing one of several things to a single name: import foo as x else import bar as x Just tossing those out there. I'm not convinced this is worth adding to the language. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From amber.yust at gmail.com Fri Dec 27 06:57:13 2013 From: amber.yust at gmail.com (Amber Yust) Date: Fri, 27 Dec 2013 05:57:13 +0000 Subject: [Python-ideas] "maybe import"? References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> Message-ID: <-8326952152688219870@gmail297201516> On Thu Dec 26 2013 at 8:57:42 PM, Gregory P. Smith wrote: > Such idioms are common. Though I don't think we should encourage their use. Is this a case of "we shouldn't encourage optional dependencies" or "we shouldn't encourage this kind of idiom as a means of implementing optional dependencies" - and if the latter, what alternative would you point people at instead? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Dec 27 09:33:11 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Dec 2013 00:33:11 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-8326952152688219870@gmail297201516> References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> <-8326952152688219870@gmail297201516> Message-ID: We shouldn't encourage optional dependencies. They often make code harder to reason about or harder to read. On Thursday, December 26, 2013, Amber Yust wrote: > On Thu Dec 26 2013 at 8:57:42 PM, Gregory P. Smith > > wrote: > > Such idioms are common. Though I don't think we should encourage their > use. > > Is this a case of "we shouldn't encourage optional dependencies" or "we > shouldn't encourage this kind of idiom as a means of implementing optional > dependencies" - and if the latter, what alternative would you point people > at instead? > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Fri Dec 27 09:36:03 2013 From: flying-sheep at web.de (Philipp A.) Date: Fri, 27 Dec 2013 09:36:03 +0100 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> <-8326952152688219870@gmail297201516> Message-ID: agreed. and the try/except pattern also allows for try: import thing except ImportError: import slower_thing_with_same_api as thing 2013/12/27 Guido van Rossum > We shouldn't encourage optional dependencies. They often make code harder > to reason about or harder to read. > > > On Thursday, December 26, 2013, Amber Yust wrote: > >> On Thu Dec 26 2013 at 8:57:42 PM, Gregory P. Smith >> wrote: >> > Such idioms are common. Though I don't think we should encourage their >> use. >> >> Is this a case of "we shouldn't encourage optional dependencies" or "we >> shouldn't encourage this kind of idiom as a means of implementing optional >> dependencies" - and if the latter, what alternative would you point people >> at instead? >> > > > -- > --Guido van Rossum (on iPad) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amber.yust at gmail.com Fri Dec 27 10:34:14 2013 From: amber.yust at gmail.com (Amber Yust) Date: Fri, 27 Dec 2013 09:34:14 +0000 Subject: [Python-ideas] "maybe import"? References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> <-8326952152688219870@gmail297201516> Message-ID: <876459810524566913@gmail297201516> On Fri Dec 27 2013 at 12:33:11 AM, Guido van Rossum wrote: > We shouldn't encourage optional dependencies. They often make code harder > to reason about or harder to read. > Is there a better alternative for utilizing functionality that may vary in availability from platform to platform? -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Dec 27 13:23:39 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Dec 2013 04:23:39 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <876459810524566913@gmail297201516> References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> <-8326952152688219870@gmail297201516> <876459810524566913@gmail297201516> Message-ID: On Dec 27, 2013, at 1:34, Amber Yust wrote: > On Fri Dec 27 2013 at 12:33:11 AM, Guido van Rossum wrote: >> We shouldn't encourage optional dependencies. They often make code harder to reason about or harder to read. > > Is there a better alternative for utilizing functionality that may vary in availability from platform to platform? The only way to use this would be to then continually check "if foo:" all throughout your module, to use it if present and do something else if not. I suspect that is a bigger readability/comprehensibility problem than the extra lines of code up top. And in many cases, once you reorganize your code to solve that problem, this one goes away as well. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 27 15:26:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 00:26:50 +1000 Subject: [Python-ideas] "maybe import"? In-Reply-To: <876459810524566913@gmail297201516> References: <-7047415593530210756@gmail297201516> <3965602355176718462@gmail297201516> <-8326952152688219870@gmail297201516> <876459810524566913@gmail297201516> Message-ID: On 27 December 2013 19:34, Amber Yust wrote: > On Fri Dec 27 2013 at 12:33:11 AM, Guido van Rossum > wrote: >> >> We shouldn't encourage optional dependencies. They often make code harder >> to reason about or harder to read. > > > Is there a better alternative for utilizing functionality that may vary in > availability from platform to platform? The most common approach I've seen is a compatibility module, so most code in the application just does something like: from myapp.foo_compat import foo and only foo_compat has the try/except logic. Depending on the number of optional dependencies and the cost of importing them even when they're not needed, this can be done on a per-dependency basis, or as a single compatibility module that handles all the optional dependencies. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bruce at leapyear.org Fri Dec 27 04:39:38 2013 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 26 Dec 2013 19:39:38 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-7047415593530210756@gmail297201516> References: <-7047415593530210756@gmail297201516> Message-ID: On Thu, Dec 26, 2013 at 5:57 PM, Amber Yust wrote: > Where the behavior would essentially be as above - attempt to run the > import normally, and in cases where the import fails, map the name to a > value of None instead. Users who want a different behavior are still free > to use the long-form syntax. A possibly variant might be to also only run > the import if the name isn't already bound, > I think you mean do the import if the name is unbound *or bound to None.* Otherwise, it doesn't work in the example you gave. Here's another use case. Some modules, e.g., pycrypto need to be compiled in order to be installed. I have a stub version of pycrypto which can be used as a substitute for testing without actually encrypting anything. Rather than using a new keyword, the try keyword would fit here fine: try import Crypto try import CryptoStub as Crypto note that there are cases where this can't be a drop in replacement: try import Crypto.Random try import CryptoStub.Random as Crypto.Random # not allowed you'd have to instead write: try import Crypto.Random as CryptoRandom try import CryptoStub.Random as CryptoRandom --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Dec 27 17:30:43 2013 From: mertz at gnosis.cx (David Mertz) Date: Fri, 27 Dec 2013 08:30:43 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-7047415593530210756@gmail297201516> References: <-7047415593530210756@gmail297201516> Message-ID: It's a fairly standard pattern to see things like this: try: import foo except ImportError: foo = None (and of course, variants with from...import et cetera). These can potentially add a lot of clutter to the imports section of a file, given that it requires 4 lines to do a conditional import. It seems like it'd be useful and clean to have a syntax that looked like this: maybe import foo from bar maybe import baz from qux maybe import quy as quz Where the behavior would essentially be as above - attempt to run the import normally, and in cases where the import fails, map the name to a value of None instead. Users who want a different behavior are still free to use the long-form syntax. A possibly variant might be to also only run the import if the name isn't already bound, so that you could do something like... from frobber_a maybe import frob as frobber from frobbler_b maybe import frobble as frobber from frobber_c maybe import frobit as frobber ...to potentially try different fallback options if the first choice for an interface provider isn't available. _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Dec 27 18:00:41 2013 From: mertz at gnosis.cx (David Mertz) Date: Fri, 27 Dec 2013 09:00:41 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> Message-ID: On Thu, Dec 26, 2013 at 5:57 PM, Amber Yust wrote: > It's a fairly standard pattern to see things like this: > try: > import foo > except ImportError: > foo = None > I would rarely, if ever, follow this pattern, although I *would* do conditional import fairly often. E.g. these are things I might do (well, not really these specific ones, but just as pattern examples): try: from math import pi except ImportError: pi = 3.1415 Or: try: import fancymodule as mod except ImportError: import simpler_version as mod Or: try: from mymod import needed_func except ImportError: def needed_func(a,b,c): "Bare bones implementation of more general function" return a*b + c Or even: try: import cool_feature FEATURE_AVAIL = True except ImportError: FEATURE_AVAIL = False What these have in common is that they are each specific to the context, and actually have nothing much in common other than the possibility import might fail. The special case of defining a missing module as 'None' is something I will probably never want to do... and therefore definitely have no desire for special syntax to do it. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. On Fri, Dec 27, 2013 at 8:30 AM, David Mertz wrote: > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > (and of course, variants with from...import et cetera). These can > potentially add a lot of clutter to the imports section of a file, given > that it requires 4 lines to do a conditional import. > > It seems like it'd be useful and clean to have a syntax that looked like > this: > > maybe import foo > from bar maybe import baz > from qux maybe import quy as quz > > Where the behavior would essentially be as above - attempt to run the > import normally, and in cases where the import fails, map the name to a > value of None instead. Users who want a different behavior are still free > to use the long-form syntax. A possibly variant might be to also only run > the import if the name isn't already bound, so that you could do something > like... > > from frobber_a maybe import frob as frobber > from frobbler_b maybe import frobble as frobber > from frobber_c maybe import frobit as frobber > > ...to potentially try different fallback options if the first choice for > an interface provider isn't available. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From grosser.meister.morti at gmx.net Fri Dec 27 18:31:07 2013 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 27 Dec 2013 18:31:07 +0100 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> Message-ID: <52BDB95B.9050708@gmx.net> I often do this: try: import json except ImportError: import simplejson as json Or: try: import optional_feature except ImportError: HAS_OPTIONAL_FEATURE = False else: HAS_OPTIONAL_FEATURE = True I think these two cases are fairly common. So why not have something like this?: import json or simplejson as json On 12/27/2013 06:00 PM, David Mertz wrote: > On Thu, Dec 26, 2013 at 5:57 PM, Amber Yust > wrote: > > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > > I would rarely, if ever, follow this pattern, although I *would* do conditional import fairly often. E.g. these are > things I might do (well, not really these specific ones, but just as pattern examples): > > try: > from math import pi > except ImportError: > pi = 3.1415 > > Or: > > try: > import fancymodule as mod > except ImportError: > import simpler_version as mod > > Or: > > try: > from mymod import needed_func > except ImportError: > def needed_func(a,b,c): > "Bare bones implementation of more general function" > return a*b + c > > Or even: > > try: > import cool_feature > FEATURE_AVAIL = True > except ImportError: > FEATURE_AVAIL = False > > What these have in common is that they are each specific to the context, and actually have nothing much in common other > than the possibility import might fail. The special case of defining a missing module as 'None' is something I will > probably never want to do... and therefore definitely have no desire for special syntax to do it. > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > > On Fri, Dec 27, 2013 at 8:30 AM, David Mertz > wrote: > > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > (and of course, variants with from...import et cetera). These can potentially add a lot of clutter to the imports > section of a file, given that it requires 4 lines to do a conditional import. > > It seems like it'd be useful and clean to have a syntax that looked like this: > > maybe import foo > from bar maybe import baz > from qux maybe import quy as quz > > Where the behavior would essentially be as above - attempt to run the import normally, and in cases where the import > fails, map the name to a value of None instead. Users who want a different behavior are still free to use the > long-form syntax. A possibly variant might be to also only run the import if the name isn't already bound, so that > you could do something like... > > from frobber_a maybe import frob as frobber > from frobbler_b maybe import frobble as frobber > from frobber_c maybe import frobit as frobber > > ...to potentially try different fallback options if the first choice for an interface provider isn't available. > From rymg19 at gmail.com Fri Dec 27 18:43:42 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 27 Dec 2013 11:43:42 -0600 Subject: [Python-ideas] "maybe import"? In-Reply-To: <2119483115886762605@gmail297201516> References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> Message-ID: I like that. I usually end up doing that with ElementTree: try: from xml.etree import cElementTree as etree except: try: from lxml import etree except: from xml.etree import ElementTree as etree It would all be: import xml.etree.cElementTree or lxml.etree or xml.etree.ElementTree as etree On Thu, Dec 26, 2013 at 10:36 PM, Amber Yust wrote: > On Thu Dec 26 2013 at 7:40:19 PM, Bruce Leban wrote: > >> I think you mean do the import if the name is unbound *or bound to None.* Otherwise, >> it doesn't work in the example you gave. >> > > Yes, that is what I meant, sorry. Another option would be "not bound to > something that is not a module" - but I think "unbound or None" is probably > the most versatile option. > > On Thu Dec 26 2013 at 8:08:45 PM, Steven D'Aprano > wrote: > >> I'm not (yet) convinced of the need for this functionality, but if >> Python did gain this, I think I would prefer the colour of this >> bike-shed to be "perhaps import" rather than "maybe import". > > > Another option would be to re-use the 'or' keyword: > > from foo import bar or None > > Where the bit after the 'or' simply specifies a default value to assign if > an ImportError occurs. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Fri Dec 27 19:18:28 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 27 Dec 2013 19:18:28 +0100 Subject: [Python-ideas] Determine Windows version in platform module Message-ID: Today I was looking for a way to determine whether I was on Windows >= Vista and couldn't find anything so I came up with this [1]. That led me to think about platform module. Honestly I can't see the usefulness of it as it doesn't give you any usable API to figure out what platform version you're on. As such would something like [1] be acceptable for inclusion? [1] http://stackoverflow.com/questions/12471772/what-is-better-way-of-getting-windows-version-in-python/20804735#20804735 --- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Dec 27 19:38:53 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 27 Dec 2013 19:38:53 +0100 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: Message-ID: <52BDC93D.9070301@egenix.com> On 27.12.2013 19:18, Giampaolo Rodola' wrote: > Today I was looking for a way to determine whether I was on Windows >= > Vista and couldn't find anything so I came up with this [1]. > > That led me to think about platform module. > Honestly I can't see the usefulness of it as it doesn't give you any usable > API to figure out what platform version you're on. Have you had a look at the documentation ? http://docs.python.org/2.7/library/platform.html?highlight=platform#windows-platform -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From storchaka at gmail.com Fri Dec 27 19:58:19 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 27 Dec 2013 20:58:19 +0200 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: Message-ID: 27.12.13 20:18, Giampaolo Rodola' ???????(??): > Today I was looking for a way to determine whether I was on Windows >= > Vista and couldn't find anything so I came up with this [1]. > > That led me to think about platform module. > Honestly I can't see the usefulness of it as it doesn't give you any > usable API to figure out what platform version you're on. > As such would something like [1] be acceptable for inclusion? > > [1] > http://stackoverflow.com/questions/12471772/what-is-better-way-of-getting-windows-version-in-python/20804735#20804735 See also funny issue19143 (http://bugs.python.org/issue19143). From g.rodola at gmail.com Fri Dec 27 19:48:52 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 27 Dec 2013 19:48:52 +0100 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: <52BDC93D.9070301@egenix.com> References: <52BDC93D.9070301@egenix.com> Message-ID: On Fri, Dec 27, 2013 at 7:38 PM, M.-A. Lemburg wrote: > On 27.12.2013 19:18, Giampaolo Rodola' wrote: > > Today I was looking for a way to determine whether I was on Windows >= > > Vista and couldn't find anything so I came up with this [1]. > > > > That led me to think about platform module. > > Honestly I can't see the usefulness of it as it doesn't give you any > usable > > API to figure out what platform version you're on. > > Have you had a look at the documentation ? > > > http://docs.python.org/2.7/library/platform.html?highlight=platform#windows-platform Yes, on Windows XP I get a tuple like this: ('XP', '5.1.2600', 'SP3', 'Uniprocessor Free') On Windows 7: ('7', '6.1.7600', '', 'Multiprocessor Free') Neither of those are helpful to make assumptions such as "if I'm on Windows >= XP with SP3: do something". The real deal would be dealing with a tuple of integers in order to use comparison operators, similarly to sys.version_info: if sys.version_info >= (3, 0): # py3-specific code --- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Dec 27 21:59:48 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Dec 2013 12:59:48 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> Message-ID: On Dec 27, 2013, at 9:43, Ryan Gonzalez wrote: > I like that. I usually end up doing that with ElementTree: > > try: > from xml.etree import cElementTree as etree Python 3 has already fixed that. You just import ElementTree and you get the C-accelerated version. And the same is true for cPickle, etc. A new language feature that offers a different way to solve something that's already been solved better doesn't seem very useful. Of course I do write stuff like this when I'm trying to write code that works with 2.6+/3.2+, but a new language feature that's a syntax error in 2.x wouldn't help there either. > except: > try: > from lxml import etree > except: > from xml.etree import ElementTree as etree > > It would all be: > > import xml.etree.cElementTree or lxml.etree or xml.etree.ElementTree as etree > > > > On Thu, Dec 26, 2013 at 10:36 PM, Amber Yust wrote: >> On Thu Dec 26 2013 at 7:40:19 PM, Bruce Leban wrote: >>> I think you mean do the import if the name is unbound or bound to None. Otherwise, it doesn't work in the example you gave. >> >> Yes, that is what I meant, sorry. Another option would be "not bound to something that is not a module" - but I think "unbound or None" is probably the most versatile option. >> >> On Thu Dec 26 2013 at 8:08:45 PM, Steven D'Aprano wrote: >>> I'm not (yet) convinced of the need for this functionality, but if >>> Python did gain this, I think I would prefer the colour of this >>> bike-shed to be "perhaps import" rather than "maybe import". >> >> Another option would be to re-use the 'or' keyword: >> >> from foo import bar or None >> >> Where the bit after the 'or' simply specifies a default value to assign if an ImportError occurs. >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Ryan > When your hammer is C++, everything begins to look like a thumb. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Dec 27 22:09:21 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Dec 2013 13:09:21 -0800 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: On Dec 27, 2013, at 10:48, "Giampaolo Rodola'" wrote: > > On Fri, Dec 27, 2013 at 7:38 PM, M.-A. Lemburg wrote: >> On 27.12.2013 19:18, Giampaolo Rodola' wrote: >> > Today I was looking for a way to determine whether I was on Windows >= >> > Vista and couldn't find anything so I came up with this [1]. >> > >> > That led me to think about platform module. >> > Honestly I can't see the usefulness of it as it doesn't give you any usable >> > API to figure out what platform version you're on. >> >> Have you had a look at the documentation ? >> >> http://docs.python.org/2.7/library/platform.html?highlight=platform#windows-platform > > Yes, on Windows XP I get a tuple like this: > > ('XP', '5.1.2600', 'SP3', 'Uniprocessor Free') > > On Windows 7: > > ('7', '6.1.7600', '', 'Multiprocessor Free') > > Neither of those are helpful to make assumptions such as "if I'm on Windows >= XP with SP3: do something". > The real deal would be dealing with a tuple of integers in order to use comparison operators, similarly to sys.version_info: > > if sys.version_info >= (3, 0): > # py3-specific code > So you want to write: if platform.win_version_info >= (5, 1, 2600): Is that really much better than: if platform.win32_ver.version >= '5.1.2600': There are a few cases with older windows versions where the third component is 3 digits, and it's always possible that could be true again, but I don't think it affects anything you're likely to be checking for in 2014. If you want some different number... then what? It would be nice if Microsoft had a nice numbering scheme so Windows 7 was 7 rather than 6.1, 2003 wasn't sometimes 5.1 and sometimes 5.2, etc. But they don't. Meanwhile, Microsoft's suggested solution is that you not check for version numbers at runtime, but instead check for features. And there are good reasons for that--for example, XPSP3 N has exactly the same version numbers as XPSP3 normal, but if you try to use the new Windows Media APIs, you'll crash. > > --- Giampaolo > https://code.google.com/p/psutil/ > https://code.google.com/p/pyftpdlib/ > https://code.google.com/p/pysendfile/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From grosser.meister.morti at gmx.net Fri Dec 27 22:03:09 2013 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 27 Dec 2013 22:03:09 +0100 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: <52BDEB0D.1040106@gmx.net> On 12/27/2013 07:48 PM, Giampaolo Rodola' wrote: > > On Fri, Dec 27, 2013 at 7:38 PM, M.-A. Lemburg > wrote: > > On 27.12.2013 19:18, Giampaolo Rodola' wrote: > > Today I was looking for a way to determine whether I was on Windows >= > > Vista and couldn't find anything so I came up with this [1]. > > > > That led me to think about platform module. > > Honestly I can't see the usefulness of it as it doesn't give you any usable > > API to figure out what platform version you're on. > > Have you had a look at the documentation ? > > http://docs.python.org/2.7/library/platform.html?highlight=platform#windows-platform > > > Yes, on Windows XP I get a tuple like this: > > ('XP', '5.1.2600', 'SP3', 'Uniprocessor Free') > > On Windows 7: > > ('7', '6.1.7600', '', 'Multiprocessor Free') > > Neither of those are helpful to make assumptions such as "if I'm on Windows >= XP with SP3: do something". > The real deal would be dealing with a tuple of integers in order to use comparison operators, similarly to sys.version_info: > > if sys.version_info >= (3, 0): > # py3-specific code > > What about this?: if tuple(int(v) for v in platform.win32_ver[1].split(".")) >= (5,1,2600): ... From skip at pobox.com Fri Dec 27 22:36:56 2013 From: skip at pobox.com (Skip Montanaro) Date: Fri, 27 Dec 2013 15:36:56 -0600 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: On Fri, Dec 27, 2013 at 3:09 PM, Andrew Barnert wrote: > So you want to write: > > if platform.win_version_info >= (5, 1, 2600): > > Is that really much better than: > > if platform.win32_ver.version >= '5.1.2600': Once either of the first two numbers rolls over to two digits, the first comparison will be correct, the second will not: >>> x = '5.9.2600' >>> y = '5.10.2600' >>> y > x False Skip From cs at zip.com.au Sat Dec 28 01:15:42 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 28 Dec 2013 11:15:42 +1100 Subject: [Python-ideas] "maybe import"? In-Reply-To: <-7047415593530210756@gmail297201516> References: <-7047415593530210756@gmail297201516> Message-ID: <20131228001542.GA35838@cskk.homeip.net> On 27Dec2013 01:57, Amber Yust wrote: > It's a fairly standard pattern to see things like this: > > try: > import foo > except ImportError: > foo = None > > (and of course, variants with from...import et cetera). These can > potentially add a lot of clutter to the imports section of a file, given > that it requires 4 lines to do a conditional import. > > It seems like it'd be useful and clean to have a syntax that looked like > this: > > maybe import foo [...snip...] The problem here is that the only reason to import as above is to have access to the name "foo". What is you contingency plan for the failed import? How does the code cope without "foo". I'd rather have an ImportError at start than an arbitrarily delayed and harder to diagnose NameError much later. Because of this, I'd argue you should almost always want the except: clause if this is realistic, because you need to have a plan for the case of failure. I agree the try/except is wordy and possibly ugly, but I think it is usually needed when a failed import is not to be fatal. Cheers, -- Cameron Simpson Every particle continues in its state of rest or uniform motion in a straight line except insofar as it doesn't. - Sir Arther Eddington From abarnert at yahoo.com Sat Dec 28 04:26:44 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Dec 2013 19:26:44 -0800 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: <453B4FE1-833E-4EB5-BEE6-698E78781EA2@yahoo.com> On Dec 27, 2013, at 13:36, Skip Montanaro wrote: > On Fri, Dec 27, 2013 at 3:09 PM, Andrew Barnert wrote: >> So you want to write: >> >> if platform.win_version_info >= (5, 1, 2600): >> >> Is that really much better than: >> >> if platform.win32_ver.version >= '5.1.2600': > > Once either of the first two numbers rolls over to two digits, the > first comparison will be correct, the second will not: > >>>> x = '5.9.2600' >>>> y = '5.10.2600' >>>> y > x Sure, but in decades of Windows versions it's never come even close to rolling over. Other than them renaming NT 3.2 as 3.5, they've never even gotten past .3. So, do you really think you need to worry about that? It seems just as plausible that they'd completely change their versioning scheme as that they'd ever go to 6.10 under the current scheme. Plus, it would be about 15 years away at their historical version numbering rates. From guido at python.org Sat Dec 28 04:30:41 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Dec 2013 19:30:41 -0800 Subject: [Python-ideas] PEP draft - Composable futures for reactive programming In-Reply-To: References: Message-ID: Hi Sergii, I'm trying to give some constructive criticism here, please bear with me. The biggest issue perhaps seems to me that the unification between concurrent and threaded Futures still feels uncomfortable to me. A symptom is the completely different semantics of result() -- when the result isn't ready yet, this either raises an exception or blocks the current thread, and that makes reasoning about what will happen difficult. A lesser issue is naming -- I read some earlier example code you posted, and I couldn't understand it, because the names for the new operations you added are pretty arbitrary. Especially grating is your reusing some well-known names of built-in Python functions for different purposes, the worst offender being map(), but all() isn't so great either. In general your FutureBaseExt class (also an awkward name IMO) introduces a bunch of new functions with a wide variety of functionality that seems to have little logic to it. Why this set of functions and not another? A separate question is why the distinction between FutureBase and FutureBaseExt. It seems you copied some phrases from the asyncio docs or PEP 3156 -- e.g. add_done_callback() references call_soon(); this seems incorrect for threaded Futures. The definition of an Executor seems incomplete (SynchronousExecutor is referenced but not defined), and very vague -- I don't believe that making it just a callable suffices for the functionality. There is also a mention of global configuration of a default executor by assigning to config.Default.CALLBACK_EXECUTOR, which seems a bad idea -- I'm sure a lot of code will in practice depend on the choice of executor. Another issue: why the try_* methods? Finally, I'm not sure I am convinced by your motivation section. Or, at least, I'd like you to address how your proposal addresses each of the bullets in your motivation, with some examples. (I may have more, but at the current rate it would take me a day per paragraph, so I'll get to more later.) --Guido On Tue, Dec 24, 2013 at 8:56 AM, Sergii Mikhtoniuk wrote: > Thanks everyone for your feedback. > > Taking all your suggestions into account I have revised my proposal. > > In short, it?s now: > - defines separate Future classes for cooperative and multithreaded cases in > concurrent.futures package > - multithreaded implementation adds thread-safety to basic implementation, > so in cooperative concurrency case there is absolutely no overhead > - cooperative future?s interface is identical to asyncio.future > - asyncio.Future inherits from concurrent.futures.cooperative.Future adding > only methods specific to `yield from` > - adds common composition methods for futures (intended to replace and > enhance asyncio.wait/gather and concurrent.futures.wait) > > There?s still some work to be done for backward compatibility of > concurrent.futures.Future, but implementation is almost ready. > > Would really appreciate if you could take a look. > > > Thanks, > Sergii > > > > On Mon, Dec 23, 2013 at 12:42 AM, Guido van Rossum wrote: >> >> Aha. That is clever. I will have to look into the details more, but the >> idea is promising. Sorry I didn't see tty his before. >> >> On Dec 22, 2013 12:30 PM, "Ben Darnell" wrote: >>> >>> On Sat, Dec 21, 2013 at 10:45 PM, Guido van Rossum >>> wrote: >>>> >>>> There's still the issue that in the threading version, you wait for a >>>> Future by blocking the current thread, while in the asyncio version, >>>> you must use "yield from" to block. For interoperability you would >>>> have to refrain from *any* blocking operations (including "yield >>>> from") so you would only be able to use callbacks. But whether you had >>>> to write "x = f.result()" or "x = concurrent.futures.wait_for(f)", >>>> either way you'd implicitly be blocking the current thread. >>> >>> >>> Threaded *consumers* of Futures wait for them by blocking, while >>> asynchronous consumers wait for them by yielding. It doesn't matter whether >>> the *producer* of the Future is threaded or asynchronous (except that if you >>> know you won't be using threads you can use a faster thread-unsafe Future >>> implementation). >>> >>> -Ben >>> >>>> >>>> >>>> Yes, a clever scheduler could run other callbacks while blocking, but >>>> that's not a complete solution, because another callback might do a >>>> similar blocking operation, and whatever that waited for could hold up >>>> the earlier blocking operation, causing an ever-deeper recursion and >>>> of event loop invocations that might never complete. (I've had to >>>> debug this in production code.) To cut through that you'd have to have >>>> some kind of stack-swapping coroutine implementation like gevent, or a >>>> syntactic preprocessor that inserts yield or yield-from operations >>>> (I've heard from people who do this), or you'd need a clairvoyant >>>> scheduler that would know which callbacks won't block. >>>> >>>> I like the C# solution, but it depends on static typing so a compiler >>>> can know when to emit the coroutine interactions. That wouldn't work >>>> in Python, unless you made the compiler recognizing the wait_for() >>>> operation by name, which feels unsavory (although we do it for super() >>>> :-). >>>> >>>> I guess for extreme interop, callbacks that never block is your only >>>> option anyway, but I'd be sad if we had to to recommend this as the >>>> preferred paradigm, or claim that it is all you need. >>>> >>>> --Guido (if I don't respond to this thread for the next two weeks, >>>> it's because I'm on vacation :-) >>>> >>>> >>>> On Sat, Dec 21, 2013 at 5:37 PM, Ben Darnell wrote: >>>> > On Sat, Dec 21, 2013 at 7:26 PM, Guido van Rossum >>>> > wrote: >>>> >> >>>> >> On Sat, Dec 21, 2013 at 2:48 PM, Sergii Mikhtoniuk >>>> >> >>>> >> wrote: >>>> >> > Indeed there is a lot of overlap between asyncio and >>>> >> > concurrent.futures >>>> >> > packages, so it would be very interesting to hear your overall >>>> >> > thoughts >>>> >> > on >>>> >> > role/future of concurrent package. Do you consider it rudimentary >>>> >> > and >>>> >> > replaceable by asyncio.Future completely? >>>> >> >>>> >> They don't really compare. concurrent.futures is about *threads*. >>>> >> asyncio.Future is about *avoiding* threads in favor of more >>>> >> lightweight "tasks" and "coroutines", which in turn are built on top >>>> >> of lower-level callbacks. >>>> > >>>> > >>>> > concurrent.futures.ThreadPoolExecutor is about threads; the Future >>>> > class >>>> > itself is broader. When I integrated Futures into Tornado I used >>>> > concurrent.futures.Future directly (when available). asyncio.Future >>>> > is just >>>> > an optimized version of c.f.Future (the optimization comes from >>>> > assuming >>>> > single-threaded usage). There should at least be a common ABC between >>>> > them. >>>> > >>>> >> >>>> >> >>>> >> (While I want to get away from callbacks as a programming paradigm, >>>> >> asyncio uses them at the lower levels both because they are a logical >>>> >> low-level building block and for interoperability with other >>>> >> frameworks like Tornado and Twisted.) >>>> >> >>>> >> > I think the question is even not much about which is your preferred >>>> >> > implementation, but rather do we want having futures as stand-alone >>>> >> > package >>>> >> > or not. Do you see all these implementations converging in future? >>>> >> >>>> >> I see them as not even competing. They use different paradigms and >>>> >> apply to different use cases. >>>> >> >>>> >> > To me Future is a very simple and self-contained primitive, >>>> >> > independent >>>> >> > of >>>> >> > thread pools, processes, IO, and networking. >>>> >> >>>> >> (Agreed on the I/O and networking part only.) >>>> >> >>>> >> > One thing concurrent.futures >>>> >> > package does a very good job at is defining an isolated namespace >>>> >> > for >>>> >> > futures, stressing out this clear boundary (let?s disregard here >>>> >> > ThreadPoolExecutor and ProcessPoolExecutor classes which for some >>>> >> > reason >>>> >> > ended up in it too). >>>> >> >>>> >> Actually the executor API is an important and integral part of that >>>> >> package, and threads underlie everything you can do with its Futures. >>>> >> >>>> >> > So when I think of schedulers and event loops implementations I see >>>> >> > them >>>> >> > as >>>> >> > ones that build on top of the Future primitive, not providing it as >>>> >> > part >>>> >> > of >>>> >> > their implementation. What I think is important is that having >>>> >> > unified >>>> >> > Future class simplifies interoperability between different kinds of >>>> >> > schedulers, not necessarily associated with asyncio event loops >>>> >> > (process >>>> >> > pools for example). >>>> >> >>>> >> The interoperability is completely missing. I can't tell if you've >>>> >> used asyncio at all, but the important operation of *waiting* for a >>>> >> result is fundamentally different there than in concurrent.futures. >>>> >> In >>>> >> the latter, you just write "x = f.result()" and your thread blocks >>>> >> until the result is available. In asyncio, you have to write "x = >>>> >> yield from f.result()" which is a coroutine block that lets other >>>> >> tasks run in the same thread. ("yield from" in this case is how >>>> >> Python >>>> >> spells the operation that C# calls "await"). >>>> > >>>> > >>>> > The way I see it, the fundamental operation on Futures is >>>> > add_done_callback. >>>> > We then have various higher-level operations that let us get away from >>>> > using >>>> > callbacks directly. One of these happens to be a method on Future: >>>> > the >>>> > blocking mode of Future.result(). Another is implemented in asyncio >>>> > and >>>> > Tornado, in the ability to "yield" a Future. The blocking mode of >>>> > result() >>>> > is just a convenience; if asyncio-style futures had existed first then >>>> > we >>>> > could instead have a function like "x = >>>> > concurrent.futures.wait_for(f)". In >>>> > fact, you could write this wait_for function today in a way that works >>>> > for >>>> > both concurrent and asyncio futures. >>>> > >>>> > This is already interoperable: Tornado's Resolver interface >>>> > >>>> > (https://github.com/facebook/tornado/blob/master/tornado/netutil.py#L184) >>>> > returns a Future, which may be generated by a ThreadPoolExecutor or an >>>> > asynchronous wrapper around pycares or twisted. In the other >>>> > direction I've >>>> > worked on hybrid apps that have one Tornado thread alongside a bunch >>>> > of >>>> > Django threads; in these apps it would work to have a Django thread >>>> > block on >>>> > f.result() for a Future returned by Tornado's AsyncHTTPClient. >>>> > >>>> > -Ben >>>> > >>>> >> >>>> >> > Futures are definitely not the final solution for concurrency >>>> >> > problem >>>> >> > but >>>> >> > rather a well-established utility for representing async >>>> >> > operations. It >>>> >> > is a >>>> >> > job of higher layer systems to provide more convenient ways for >>>> >> > dealing >>>> >> > with >>>> >> > asyncs (yield from coroutines, async/await rewrites etc.), >>>> >> >>>> >> But unless you are proposing some kind of radical change to add >>>> >> compile-time type checking/inference to Python, the rewrite option is >>>> >> unavailable in Python. >>>> >> >>>> >> > so I would not >>>> >> > say that futures encourage callback-style programming, >>>> >> >>>> >> The concurrent.futures.Future class does not. But unless I misread >>>> >> your proposal, your extensions do. >>>> >> >>>> >> > it?s simply a >>>> >> > lower-layer functionality. On the contrary, monadic >>>> >> >>>> >> (Say that word one more time and everyone tunes out. :-) >>>> >> >>>> >> > methods for futures >>>> >> > composition (e.g. map(), all(), first() etc.) ensure that no errors >>>> >> > would be >>>> >> > lost in the process, so I think they would complement yield from >>>> >> > model >>>> >> > quite >>>> >> > nicely by hiding complexity of state maintenance from user and >>>> >> > reducing >>>> >> > the >>>> >> > number of back-and-forth communications between event loop and >>>> >> > coroutines. >>>> >> >>>> >> I'm not sure I follow. Again, I'm not sure if you've actually written >>>> >> any code using asyncio. >>>> >> >>>> >> TBH I've written a fair number of example programs for asyncio and >>>> >> I've very rarely felt the need for these composition functions. The >>>> >> main composition primitive I tend to use is "yield from". >>>> >> >>>> >> > Besides Futures, reactive programming >>>> >> >>>> >> What *is* reactive programming? If you're talking about >>>> >> http://en.wikipedia.org/wiki/Reactive_programming, >>>> >> I'm not sure that it maps well to Python. >>>> >> >>>> >> > has more utilities to offer, such as >>>> >> > Observables (representing asynchronous streams of values). It is >>>> >> > also a >>>> >> > very >>>> >> > useful abstraction with a rich set of composition strategies >>>> >> > (merging, >>>> >> > concatenation, grouping), and may deserve its place in separate >>>> >> > package. >>>> >> >>>> >> It all sounds very abstract and academic. :-) >>>> >> >>>> >> > Hope to hear back from you to get better picture on overall design >>>> >> > direction >>>> >> > here before jumping to implementation details. >>>> >> > >>>> >> > Brief follow-up to your questions: >>>> >> > - Idea behind the Future/Promise separation is to draw a clean >>>> >> > line >>>> >> > between >>>> >> > client-facing and scheduler-side APIs respectively. Futures should >>>> >> > not >>>> >> > expose any completion methods, which clients should not call >>>> >> > anyway. >>>> >> >>>> >> Given that Future and Promise are often used synonymously, using them >>>> >> to make this distinction sounds confusing. I agree that Futures have >>>> >> two different APIs, one for the consumer and another for the >>>> >> producer. >>>> >> But I'm not sure that it's necessary to separate them more strictly >>>> >> -- >>>> >> convention seems good enough to me here. (It's the same with many >>>> >> communication primitives, like queues and even threads.) >>>> >> >>>> >> (The one thing that trips people up frequently is that, while >>>> >> set_result() and set_exception() are producer APIs, cancel() is a >>>> >> consumer API, and the cancellation signal travels from the consumer >>>> >> to >>>> >> the producer.) >>>> >> >>>> >> > - Completely agree with you that Twisted-style callbacks are evil >>>> >> > and >>>> >> > it is >>>> >> > better to have single code path for getting result or raising >>>> >> > exception >>>> >> > - Sorry for bad grammar in the proposal, it?s an early draft >>>> >> > written in >>>> >> > 3 >>>> >> > AM, so I will definitely improve on it if we decide to move >>>> >> > forward. >>>> >> >>>> >> No problem! >>>> >> >>>> >> > Thanks, >>>> >> > Sergii >>>> >> >>>> >> -- >>>> >> --Guido van Rossum (python.org/~guido) >>>> >> _______________________________________________ >>>> >> Python-ideas mailing list >>>> >> Python-ideas at python.org >>>> >> https://mail.python.org/mailman/listinfo/python-ideas >>>> >> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> > >>>> > >>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>> >>> > -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Sat Dec 28 04:38:09 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Dec 2013 19:38:09 -0800 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: <2F2A1829-56A7-4571-857C-E6010CE5C58B@yahoo.com> On Dec 27, 2013, at 13:09, Andrew Barnert wrote: > Meanwhile, Microsoft's suggested solution is that you not check for version numbers at runtime, but instead check for features. And there are good reasons for that--for example, XPSP3 N has exactly the same version numbers as XPSP3 normal, but if you try to use the new Windows Media APIs, you'll crash. Also, in forgot to mention that there are features added to Windows versions discontinuously. For example, there are features that are in XPSP3 (5.1.2600) and 2003R2SP2 (5.2.3790) but not original 2003 (5.2.3376). So, a check for >= 5, 1, 2600 would give you the wrong information anyway. And that's not even considering the fact that some kernels are used for multiple different Windows versions with different features (e.g., that 5.2.3790 is also the kernel for 64-bit XP). To make things even more fun, 2003SP2 originally had the same 3376 kernel as stock 2003, then got bumped to the same 3790 kernel as 2003R2SP2 in a non-service-pack update. These aren't anomalies; this is the way Windows versioning works. The kernel build number is useless as an indication of anything but kernel features that Python code isn't going to care about; the major.minor can be useful in a few cases, but usually even that doesn't tell you as much as you probably expect. Any change to make it easier to write misleadingly broken code is probably not a good change. From steve at pearwood.info Sat Dec 28 13:27:00 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 28 Dec 2013 23:27:00 +1100 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> Message-ID: <20131228122700.GV29356@ando> On Fri, Dec 27, 2013 at 12:59:48PM -0800, Andrew Barnert wrote: > On Dec 27, 2013, at 9:43, Ryan Gonzalez wrote: > > > I like that. I usually end up doing that with ElementTree: > > > > try: > > from xml.etree import cElementTree as etree > > Python 3 has already fixed that. You just import ElementTree and you > get the C-accelerated version. > > And the same is true for cPickle, etc. Not if the C-accelerated version isn't available. Or perhaps there are two pure-Python modules with the same API, and you're not sure which is installed. This issue is bigger than just C-accelerators in the CPython standard library. It applies any time you have two or more alternatives with the same API, a preferred module and an fallback module. E.g. readline in the stdlib, versus pyreadline, a third-party Windows port. A point against this suggestion: the best syntactic sugar opens up new and successful idioms. List comps and decorators have been great successes. Being able to embed a list comp as an expression, instead of a for-loop as a statement, is a great win. This new syntax doesn't open up new idioms for writing code. It is purely sugar, and as such, it isn't such a compelling feature. On the other hand, I think it does simplify a very common use-case. I often use fallback modules with the same API, and if only this new syntax had been introduced back in Python 2.4 or 2.5 I would be using it all the time now. Unfortunately, most (but not all) of my code has to support 2.4 or 2.5, or at least 3.3, so the usefulness of new syntax is severely limited *right now*. But if this syntax were introduced into Python 3.5 (its surely too late for 3.4) I reckon I would be able to use it around Python 3.6 or 3.7. And of course those who won't need to support versions of Python prior to 3.5 could use it straight away. So don't think of this as something you are going to use. Think of this as an investment for future Python users who are starting with 3.5 and don't care about backwards compatibility with Python 3.3 and 3.4 (let alone older versions). On that basis, I'm going to vote: +1 on "import preferred or fallback as name". +0 on allowing "import module or None", to cover the idiom: try: import module except ImportError: module = None which may not be encouraged, but it is occasionally useful. -0 on "from preferred or fallback import name", simply because I'm not sure how to extend it to multiple names, so I'd rather leave it out. -1 on Amber's original suggestion "maybe import name", as the benefit is too little to justify introducing a new keyword. -- Steven From ncoghlan at gmail.com Sat Dec 28 14:30:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 23:30:25 +1000 Subject: [Python-ideas] "maybe import"? In-Reply-To: <20131228122700.GV29356@ando> References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> <20131228122700.GV29356@ando> Message-ID: On 28 December 2013 22:27, Steven D'Aprano wrote: > So don't think of this as something you are going to use. Think of this > as an investment for future Python users who are starting with 3.5 and > don't care about backwards compatibility with Python 3.3 and 3.4 (let > alone older versions). > > On that basis, I'm going to vote: > > +1 on "import preferred or fallback as name". > > +0 on allowing "import module or None", to cover the idiom: > > try: > import module > except ImportError: > module = None > > which may not be encouraged, but it is occasionally useful. > > -0 on "from preferred or fallback import name", simply because I'm not > sure how to extend it to multiple names, so I'd rather leave it out. > > -1 on Amber's original suggestion "maybe import name", as the > benefit is too little to justify introducing a new keyword. I used to be a fan of the "or" based variants, but I eventually went off the idea. My reasoning went as follows: 1. For a single file script, you're only doing the import once, so there's no repetition, and hence little to be gained from the more concise syntax (and the cost of a new thing for readers to learn) 2. For a multi file package or application, it's trivial to factor out the compatibility checking to a helper module, so there's *still* no need for repetition. A compatibility module also lets you build an adaptation layer if the APIs aren't *quite* the same (for example, RHEL5 has the ancient pre-simplejson JSON module, so naive "import json" code will mistakenly think it has the 2.6+ stdlib version - a helper module can detect oddball discrepancies like that one and work around them, but naive syntax would do the wrong thing if a similar situation arose again in the future). Essentially, the niche that the syntax variants can cover is too small to be worth making the already horrendously complex import statement syntax and semantics even *more* complicated, when you can either spell it out explicitly in the single file case, or use a helper module to centralise the selection logic in the multiple file case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From denis.spir at gmail.com Sat Dec 28 15:59:52 2013 From: denis.spir at gmail.com (spir) Date: Sat, 28 Dec 2013 15:59:52 +0100 Subject: [Python-ideas] "maybe import"? In-Reply-To: <20131227040808.GP29356@ando> References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> Message-ID: <52BEE768.7060401@gmail.com> On 12/27/2013 05:08 AM, Steven D'Aprano wrote: > Hmmm. The basic idea makes a certain level of sense to me, but I'm not > sure it makes enough sense to overcome the barrier required before > adding a new keyword. Same for me. In fact, while the use case is clear and common, I'm not sure syntax it's worth dedicated syntax at all, especially because this happens once in a module and at a definite place (import section). > [...] > > If we're entertaining changes to imports, another possibility would be > to allow fallback module names: > > import this or that or another as this I like that form. > Each of "this", "that", "another" will be attempted, the first > successful import being bound to the name "this". The "as this" part > would be mandatory, so as to require a consistent name regardless of > which module was imported. This would be a syntax error, since it isn't > clear what name would be bound at the end: > > import this or that or another Right. > This would also be allowed: > > from this or that or another import spam We'd also, or rather, need: from this import foo or that import bar or another import baz as spam which starts to be a bit complicated... Maybe (or perhaps) with parens: from ((this import foo) or (that import bar) or (another import baz)) as spam ? > With this syntax, we could add None as a special case: > > import this or that or another or None as this Yep! > would be equivalent to: > > module_names = ("this", "that", "another", "None") > for name in module_names: > if name == "None": > spam = None > else: > try: > this = __import__(name) > except ImportError: > continue > break > else: > raise ImportError Seems correct, expected semantics. > and the "from...import" case could be written as: > > from this or that or None import spam > > > roughly equivalent to: > > module_names = ("this", "that", "another", "None") > for name in module_names: > if name == "None": > spam = None > else: > try: > temp = __import__(name) > spam = temp.spam > except ImportError: > continue > break > else: > raise ImportError Right. > Advantages: > > - covers both use-cases where you want to try a series of > modules, and the one where you fall back to None; Plus, as shown above we need to cover the case were a imported element have various names in original module. (Either all cases are covered, or it's definitely not worth bothering, imo.) > - "or" is already a keyword, no new keywords needed; > > - reads more like English; Not clearer than "maybe", in my ears (sic!) but I'm not english native speaker. > - "import None" currently gives SyntaxError, so this > can't interfere with modules actually called "None". > > > Disadvantages: > > - more complexity to imports; > > - only saves a few lines; > > - this usage of "or" is not quite the same as the usage > as a boolean operator, e.g. different from "x in a or b". Yes, it's more like prioritized choice in parsing (an operand in either a symbol or a real or an int), which is at times termed 'or' as well. Denis From rosuav at gmail.com Sat Dec 28 17:10:10 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Dec 2013 03:10:10 +1100 Subject: [Python-ideas] "maybe import"? In-Reply-To: <52BEE768.7060401@gmail.com> References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: On Sun, Dec 29, 2013 at 1:59 AM, spir wrote: >> This would also be allowed: >> >> from this or that or another import spam > > > We'd also, or rather, need: > > from this import foo or that import bar or another import baz as > spam > > which starts to be a bit complicated... Maybe (or perhaps) with parens: > > from ((this import foo) or (that import bar) or (another import > baz)) as spam Can't say I like that variant - getting complicated and hard to read, even with parens. If you're importing different names from different modules, it's going to get very long, and then people are going to want to split it across lines, and then the obvious question is: what's been gained over the try/except variant? There is one thing, though, that I'm seeing of all this. Exception throwing is asymmetrical: you can attempt a series of statements until one fails, but there's no convenient syntax to attempt a series of statements until one succeeds. I wonder, could the more general case be solved? Is there a way to, without stupid stuff like eval, wrap up a few statements so they can be executed in a loop: def import_any(statement_list): for try_me in statement_list: try: # uhh, this is the bit I'm not sure about... try_me() # this would work if they're functions instead! return except ImportError: pass raise ImportError This works for a set of functions, but not for a bunch of "from this import that" statements. Would it be worth mangling the top of your script until it can be done with importlib or __import__? Doesn't seem nearly as clean, somehow. ChrisA From g.rodola at gmail.com Sat Dec 28 17:12:57 2013 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 28 Dec 2013 17:12:57 +0100 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: > Meanwhile, Microsoft's suggested solution is that you not check for > version numbers at runtime, but instead check for features. And there are > good reasons for that--for example, XPSP3 N has exactly the same version > numbers as XPSP3 normal, but if you try to use the new Windows Media APIs, > you'll crash. > Mmm. What's weird is that if you use MSDN as a reference every Windows API has a "minimum supported client" section near the bottom of the page. See for example: http://msdn.microsoft.com/en-us/library/windows/desktop/ms684874(v=vs.85).aspx To me that suggests that (most of the times at least) you are encouraged to use the windows version to decide whether to support a feature or not. --- Giampaolo https://code.google.com/p/psutil/ https://code.google.com/p/pyftpdlib/ https://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From amber.yust at gmail.com Sat Dec 28 20:14:28 2013 From: amber.yust at gmail.com (Amber Yust) Date: Sat, 28 Dec 2013 19:14:28 +0000 Subject: [Python-ideas] "maybe import"? References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: <360207118627343726@gmail297201516> I'm fairly convinced by the "too small" reasoning (summed up well by Nick). I could potentially see a small and not incredibly intrusive improvement of allowing imports from a tuple-style group of alternatives being useful for things like c-accelerators, e.g... import (accelerated_foo, alternative_foo) as foo from (accelerated_foo, alternative_foo) import bar But that would be a "nice to have" thing rather than a big difference. On Sat Dec 28 2013 at 8:10:47 AM, Chris Angelico wrote: > On Sun, Dec 29, 2013 at 1:59 AM, spir wrote: > >> This would also be allowed: > >> > >> from this or that or another import spam > > > > > > We'd also, or rather, need: > > > > from this import foo or that import bar or another import baz as > > spam > > > > which starts to be a bit complicated... Maybe (or perhaps) with parens: > > > > from ((this import foo) or (that import bar) or (another import > > baz)) as spam > > Can't say I like that variant - getting complicated and hard to read, > even with parens. If you're importing different names from different > modules, it's going to get very long, and then people are going to > want to split it across lines, and then the obvious question is: > what's been gained over the try/except variant? > > There is one thing, though, that I'm seeing of all this. Exception > throwing is asymmetrical: you can attempt a series of statements until > one fails, but there's no convenient syntax to attempt a series of > statements until one succeeds. I wonder, could the more general case > be solved? Is there a way to, without stupid stuff like eval, wrap up > a few statements so they can be executed in a loop: > > def import_any(statement_list): > for try_me in statement_list: > try: > # uhh, this is the bit I'm not sure about... > try_me() # this would work if they're functions instead! > return > except ImportError: > pass > raise ImportError > > This works for a set of functions, but not for a bunch of "from this > import that" statements. Would it be worth mangling the top of your > script until it can be done with importlib or __import__? Doesn't seem > nearly as clean, somehow. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Dec 28 20:46:56 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Dec 2013 11:46:56 -0800 Subject: [Python-ideas] Determine Windows version in platform module In-Reply-To: References: <52BDC93D.9070301@egenix.com> Message-ID: On Dec 28, 2013, at 8:12, "Giampaolo Rodola'" wrote: > >> Meanwhile, Microsoft's suggested solution is that you not check for version numbers at runtime, but instead check for features. And there are good reasons for that--for example, XPSP3 N has exactly the same version numbers as XPSP3 normal, but if you try to use the new Windows Media APIs, you'll crash. > > Mmm. What's weird is that if you use MSDN as a reference every Windows API has a "minimum supported client" section near the bottom of the page. > See for example: > http://msdn.microsoft.com/en-us/library/windows/desktop/ms684874(v=vs.85).aspx > To me that suggests that (most of the times at least) you are encouraged to use the windows version to decide whether to support a feature or not. Those versions don't correspond to any unique kernel version number, so you _can't_ check for it that way. What you _can_ do with that information is write your app's version requirements, so your end users know not to buy your all if they have xpsp1. http://msdn.microsoft.com/en-us/library/windows/desktop/ms724832(v=vs.85).aspx is one of the places they recommend against using version numbers as a substitute for feature test. it says: way to determine whether a particular operating system feature is present. This is because the operating system may have had new features added in a redistributable DLL. Rather than using the Version API Helper functions to determine the operating system platform or version number, test for the presence of the feature itself. To determine the best way to test for a feature, refer to the documentation for the feature of interest. The following list discusses some common techniques for feature detection: You can test for the presence of the functions associated with a feature. To test for the presence of a function in a system DLL, call the LoadLibrary function to load the DLL. Then call the GetProcAddress function to determine whether the function of interest is present in the DLL. Use the pointer returned by GetProcAddress to call the function. Note that even if the function is present, it may be a stub that just returns an error code such as ERROR_CALL_NOT_IMPLEMENTED. You can determine the presence of some features by using the GetSystemMetrics function. For example, you can detect multiple display monitors by callingGetSystemMetrics(SM_CMONITORS). There are several versions of the redistributable DLLs that implement shell and common control features. For information about determining which versions are present on the system your application is running on, see the topicShell and Common Controls Versions. If you must require a particular operating system, be sure to use it as a minimum supported version, rather than design the test for the one operating system. This way, your detection code will continue to work on future versions of Windows. Note that a 32-bit application can detect whether it is running under WOW64 by calling the IsWow64Process function. It can obtain additional processor information by calling theGetNativeSystemInfo function. > > > --- Giampaolo > https://code.google.com/p/psutil/ > https://code.google.com/p/pyftpdlib/ > https://code.google.com/p/pysendfile/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Dec 28 20:55:19 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Dec 2013 11:55:19 -0800 Subject: [Python-ideas] "maybe import"? In-Reply-To: <20131228122700.GV29356@ando> References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> <20131228122700.GV29356@ando> Message-ID: On Dec 28, 2013, at 4:27, Steven D'Aprano wrote: > On Fri, Dec 27, 2013 at 12:59:48PM -0800, Andrew Barnert wrote: >> On Dec 27, 2013, at 9:43, Ryan Gonzalez wrote: >> >>> I like that. I usually end up doing that with ElementTree: >>> >>> try: >>> from xml.etree import cElementTree as etree >> >> Python 3 has already fixed that. You just import ElementTree and you >> get the C-accelerated version. >> >> And the same is true for cPickle, etc. > > > Not if the C-accelerated version isn't available. Right, in which case you get the pure Python version--exactly as if you'd done the try/import/except/import, or used this proposed new feature. So there is no advantage to doing the more complicated thing, except 2.x compatibility--which you obviously wouldn't get with a new language feature. > Or perhaps there are > two pure-Python modules with the same API, and you're not sure which is > installed. This issue is bigger than just C-accelerators in the CPython > standard library. It applies any time you have two or more alternatives > with the same API, a preferred module and an fallback module. E.g. > readline in the stdlib, versus pyreadline, a third-party Windows port. There are maybe three such examples that people use regularly. Is that enough to warrant a change in the language? Well, that's not quite true. It's very common for modules that were renamed in 3.0, or that were recently added but have backward compat ports on PyPI. But again, a language change won't help those cases anyway. Unless you're expecting another mass renaming in 4.0? From abarnert at yahoo.com Sat Dec 28 21:05:28 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Dec 2013 12:05:28 -0800 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: On Dec 28, 2013, at 8:10, Chris Angelico wrote: > There is one thing, though, that I'm seeing of all this. Exception > throwing is asymmetrical: you can attempt a series of statements until > one fails, but there's no convenient syntax to attempt a series of > statements until one succeeds. I wonder, could the more general case > be solved? Is there a way to, without stupid stuff like eval, wrap up > a few statements so they can be executed in a loop: > > def import_any(statement_list): > for try_me in statement_list: > try: > # uhh, this is the bit I'm not sure about... > try_me() # this would work if they're functions instead! > return > except ImportError: > pass > raise ImportError > > This works for a set of functions, but not for a bunch of "from this > import that" statements. Would it be worth mangling the top of your > script until it can be done with importlib or __import__? Doesn't seem > nearly as clean, somehow. This is a clever idea. If you could work out a clean syntax for the "try until one works" syntax, it would solve a wider range of problems than just this import issue. It strikes me that there's some similarity with if/elif/else. Maybe just: try: stuff except try: other stuff except try: different stuff except Exception as e: deal with it But there are a lot of open questions. First, "except try" looks horrible, but I can't think of anything better. If they all fail, do you only get the last exception, or are they all chained together? Would there be a use for a sequence like this without except on the end (so if they all fail, it just raises)? Can you put an exception type or tuple between the "except try"? What about an as clause (so the block could use it)? From steve at pearwood.info Sun Dec 29 00:43:23 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Dec 2013 10:43:23 +1100 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> <20131228122700.GV29356@ando> Message-ID: <20131228234322.GW29356@ando> On Sat, Dec 28, 2013 at 11:55:19AM -0800, Andrew Barnert wrote: > On Dec 28, 2013, at 4:27, Steven D'Aprano wrote: > > Or perhaps there are > > two pure-Python modules with the same API, and you're not sure which is > > installed. This issue is bigger than just C-accelerators in the CPython > > standard library. It applies any time you have two or more alternatives > > with the same API, a preferred module and an fallback module. E.g. > > readline in the stdlib, versus pyreadline, a third-party Windows port. > > There are maybe three such examples that people use regularly. Is that > enough to warrant a change in the language? There's only one *common* example of augmented assignment, x += 1, but that was enough to justify a whole raft of new syntax, += -= *= |= etc because the *idiom* of "x = x op arg" is common. What matters is not the specific examples of which modules get used in this way, but that the idiom of try: import this except ImportError: import that as this is common, not the specific values of this/that. I do not expect that the idiom of having a fallback module when the preferred module isn't available will ever disappear, simply because there are too many circumstances in which it comes up. Having said that, Nick's comments strike me as critical. "import this or that" doesn't buy us anything that can't already be solved in other ways. If it were easy to add, then perhaps it would be justified. But the implementation of import is already gnarly and hard to maintain (so I'm told), so adding more complexity to it for something which is purely syntactic sugar is probably not the best thing to do. Maybe in Python 4000 :-) -- Steven From bruce at leapyear.org Sun Dec 29 05:39:50 2013 From: bruce at leapyear.org (Bruce Leban) Date: Sat, 28 Dec 2013 20:39:50 -0800 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: On Sat, Dec 28, 2013 at 12:05 PM, Andrew Barnert wrote: > Would there be a use for a sequence like this without except on the end > (so if they all fail, it just raises)? > I think what you're talking about is just leaving the try off the last clause (as in the example below). > Can you put an exception type or tuple between the "except try"? What > about an as clause (so the block could use it)? > I started writing a mail with exactly this subject line and discarded it because of the latter two issues. In code I've recently written I have a chain of three and while the indentation is ugly, it's not so ugly that it cries out as a must fix language feature. And the problem is that I really want to limit to specific exceptions. In one case I have something like this: try: .... except KeyError: try: .... except AttributeError: try: .... except IndexError: .... and if I can't specify the specific exception, this is not helpful. On the other hand try: .... except KeyError try: .... except AttributeError try: .... except IndexError: .... only really saves me two colons and some whitespace. If the language had this feature, I'd use it, but that doesn't make it worth adding. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Dec 29 10:07:10 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Dec 2013 01:07:10 -0800 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: <2C9DDD38-F5E1-44E5-91C0-949A1FC0752A@yahoo.com> On Dec 28, 2013, at 20:39, Bruce Leban wrote: > > On Sat, Dec 28, 2013 at 12:05 PM, Andrew Barnert wrote: >> Can you put an exception type or tuple between the "except try"? What about an as clause (so the block could use it)? > > I started writing a mail with exactly this subject line and discarded it because of the latter two issues. The second one is pretty minor; it's just a matter of making the syntax clean, and I'm sure someone would come up with something if it were worth doing. The part that worries me about the first issue is that there seem to be two reasonable ways to read it. In your example below, if the first attempt raises an AttributeError, should it go to the "except AttributeError try:" block, or should that only happen if the second attempt raises one? The first variation means it's _not_ just syntactic sugar for the overly indented form you gave before it. But that also means (I think) it requires a change to the way exceptions are handled. (I'd have to sit down and work out the bytecode to be sure about that.) And, more importantly, I don't think the meaning would be obvious to a reader--the fact that you assumed the other meaning was the only one possible is pretty strong evidence for that. > In code I've recently written I have a chain of three and while the indentation is ugly, it's not so ugly that it cries out as a must fix language feature. And the problem is that I really want to limit to specific exceptions. In one case I have something like this: > > try: > .... > except KeyError: > try: > .... > except AttributeError: > try: > .... > except IndexError: > .... > > and if I can't specify the specific exception, this is not helpful. On the other hand > > try: > .... > except KeyError try: > .... > except AttributeError try: > .... > except IndexError: > .... > > only really saves me two colons and some whitespace. If the language had this feature, I'd use it, but that doesn't make it worth adding. > > --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From denis.spir at gmail.com Sun Dec 29 10:17:00 2013 From: denis.spir at gmail.com (spir) Date: Sun, 29 Dec 2013 10:17:00 +0100 Subject: [Python-ideas] "maybe import"? In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: <52BFE88C.7080407@gmail.com> On 12/28/2013 05:10 PM, Chris Angelico wrote: > On Sun, Dec 29, 2013 at 1:59 AM, spir wrote: >>> This would also be allowed: >>> >>> from this or that or another import spam >> >> >> We'd also, or rather, need: >> >> from this import foo or that import bar or another import baz as >> spam >> >> which starts to be a bit complicated... Maybe (or perhaps) with parens: >> >> from ((this import foo) or (that import bar) or (another import >> baz)) as spam > > Can't say I like that variant - getting complicated and hard to read, > even with parens. If you're importing different names from different > modules, it's going to get very long, and then people are going to > want to split it across lines, and then the obvious question is: > what's been gained over the try/except variant? > > There is one thing, though, that I'm seeing of all this. Exception > throwing is asymmetrical: you can attempt a series of statements until > one fails, but there's no convenient syntax to attempt a series of > statements until one succeeds. This is the logic of ordered choice (in parsing) I evoked in previous post, isn't it? Also same logic as 'find_first' funcs or the like (or of python's 'any' I guess). > I wonder, could the more general case > be solved? Is there a way to, without stupid stuff like eval, wrap up > a few statements so they can be executed in a loop: > > def import_any(statement_list): > for try_me in statement_list: > try: > # uhh, this is the bit I'm not sure about... > try_me() # this would work if they're functions instead! > return > except ImportError: > pass > raise ImportError > > This works for a set of functions, but not for a bunch of "from this > import that" statements. Would it be worth mangling the top of your > script until it can be done with importlib or __import__? Doesn't seem > nearly as clean, somehow. You seem to be asking for kinds of ruby blocks (or procs?), aren't you? Very practicle at times (while I'm not a fan of them, abuse leads to less readable code, as in FP). I'd enjoy them in python to encode conditional actions, like game events: a kind of pair (cause_state, effect_action) with a generic logic. In Python, one has to encode every cause and every action as separate functions, because they are distinct pieces of (named) data, while typically they're both one-liners (eg: if a unit steps on this very hex, add 15% to its health points). Denis From denis.spir at gmail.com Sun Dec 29 10:41:19 2013 From: denis.spir at gmail.com (spir) Date: Sun, 29 Dec 2013 10:41:19 +0100 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> Message-ID: <52BFEE3F.8050901@gmail.com> On 12/28/2013 09:05 PM, Andrew Barnert wrote: > On Dec 28, 2013, at 8:10, Chris Angelico wrote: > >> There is one thing, though, that I'm seeing of all this. Exception >> throwing is asymmetrical: you can attempt a series of statements until >> one fails, but there's no convenient syntax to attempt a series of >> statements until one succeeds. I wonder, could the more general case >> be solved? Is there a way to, without stupid stuff like eval, wrap up >> a few statements so they can be executed in a loop: >> >> def import_any(statement_list): >> for try_me in statement_list: >> try: >> # uhh, this is the bit I'm not sure about... >> try_me() # this would work if they're functions instead! >> return >> except ImportError: >> pass >> raise ImportError >> >> This works for a set of functions, but not for a bunch of "from this >> import that" statements. Would it be worth mangling the top of your >> script until it can be done with importlib or __import__? Doesn't seem >> nearly as clean, somehow. > > This is a clever idea. > > If you could work out a clean syntax for the "try until one works" syntax, it would solve a wider range of problems than just this import issue. > > It strikes me that there's some similarity with if/elif/else. Maybe just: > > try: > stuff > except try: > other stuff > except try: > different stuff > except Exception as e: > deal with it > > But there are a lot of open questions. First, "except try" looks horrible, but I can't think of anything better. I would buy (if cheap!): try: stuff else try: other stuff else try: different stuff except Exception as e: deal with it > If they all fail, do you only get the last exception, or are they all chained together? Same issue in parsing in front of a choice (ordered or not): when match fails, what helpful error message? The parser cannot know which branch should have matched, so the only solution is to list failure causes of every branch; but branches may hold choices themselves, etc... expected: p = a | b| c at position: .... found: ... expected: a = ... at position: .... found: ... expected: b = x | y at position: .... found: ... expected: x = ... at position: .... found: ... expected: y = ... at position: .... found: ... expected: c = ... at position: .... found: ... This suggests that a general pattern for such an (ordered) choice of potential actions should not be modeled after exception catching; except (sic) for the failure branch in case none of them runs. My solution in parsing is only giving a general error (the first block above), but providing a solution (a test check tool func, the building block of test suites) for the user to try each potential branch individually (easily, quickly, and with helpful feedback). pat.check(str) > Would there be a use for a sequence like this without except on the end (so if they all fail, it just raises)? Can you put an exception type or tuple between the "except try"? What about an as clause (so the block could use it)? I would require the general failure branch in syntax, for safety, like ordinary; with the same contents contents beeing the same: "pass" or "assert False". [1] Denis [1] I take the opportunity to ask: is there a builtin exception or other means to say "this block should never be reached"? From techtonik at gmail.com Sun Dec 29 12:28:11 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 29 Dec 2013 14:28:11 +0300 Subject: [Python-ideas] os.architecture Message-ID: if os.architecture == 32: download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') else: download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') ? From rosuav at gmail.com Sun Dec 29 13:15:10 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Dec 2013 23:15:10 +1100 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On Sun, Dec 29, 2013 at 10:28 PM, anatoly techtonik wrote: > if os.architecture == 32: > download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') > else: > download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') > Could you get that info from platform.uname()[4]? ChrisA From p.f.moore at gmail.com Sun Dec 29 13:39:52 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 29 Dec 2013 12:39:52 +0000 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On 29 December 2013 12:15, Chris Angelico wrote: > On Sun, Dec 29, 2013 at 10:28 PM, anatoly techtonik wrote: >> if os.architecture == 32: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') >> else: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') >> > > Could you get that info from platform.uname()[4]? I would have thought so, and more accurately too. An AMD64 executable won't be much use on SPARC64, no matter that they are both 64-bit - the OP doesn't have this problem as his example is for Windows (at least as long as he's happy to ignore old Itanium systems). I hadn't realised that platform.uname() gave this information on Windows, but given that it does, it looks like a better solution for the OP's problem. Paul From ncoghlan at gmail.com Sun Dec 29 16:12:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Dec 2013 01:12:36 +1000 Subject: [Python-ideas] "maybe import"? In-Reply-To: <20131228234322.GW29356@ando> References: <-7047415593530210756@gmail297201516> <2119483115886762605@gmail297201516> <20131228122700.GV29356@ando> <20131228234322.GW29356@ando> Message-ID: On 29 Dec 2013 09:44, "Steven D'Aprano" wrote: > > On Sat, Dec 28, 2013 at 11:55:19AM -0800, Andrew Barnert wrote: > > On Dec 28, 2013, at 4:27, Steven D'Aprano wrote: > > > > Or perhaps there are > > > two pure-Python modules with the same API, and you're not sure which is > > > installed. This issue is bigger than just C-accelerators in the CPython > > > standard library. It applies any time you have two or more alternatives > > > with the same API, a preferred module and an fallback module. E.g. > > > readline in the stdlib, versus pyreadline, a third-party Windows port. > > > > There are maybe three such examples that people use regularly. Is that > > enough to warrant a change in the language? > > There's only one *common* example of augmented assignment, x += 1, but > that was enough to justify a whole raft of new syntax, += -= *= |= etc > because the *idiom* of "x = x op arg" is common. > > What matters is not the specific examples of which modules get used in > this way, but that the idiom of > > try: > import this > except ImportError: > import that as this > > is common, not the specific values of this/that. I do not expect that > the idiom of having a fallback module when the preferred module isn't > available will ever disappear, simply because there are too many > circumstances in which it comes up. > > Having said that, Nick's comments strike me as critical. "import this or > that" doesn't buy us anything that can't already be solved in other > ways. If it were easy to add, then perhaps it would be justified. But > the implementation of import is already gnarly and hard to maintain (so > I'm told), so adding more complexity to it for something which is purely > syntactic sugar is probably not the best thing to do. I suspect this would actually be pretty easy to implement (for a given definition of easy, since we'd likely handle it in the bytecode generation phase). However, it would still make the import statement even harder to fully comprehend than it already is, which is actually quite a notable achievement :) Cheers, Nick. > > Maybe in Python 4000 :-) > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sun Dec 29 20:38:39 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 29 Dec 2013 22:38:39 +0300 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On Sun, Dec 29, 2013 at 3:15 PM, Chris Angelico wrote: > On Sun, Dec 29, 2013 at 10:28 PM, anatoly techtonik wrote: >> if os.architecture == 32: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') >> else: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') >> > > Could you get that info from platform.uname()[4]? If Python is about readability then looking at your example, I have two questions: 1. What is platform - hardware, CPU, OS, Python flavor? 2. What is uname()[4]? For os.architecture I have only one question - what is the returned value? And it is easy to remember once you get it. It is also easy to debug. -- anatoly t. From amber.yust at gmail.com Sun Dec 29 21:53:15 2013 From: amber.yust at gmail.com (Amber Yust) Date: Sun, 29 Dec 2013 20:53:15 +0000 Subject: [Python-ideas] os.architecture References: Message-ID: <2403736338056934130@gmail297201516> Until an architecture that differs in more than just number of bits comes out, at which point 'os.architecture == 32' starts becoming awkward. On Sun Dec 29 2013 at 12:34:09 PM, anatoly techtonik wrote: > On Sun, Dec 29, 2013 at 3:15 PM, Chris Angelico wrote: > > On Sun, Dec 29, 2013 at 10:28 PM, anatoly techtonik > wrote: > >> if os.architecture == 32: > >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') > >> else: > >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') > >> > > > > Could you get that info from platform.uname()[4]? > > If Python is about readability then looking at your example, I have > two questions: > 1. What is platform - hardware, CPU, OS, Python flavor? > 2. What is uname()[4]? > > For os.architecture I have only one question - what is the returned > value? And it is easy to remember once you get it. It is also easy to > debug. > -- > anatoly t. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Dec 29 22:17:06 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Dec 2013 08:17:06 +1100 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On Mon, Dec 30, 2013 at 6:38 AM, anatoly techtonik wrote: > If Python is about readability then looking at your example, I have > two questions: > 1. What is platform - hardware, CPU, OS, Python flavor? > 2. What is uname()[4]? > > For os.architecture I have only one question - what is the returned > value? And it is easy to remember once you get it. It is also easy to > debug. You can also use platform.uname().machine if you prefer readability, but that doesn't work on Python 2. Number of bits is sufficient only if you can always assume you're on the same architecture. It's much better to simply quote the uname value. And hey, you can even cut your code down from four lines to one: download('http://www.libsdl.org/release/SDL2-2.0.1-win32-'+uname()[4])+'.zip') If you get back a 404, obviously this platform's not supported. Bingo! You've just protected yourself against trying to download and run Intel code on an ARM chip. ChrisA From abarnert at yahoo.com Sun Dec 29 23:21:08 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Dec 2013 14:21:08 -0800 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: <52BFEE3F.8050901@gmail.com> References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> <52BFEE3F.8050901@gmail.com> Message-ID: <9E44C300-08A6-4883-BC71-DC8BD4F0DC41@yahoo.com> On Dec 29, 2013, at 1:41, spir wrote: >> "except try" looks horrible, but I can't think of anything better. > > I would buy (if cheap!): > > try: > stuff > else try: > other stuff > else try: > different stuff > except Exception as e: > deal with it The advantage of "except try" is that it leaves the door open for specifying only certain exceptions instead of all, which would be useful in some cases. But the advantage of "else try" is that it closes that door, eliminating most of the other questions I and others raised. It's clearly pure syntactic sugar with no ambiguity. Maybe that's a better idea. From abarnert at yahoo.com Sun Dec 29 23:31:43 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Dec 2013 14:31:43 -0800 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> On Dec 29, 2013, at 3:28, anatoly techtonik wrote: > if os.architecture == 32: > download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') > else: > download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') Love it! A win32-x86 build is exactly what you want on 32-bit ARM9 linux, a win32-x64 build is perfect for ppc64 OS X. And if you're running a 32-bit python on 64-bit Windows you almost certainly want to download 64-bit libraries, right? I also love hard coding SDL 2.0.1 into your portable application, because there's no way they'll ever release a newer version without telling you in advance so you can forcibly update all your users, which makes their "latest" links useless. From rosuav at gmail.com Sun Dec 29 23:38:34 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Dec 2013 09:38:34 +1100 Subject: [Python-ideas] os.architecture In-Reply-To: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> References: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> Message-ID: On Mon, Dec 30, 2013 at 9:31 AM, Andrew Barnert wrote: > And if you're running a 32-bit python on 64-bit Windows you almost certainly want to download 64-bit libraries, right? Actually, that one's a bit harder to work out. I'd say uname() would say that it's a 64-bit system (of whatever architecture), so all you have is the platform module and its methods of looking at sys.executable - and the docs comments kinda scare me off using that anywhere other than Linux (and even there, it looks inefficient). Is there a best-practice way of finding out what binary libraries you should be using for this process? Surely that should be a simple thing? ChrisA From abarnert at yahoo.com Mon Dec 30 00:14:33 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Dec 2013 15:14:33 -0800 Subject: [Python-ideas] os.architecture In-Reply-To: References: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> Message-ID: <4A8405B6-586A-43C2-B653-B4F30137C97B@yahoo.com> On Dec 29, 2013, at 14:38, Chris Angelico wrote: > On Mon, Dec 30, 2013 at 9:31 AM, Andrew Barnert wrote: >> And if you're running a 32-bit python on 64-bit Windows you almost certainly want to download 64-bit libraries, right? > > Actually, that one's a bit harder to work out. I'd say uname() would > say that it's a 64-bit system (of whatever architecture), so all you > have is the platform module and its methods of looking at > sys.executable - and the docs comments kinda scare me off using that > anywhere other than Linux (and even there, it looks inefficient). Is > there a best-practice way of finding out what binary libraries you > should be using for this process? Surely that should be a simple > thing? Sure. And this is all documented in the first place I would have thought to look--right near the top of the platform module, the first thing that comes up when I google "python platform" or "python architecture". There are actually 4 different things you could be asking for, not just 2 (even ignoring the fact that you can have different architectures that aren't just 32- and 64-bit versions of the same thing). What you really want here is probably not whether you're on a 64-bit OS, or whether Python was built 64-bit--if you ever care about any platform besides Windows, you may be running under a fat/universal binary that could be built for both 32- and 64-bit, and you want to know which one it's being _run_ as. So, as the docs say: To get at the "64-bitness" of the current interpreter, it is more reliable to query the sys.maxsize attribute. Maybe that could be given a nicer name, but adding an "os.architecture" that gives you a different of the four values than "platform.architecture" and probably isn't the one you want as Anatoly suggested doesn't seem like an improvement. Maybe we could have different functions in platform for hardware_architecture, os_architecture, executable_architecture, and interpreter_architecture? But I don't think they would get much use. Besides, historically I've been as interested in knowing whether I was on, say, real ARM7 or ARM9 emulating ARM7 as in real i386 or x86_64 emulating i386, where there's no simple 32 vs. 64 cross-platform answer that has any meaning, and I don't see any reason to believe that the future will be different, just because the present happens to be different on one platform. From amber.yust at gmail.com Mon Dec 30 00:17:30 2013 From: amber.yust at gmail.com (Amber Yust) Date: Sun, 29 Dec 2013 23:17:30 +0000 Subject: [Python-ideas] os.architecture References: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> Message-ID: <-4609820619587005742@gmail297201516> On Sun Dec 29 2013 at 2:35:07 PM, Andrew Barnert wrote: > I also love hard coding SDL 2.0.1 into your portable application, because > there's no way they'll ever release a newer version without telling you in > advance so you can forcibly update all your users, which makes their > "latest" links useless. > Using a fixed version isn't necessarily a terrible idea. While lots of libraries strive for backwards compatibility, that doesn't mean they don't sometimes break it (intentionally or accidentally). Always pulling down the latest version of a library can lead to trickier debugging. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Dec 30 00:42:42 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Dec 2013 10:42:42 +1100 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On Mon, Dec 30, 2013 at 10:39 AM, anatoly techtonik wrote: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-'+uname()[4])+'.zip') > > Nothing personal, but that code is awful. You're right, it's buggy because I didn't test it. Superfluous close parenthesis there. But my point is that the architecture is right there in the name, so you can save yourself the trouble and just use what comes from uname directly. ChrisA From stephen at xemacs.org Mon Dec 30 00:30:26 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 30 Dec 2013 08:30:26 +0900 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: <87a9fjkzfx.fsf@uwakimon.sk.tsukuba.ac.jp> anatoly techtonik writes: > On Sun, Dec 29, 2013 at 3:15 PM, Chris Angelico wrote: > If Python is about readability then looking at your example, I have > two questions: > 1. What is platform - hardware, CPU, OS, Python flavor? Everything you might want in this area. (Note: 'os' is the wrong namespace for this. 'os' provides wrappers for OS services, not platform information.) In particular sum-65-1 08:16$ python Python 2.6.1 (r261:67515, Aug 2 2010, 20:10:18) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.architecture() ('64bit', '') >>> ^D Duh. Watch out for that time machine, Anatoly. For more information, RTFM or try python-list. From rosuav at gmail.com Mon Dec 30 01:08:16 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Dec 2013 11:08:16 +1100 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: On Mon, Dec 30, 2013 at 10:57 AM, anatoly techtonik wrote: > On Mon, Dec 30, 2013 at 2:42 AM, Chris Angelico wrote: >> On Mon, Dec 30, 2013 at 10:39 AM, anatoly techtonik wrote: >>>> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-'+uname()[4])+'.zip') >>> >>> Nothing personal, but that code is awful. >> >> You're right, it's buggy because I didn't test it. Superfluous close >> parenthesis there. But my point is that the architecture is right >> there in the name, so you can save yourself the trouble and just use >> what comes from uname directly. > > I write code for aliens. They might be confused that x86 < x64. Who cares? You're building up a URL and going and downloading it. The user needn't even be aware of it. You're actually writing code for code, here. ChrisA From ncoghlan at gmail.com Mon Dec 30 01:24:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Dec 2013 10:24:13 +1000 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: <9E44C300-08A6-4883-BC71-DC8BD4F0DC41@yahoo.com> References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> <52BFEE3F.8050901@gmail.com> <9E44C300-08A6-4883-BC71-DC8BD4F0DC41@yahoo.com> Message-ID: On 30 Dec 2013 08:24, "Andrew Barnert" wrote: > > On Dec 29, 2013, at 1:41, spir wrote: > > >> "except try" looks horrible, but I can't think of anything better. > > > > I would buy (if cheap!): > > > > try: > > stuff > > else try: > > other stuff > > else try: > > different stuff > > except Exception as e: > > deal with it > > The advantage of "except try" is that it leaves the door open for specifying only certain exceptions instead of all, which would be useful in some cases. > > But the advantage of "else try" is that it closes that door, eliminating most of the other questions I and others raised. It's clearly pure syntactic sugar with no ambiguity. Maybe that's a better idea. The else block triggers when no exception was thrown, so even if this idea was to be accepted at all (which seems unlikely), that particular spelling definitely wouldn't be accepted for something that only triggers when an exception is thrown. In practice, the existing more general solution of extracting an appropriately named function is likely to improve readability more than adding a new variation on the already complex try/except/else/finally construct, which is why I consider it unlikely that a change along these lines would be accepted. Cheers, Nick. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Mon Dec 30 01:27:17 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 30 Dec 2013 09:27:17 +0900 Subject: [Python-ideas] os.architecture In-Reply-To: References: Message-ID: <8761q7kwt6.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > But my point is that the architecture is right there in the name, > so you can save yourself the trouble and just use what comes from > uname directly. Except you probably can't. "x64" != "x86_64". (The latter is what my Intel Mac says; don't have a 64-bit machine with win32 on it handy.) From denis.spir at gmail.com Mon Dec 30 09:24:16 2013 From: denis.spir at gmail.com (spir) Date: Mon, 30 Dec 2013 09:24:16 +0100 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> <52BFEE3F.8050901@gmail.com> <9E44C300-08A6-4883-BC71-DC8BD4F0DC41@yahoo.com> Message-ID: <52C12DB0.40101@gmail.com> On 12/30/2013 01:24 AM, Nick Coghlan wrote: > On 30 Dec 2013 08:24, "Andrew Barnert" wrote: >> >> On Dec 29, 2013, at 1:41, spir wrote: >> >>>> "except try" looks horrible, but I can't think of anything better. >>> >>> I would buy (if cheap!): >>> >>> try: >>> stuff >>> else try: >>> other stuff >>> else try: >>> different stuff >>> except Exception as e: >>> deal with it >> >> The advantage of "except try" is that it leaves the door open for > specifying only certain exceptions instead of all, which would be useful in > some cases. >> >> But the advantage of "else try" is that it closes that door, eliminating > most of the other questions I and others raised. It's clearly pure > syntactic sugar with no ambiguity. Maybe that's a better idea. > > The else block triggers when no exception was thrown, so even if this idea > was to be accepted at all (which seems unlikely), that particular spelling > definitely wouldn't be accepted for something that only triggers when an > exception is thrown. I guess mean the else block in a 'normal' "try...except...else" construct? In this different form, there "else try", not just "else". I understood it as a single keyword (maybe read it elsetry, somewhat like elif) that denotes different semantics. > In practice, the existing more general solution of extracting an > appropriately named function is likely to improve readability more than > adding a new variation on the already complex try/except/else/finally > construct, which is why I consider it unlikely that a change along these > lines would be accepted. After some more reflexion, I find this general pattern of "chained try" (new thread title) really meaningful and useful, but maybe too rare to deserve proper syntax. However, what is the proper way to express it using current Python features? Clearly, a series of nested try/except (compare: a series of nested if/else if python had no elif) is not the right tool, in my view, because try/except is to be used whenever the except branch is rare (semantically and practically, because itis clstly). What else? Try/elsetry would not need to trigger actual exceptions with full error objects, exception catching, and stack unwinding (the whole machinary), but only set a plain flag meaning "this branch fails to execute normally [1]. Denis [1] Could be a global bit or other var in the implementation, a CPU register, or even the carry flag in the CPU. See also: [https://www.lyngvig.org/%2FTeknik%2FA-Proposal-for-Exception-Handling-in-C] From jeanpierreda at gmail.com Mon Dec 30 11:23:12 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 30 Dec 2013 02:23:12 -0800 Subject: [Python-ideas] chained try (was Re: "maybe import"?) In-Reply-To: <52C12DB0.40101@gmail.com> References: <-7047415593530210756@gmail297201516> <20131227040808.GP29356@ando> <52BEE768.7060401@gmail.com> <52BFEE3F.8050901@gmail.com> <9E44C300-08A6-4883-BC71-DC8BD4F0DC41@yahoo.com> <52C12DB0.40101@gmail.com> Message-ID: On Mon, Dec 30, 2013 at 12:24 AM, spir wrote: > > After some more reflexion, I find this general pattern of "chained try" (new > thread title) really meaningful and useful, but maybe too rare to deserve > proper syntax. However, what is the proper way to express it using current > Python features? > > Clearly, a series of nested try/except (compare: a series of nested if/else > if python had no elif) is not the right tool, in my view, because try/except > is to be used whenever the except branch is rare (semantically and > practically, because itis clstly). What else? > > Try/elsetry would not need to trigger actual exceptions with full error > objects, exception catching, and stack unwinding (the whole machinary), but > only set a plain flag meaning "this branch fails to execute normally [1]. If you, instead of raising an exception, return None or that thing, then you can do things as so: return foo() or bar() or baz() This can't work if None is a reasonable value rather than just an error value (which is often why one switches to exceptions to begin with), but it works OK much of the time, and works OK all of the time in some other programming languages that are built around this idiom. -- Devin From techtonik at gmail.com Mon Dec 30 00:45:01 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 30 Dec 2013 02:45:01 +0300 Subject: [Python-ideas] os.architecture In-Reply-To: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> References: <363CF3DE-58AF-40AA-96DA-3CFB62D50BF0@yahoo.com> Message-ID: On Mon, Dec 30, 2013 at 1:31 AM, Andrew Barnert wrote: > On Dec 29, 2013, at 3:28, anatoly techtonik wrote: > >> if os.architecture == 32: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x86.zip') >> else: >> download('http://www.libsdl.org/release/SDL2-2.0.1-win32-x64.zip') > > Love it! A win32-x86 build is exactly what you want on 32-bit ARM9 linux, a win32-x64 build is perfect for ppc64 OS X. And if you're running a 32-bit python on 64-bit Windows you almost certainly want to download 64-bit libraries, right? I am glad you've noticed this piece of code is for Windows. It is also nice that you understand the difference between OS bitness and Python bitness. > I also love hard coding SDL 2.0.1 into your portable application, because there's no way they'll ever release a newer version without telling you in advance so you can forcibly update all your users, which makes their "latest" links useless. You'll like it even more if I tell you there is also a hash and size info hard coded. https://bitbucket.org/techtonik/locally/src/8367fad824c111367b99baf443c5d38b525111a5/03.fetch-sdl2.py?at=default#cl-85 From techtonik at gmail.com Mon Dec 30 00:54:29 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 30 Dec 2013 02:54:29 +0300 Subject: [Python-ideas] os.architecture In-Reply-To: <87a9fjkzfx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87a9fjkzfx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Dec 30, 2013 at 2:30 AM, Stephen J. Turnbull wrote: > anatoly techtonik writes: > > On Sun, Dec 29, 2013 at 3:15 PM, Chris Angelico wrote: > > > If Python is about readability then looking at your example, I have > > two questions: > > 1. What is platform - hardware, CPU, OS, Python flavor? > > Everything you might want in this area. I want to know if I am running 32-bit or 64-bit Windows on my 64-bit CPU. So, how to resolve the ambiguity? > (Note: 'os' is the wrong > namespace for this. 'os' provides wrappers for OS services, not > platform information.) In particular For OS information, 'os' is a logical namespace. I don't know about CPU info though. Would be nice to have 'hardware' or 'hwinfo' module in stdlib. I don't know where info about Python bitness should be. >>>> import platform >>>> platform.architecture() > ('64bit', '') >>>> ^D From techtonik at gmail.com Mon Dec 30 09:56:33 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 30 Dec 2013 11:56:33 +0300 Subject: [Python-ideas] os.architecture In-Reply-To: <8761q7kwt6.fsf@uwakimon.sk.tsukuba.ac.jp> References: <8761q7kwt6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Ok. Architecture is a fail in terminology. The word "OS architecture" can mean many things, and it will be the same design flaw as os.name. How about os.bitness instead? From abarnert at yahoo.com Mon Dec 30 13:13:45 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Dec 2013 04:13:45 -0800 Subject: [Python-ideas] os.architecture In-Reply-To: References: <8761q7kwt6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Dec 30, 2013, at 0:56, anatoly techtonik wrote: > Ok. Architecture is a fail in terminology. The word "OS architecture" > can mean many things, and it will be the same design flaw as os.name. > > How about os.bitness instead? You missed the part where you were told that os is for OS services, not platform (including hardware, interpreter, and OS) information. That, as you might guess, goes in platform. And it's already there, as you've been told at least three times, so there is no need to add it. Anyway, "bitness" by itself doesn't tell you whether it will return 32 or 64 when running a 32-bit Python on 64-bit Windows (much less when running a universal 64-bit-capable Python on 64-bit Mac in 32-bit mode). It's just as potentially ambiguous as the functions that already exist, and therefore no better than them. And of course they have the advantage of already existing, and working in older versions of Python, and so on. Instead of being bewildered by "what could platform possibly mean", just skim the docs, or use the builtin help.