From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 1 03:10:54 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 1 Nov 2016 16:10:54 +0900 Subject: [Python-ideas] Non-ASCII in Python syntax? [was: Null coalescing operator] In-Reply-To: <20161031142940.GC3365@ando.pearwood.info> References: <22548.55868.907868.875328@turnbull.sk.tsukuba.ac.jp> <22549.39569.330494.335762@turnbull.sk.tsukuba.ac.jp> <20161031001734.GA15983@ando.pearwood.info> <22550.39998.349344.967100@turnbull.sk.tsukuba.ac.jp> <20161031142940.GC3365@ando.pearwood.info> Message-ID: <22552.16382.182390.621292@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > In other words, ? behaves as a unary postfix operator that squares > its argument. Likewise for ?, etc. You can even combine them: x?? > would be the same as x**33. There's more here: I hope that's configurable. I use superscripts to indicate an index as often as I use them to indicate an exponent. From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 1 03:15:53 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 1 Nov 2016 16:15:53 +0900 Subject: [Python-ideas] More user-friendly version for string.translate() In-Reply-To: References: <20161025023704.GD15983@ando.pearwood.info> <22543.37392.571089.528253@turnbull.sk.tsukuba.ac.jp> <22544.64697.310751.462520@turnbull.sk.tsukuba.ac.jp> Message-ID: <22552.16681.544472.5900@turnbull.sk.tsukuba.ac.jp> Chris Barker writes: > pretty slick -- but any hope of it being as fast as a C implemented method? I would expect not in CPython, but if "fast" matters, why are you using CPython rather than PyPy or Cython? If it matters *that* much, you can afford to write your own C implementation. But I doubt that fast matters "that much" often enough to be worth maintaining yet another string method in Python. Byte-shoveling servers might want it for bytes, though. > I've always figured that Python's rich string methods provided two things: > > 1) single method call to do common things > > 2) nice fast, pure C performance > > so I think a "keep these" method would help with both of these > goals. Sure, but the translate method already gives you that, and a lot more. Note that when you're talking about working with Unicode characters, no natural language activity I can imagine (not even translating Buddhist texts, which involves a couple of Indian scripts as well as Han ideographs) uses more than a fraction of defined characters. So really translate with defaultdict is a specialized loop that marries an algorithmic body (which could do things like look up the original script or other character properties to decide on the replacement for the generic case) with a (usually "small") table of exceptions. That seems like inspired design to me. From pavel.velikhov at gmail.com Tue Nov 1 04:33:46 2016 From: pavel.velikhov at gmail.com (Pavel Velikhov) Date: Tue, 1 Nov 2016 11:33:46 +0300 Subject: [Python-ideas] Query Language extension to Python Message-ID: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Hi Folks, We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language, drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have an interactive interpreter and Jupyter support planned. Best regards! PythonQL team From p.f.moore at gmail.com Tue Nov 1 05:09:13 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 1 Nov 2016 09:09:13 +0000 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Message-ID: On 1 November 2016 at 08:33, Pavel Velikhov wrote: > We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language, > drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! > > The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have > an interactive interpreter and Jupyter support planned. Nice! Paul From ncoghlan at gmail.com Tue Nov 1 06:11:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Nov 2016 20:11:15 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 1 November 2016 at 01:51, Mark E. Haase wrote: > Stephen J. Turnbull wrote: >> >> I gather you think you have a deadlock here. The way to break it is >> to just do it. Pick a syntax and do the rewriting. My memory of some >> past instances is that many of the senior devs (especially Guido) will >> "see through the syntax" to evaluate the benefits of the proposal, >> even if they've said they don't particularly like the initially- >> proposed syntax. > > I don't feel deadlocked, but I think you're right about committing to a > syntax. So I updated the PEP, summarized here: > > Spelling a new operator as a keyword is difficult due to backward > compatibility. It can be done (see PEP-308 and PEP-492) but requires extreme > care. > A keyword operator is considered less ugly than punctuation, but it makes > assignment shortcut syntax very ugly. Assume the coalesce operator is "foo", > then the assignment shortcut would be "x foo= y". This is unacceptable. > If eliminate the possibility of a keyword and focus on punctuation, we find > that most people think "??" ? the syntax that exists in several other > mainstream languages ? is ugly and not Pythonic. > However, any other punctuation spelling will be at least as ugly and will > not have the benefit of being familiar to programmers who have seen null > coalescing in other languages. > Therefore, the most reasonable spelling is to borrow the same spelling that > other languages use, e.g. "??", "?.", and "?[". > > I did go down the road of trying to create a new keyword, trying some > mundane ideas ("foo else bar") and some more exotic ideas ("try foo then > bar"), but I don't know if those syntaxes are even parseable, and as I > worked through a bunch of examples, I realized that all of the keywords I > was trying were very awkward in practical use, especially when combined with > other expressions. I do think it would be worth covering the symbol+keyword option discussed in PEP 531 (i.e. "?else" instead of "??", but keeping "?.", and "?[") Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Nov 1 06:28:25 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 1 Nov 2016 10:28:25 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 1 November 2016 at 10:11, Nick Coghlan wrote: > > I do think it would be worth covering the symbol+keyword option > discussed in PEP 531 (i.e. "?else" instead of "??", but keeping "?.", > and "?[") FWIW, I'm not keen on it. As a technical question, would it be treated in the syntax as a keyword, which happened to be made up of a punctuation character followed by letters, or as a ? symbol followed by a keyword? The difference would be in how "? else" was treated. If the space is not allowed, we have a unique situation in Python's grammar (where whitespace between a symbol and a keyword is explicitly disallowed rather than being optional). If it is allowed, I suspect a lot of people would prefer to write "? else" and aesthetically the two seem very different to me. Paul From ncoghlan at gmail.com Tue Nov 1 07:02:13 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Nov 2016 21:02:13 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 1 November 2016 at 20:28, Paul Moore wrote: > On 1 November 2016 at 10:11, Nick Coghlan wrote: >> >> I do think it would be worth covering the symbol+keyword option >> discussed in PEP 531 (i.e. "?else" instead of "??", but keeping "?.", >> and "?[") > > FWIW, I'm not keen on it. > > As a technical question, would it be treated in the syntax as a > keyword, which happened to be made up of a punctuation character > followed by letters, or as a ? symbol followed by a keyword? Combined keyword, rather than two distinct tokens. If you don't do that, you end up creating ambiguities for other possible uses of "?" (like the "." and "?[]" suggestions) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Tue Nov 1 10:46:30 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Nov 2016 07:46:30 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: I personally find the ?keyword pattern has less appeal than ?, ?? or ?. . On Tue, Nov 1, 2016 at 4:02 AM, Nick Coghlan wrote: > On 1 November 2016 at 20:28, Paul Moore wrote: > > On 1 November 2016 at 10:11, Nick Coghlan wrote: > >> > >> I do think it would be worth covering the symbol+keyword option > >> discussed in PEP 531 (i.e. "?else" instead of "??", but keeping "?.", > >> and "?[") > > > > FWIW, I'm not keen on it. > > > > As a technical question, would it be treated in the syntax as a > > keyword, which happened to be made up of a punctuation character > > followed by letters, or as a ? symbol followed by a keyword? > > Combined keyword, rather than two distinct tokens. > > If you don't do that, you end up creating ambiguities for other > possible uses of "?" (like the "." and "?[]" suggestions) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Nov 1 12:42:14 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 1 Nov 2016 11:42:14 -0500 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Message-ID: Cool! https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial How do I determine how much computation is pushed to the data? (Instead of pulling all the data and running the computation with one local node) ... https://en.wikipedia.org/wiki/Bulk_synchronous_parallel (MapReduce,) - http://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries - http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html - https://github.com/yhat/pandasql/ - http://docs.ibis-project.org/sql.html#common-column-expressions - https://github.com/cloudera/ibis/blob/master/ibis/sql/alchemy.py On Tuesday, November 1, 2016, Pavel Velikhov wrote: > Hi Folks, > > We have released PythonQL, a query language extension to Python (we have > extended Python?s comprehensions with a full-fledged query language, > drawing from the useful features of SQL, XQuery and JSONiq). Take a look > at the project here: http://www.pythonql.org and lets us know what you > think! > > The way PythonQL currently works is you mark PythonQL files with a > special encoding and the system runs a preprocessor for all such files. We > have > an interactive interpreter and Jupyter support planned. > > Best regards! > PythonQL team > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavel.velikhov at gmail.com Tue Nov 1 12:45:31 2016 From: pavel.velikhov at gmail.com (Pavel Velikhov) Date: Tue, 1 Nov 2016 19:45:31 +0300 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Message-ID: <20FAA4F4-BCD2-4BBB-B319-941C195368ED@gmail.com> Hi Wes! Right now we don?t push anything yet, we fetch everything into the Python?s runtime. But going forward the current idea is to push as much computation to the database as possible (most of the time the database will do a better job then our engine). If we run on top PySpark/Hadoop I think we should be able to completely translate 100% of PythonQL into these jobs. > On 1 Nov 2016, at 19:42, Wes Turner wrote: > > Cool! > > https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial > > How do I determine how much computation is pushed to the data? (Instead of pulling all the data and running the computation with one local node) ... https://en.wikipedia.org/wiki/Bulk_synchronous_parallel (MapReduce,) > > - http://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries > - http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html > - https://github.com/yhat/pandasql/ > - http://docs.ibis-project.org/sql.html#common-column-expressions > - https://github.com/cloudera/ibis/blob/master/ibis/sql/alchemy.py > > On Tuesday, November 1, 2016, Pavel Velikhov > wrote: > Hi Folks, > > We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language, > drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! > > The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have > an interactive interpreter and Jupyter support planned. > > Best regards! > PythonQL team > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Nov 1 13:17:09 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 1 Nov 2016 12:17:09 -0500 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: <20FAA4F4-BCD2-4BBB-B319-941C195368ED@gmail.com> References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> <20FAA4F4-BCD2-4BBB-B319-941C195368ED@gmail.com> Message-ID: On Tuesday, November 1, 2016, Pavel Velikhov wrote: > Hi Wes! > > Right now we don?t push anything yet, we fetch everything into the > Python?s runtime. > But going forward the current idea is to push as much computation to the > database as > possible (most of the time the database will do a better job then our > engine). > > If we run on top PySpark/Hadoop I think we should be able to completely > translate 100% of > PythonQL into these jobs. > That would be great; and fast. A few more links that may be of use (in addition to ops._ in alchemy.py): - https://github.com/pythonql/pythonql/blob/master/Grammar.md#query-expressions - https://github.com/cloudera/ibis/blob/master/ibis/expr/window.py - http://www.ibis-project.org/faq.html#ibis-and-spark-pyspark - https://github.com/cloudera/ibis/tree/master/ibis/spark - http://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala - http://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes > > On 1 Nov 2016, at 19:42, Wes Turner > wrote: > > Cool! > > https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial > > How do I determine how much computation is pushed to the data? (Instead of > pulling all the data and running the computation with one local node) ... > https://en.wikipedia.org/wiki/Bulk_synchronous_parallel (MapReduce,) > > - http://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries > - http://pandas.pydata.org/pandas-docs/stable/generated/ > pandas.read_sql_query.html > - https://github.com/yhat/pandasql/ > - http://docs.ibis-project.org/sql.html#common-column-expressions > - https://github.com/cloudera/ibis/blob/master/ibis/sql/alchemy.py > > On Tuesday, November 1, 2016, Pavel Velikhov > wrote: > >> Hi Folks, >> >> We have released PythonQL, a query language extension to Python (we >> have extended Python?s comprehensions with a full-fledged query language, >> drawing from the useful features of SQL, XQuery and JSONiq). Take a look >> at the project here: http://www.pythonql.org and lets us know what you >> think! >> >> The way PythonQL currently works is you mark PythonQL files with a >> special encoding and the system runs a preprocessor for all such files. We >> have >> an interactive interpreter and Jupyter support planned. >> >> Best regards! >> PythonQL team >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Nov 1 14:22:51 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 1 Nov 2016 18:22:51 +0000 Subject: [Python-ideas] Non-ASCII in Python syntax? [was: Null coalescing operator] In-Reply-To: <22552.16382.182390.621292@turnbull.sk.tsukuba.ac.jp> References: <22548.55868.907868.875328@turnbull.sk.tsukuba.ac.jp> <22549.39569.330494.335762@turnbull.sk.tsukuba.ac.jp> <20161031001734.GA15983@ando.pearwood.info> <22550.39998.349344.967100@turnbull.sk.tsukuba.ac.jp> <20161031142940.GC3365@ando.pearwood.info> <22552.16382.182390.621292@turnbull.sk.tsukuba.ac.jp> Message-ID: <6ef80298-4f62-129b-2345-e04fbe389b98@mrabarnett.plus.com> On 2016-11-01 07:10, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > In other words, ? behaves as a unary postfix operator that squares > > its argument. Likewise for ?, etc. You can even combine them: x?? > > would be the same as x**33. There's more here: > > I hope that's configurable. I use superscripts to indicate an index > as often as I use them to indicate an exponent. > That's a strange thing to do. It's more usual to use a _subscript_ to indicate an index: a? vs a? From mertz at gnosis.cx Tue Nov 1 14:17:27 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 1 Nov 2016 11:17:27 -0700 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Message-ID: How do you see this as different from Blaze ( http://blaze.readthedocs.io/en/latest/index.html)? A On Nov 1, 2016 1:34 AM, "Pavel Velikhov" wrote: > Hi Folks, > > We have released PythonQL, a query language extension to Python (we have > extended Python?s comprehensions with a full-fledged query language, > drawing from the useful features of SQL, XQuery and JSONiq). Take a look > at the project here: http://www.pythonql.org and lets us know what you > think! > > The way PythonQL currently works is you mark PythonQL files with a > special encoding and the system runs a preprocessor for all such files. We > have > an interactive interpreter and Jupyter support planned. > > Best regards! > PythonQL team > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavel.velikhov at gmail.com Tue Nov 1 15:02:53 2016 From: pavel.velikhov at gmail.com (Pavel Velikhov) Date: Tue, 1 Nov 2016 22:02:53 +0300 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> Message-ID: Hi David! I haven?t used blaze, but its looks quite similar to pandas, at least conceptually. Thanks for the reference! The big difference with PythonQL is that we actually extend the syntax of Python with a few constructs that are typically used in query languages (group by, order by, window, let clause). The language extension is quite small and easy to grasp, but its very powerful: you can use this language to easily formulate pretty complex queries is a rather simple way. So traditionally - a query language (PythonQL) is good at expressing complex things easily, but then you need a lot of work from the optimizer and the database to turn it into an efficient plan. A library like blaze or pandas is more of an ?algebra? - its really a plan specification. It will usually take much longer to memorize all the operators and ways of doing things in such a library and typically you have to go back to the documentation to do things that differ slightly from what you typically do. Oh yeah, so far our execution engine is pretty simple and not too efficient, but we plan to fix this in the future and be at least comparable to pandas performance (need to look at what? s under the hood in blaze). Of course this is my take (although I heard a few similar things from our early users). It would be interesting to see how other folks compare the two approaches. Btw. we have built a library for working with pandas Dataframes, we could do it for blaze too, I suppose. > On 1 Nov 2016, at 21:17, David Mertz wrote: > > How do you see this as different from Blaze (http://blaze.readthedocs.io/en/latest/index.html )? > A > > > On Nov 1, 2016 1:34 AM, "Pavel Velikhov" > wrote: > Hi Folks, > > We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language, > drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! > > The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have > an interactive interpreter and Jupyter support planned. > > Best regards! > PythonQL team > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 1 20:09:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 Nov 2016 11:09:21 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> Message-ID: <20161102000920.GI3365@ando.pearwood.info> On Mon, Oct 31, 2016 at 05:44:07PM -0400, Random832 wrote: > Right now, foo.bar.baz(bletch) is Call(Attribute(Attribute(Name('foo'), > 'bar'), 'baz'), [Name('bletch')])), which is identical to > (foo.bar).baz(bletch) or (foo.bar.baz)(bletch). These are treated, > essentially, as postfix operators, where you can parenthesize any left > part of the expression and leave its meaning [and its AST] unchanged. > > Is the AST going to be unchanged, leading to the conclusion that the > short-circuiting in (foo?.bar).baz will "reach outside of" the > parentheses, and relying on the fact that wanting to do that with None > is a silly thing to do in almost all cases? I hope not. Even if it is "a silly thing to do in almost all cases", nevertheless it makes it hard to think about code if the ?. operator can reach outside of parentheses. If it can do that, just how far outside will it reach? Besides, "almost all" is not "all". For example: spam?.attr.__class__.__name__ (spam?.attr).__class__.__name__ I expect[1] that the first case would be equivalent to: None if spam is None else spam.attr.__class__.__name__ and the second case would be equivalent to: (None if spam is None else spam.attr).__class__.__name__ Concrete example: given spam = None, the first unparenthised version will return None, while the second parenthised version will return 'NoneType'. I don't know when I would ever want to actually do this in practice, but allowing the ?. operator to magically effect code outside of the parentheses definitely counts as "spooky action at a distance". Guido's rule of "everything to the right" is easy to reason about if "to the right" ends where the parenthised expression ends. [1] Actually I don't, or at least I didn't. I expected ?. to apply only to a single look-up. But Guido's description of the "everything to the right" rule seems like it will probably be more useful in practice and allow us to avoid writing long chains of spam?.eggs?.cheese?.tomato. So I'm changing my expectations. -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 1 21:02:00 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 2 Nov 2016 10:02:00 +0900 Subject: [Python-ideas] Non-ASCII in Python syntax? [was: Null coalescing operator] In-Reply-To: <6ef80298-4f62-129b-2345-e04fbe389b98@mrabarnett.plus.com> References: <22548.55868.907868.875328@turnbull.sk.tsukuba.ac.jp> <22549.39569.330494.335762@turnbull.sk.tsukuba.ac.jp> <20161031001734.GA15983@ando.pearwood.info> <22550.39998.349344.967100@turnbull.sk.tsukuba.ac.jp> <20161031142940.GC3365@ando.pearwood.info> <22552.16382.182390.621292@turnbull.sk.tsukuba.ac.jp> <6ef80298-4f62-129b-2345-e04fbe389b98@mrabarnett.plus.com> Message-ID: <22553.15112.406785.61738@turnbull.sk.tsukuba.ac.jp> MRAB writes: > That's a strange thing to do. It's more usual to use a _subscript_ to > indicate an index: a? vs a? Oh, we economic theorists do that too. It's typically a double-indexed array of parameters, where both rows and columns can be meaningfully be treated as vectors. So a? is the vector of quantities of good 3 produced by all firms, while a? is the vector of quantities of all goods produced by firm 3. Or in analysis of international or interregional trade, there's an index indicating which country exports which good to which importing country. Some people put the good index in the superscript and the two countries in the subscript, others the opposite. IIRC, mathematical physicist use both subscript and superscript in tensor notation, nuclear physicists use one for atomic number and the other for atomic weight (and thus would expect both subscript and superscript to be treated lexically as identifier components, not as expression components). The point is not that polynomials are not the most common use of superscript notation -- I don't care one way or the other. It's that there are many uses, important to those fields, that aren't polynomials. From rosuav at gmail.com Tue Nov 1 23:09:24 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 2 Nov 2016 14:09:24 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161102000920.GI3365@ando.pearwood.info> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> <20161102000920.GI3365@ando.pearwood.info> Message-ID: On Wed, Nov 2, 2016 at 11:09 AM, Steven D'Aprano wrote: > I don't know when I would ever want to actually do this in practice, but > allowing the ?. operator to magically effect code outside of the > parentheses definitely counts as "spooky action at a distance". Guido's > rule of "everything to the right" is easy to reason about if "to the > right" ends where the parenthised expression ends. > We already expect "to the left" and "to the right" to end based on operator precedence rules. Parentheses are used to control operator precedence. It would surprise people *greatly* if they didn't bound the effect of the question mark. ChrisA From gvanrossum at gmail.com Tue Nov 1 23:59:59 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 1 Nov 2016 20:59:59 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> <20161102000920.GI3365@ando.pearwood.info> Message-ID: Geez. --Guido (mobile) On Nov 1, 2016 8:10 PM, "Chris Angelico" wrote: > On Wed, Nov 2, 2016 at 11:09 AM, Steven D'Aprano > wrote: > > I don't know when I would ever want to actually do this in practice, but > > allowing the ?. operator to magically effect code outside of the > > parentheses definitely counts as "spooky action at a distance". Guido's > > rule of "everything to the right" is easy to reason about if "to the > > right" ends where the parenthised expression ends. > > > > We already expect "to the left" and "to the right" to end based on > operator precedence rules. Parentheses are used to control operator > precedence. It would surprise people *greatly* if they didn't bound > the effect of the question mark. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Nov 2 06:18:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 Nov 2016 21:18:24 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> <20161102000920.GI3365@ando.pearwood.info> Message-ID: <20161102101823.GK3365@ando.pearwood.info> On Wed, Nov 02, 2016 at 02:09:24PM +1100, Chris Angelico wrote: > We already expect "to the left" and "to the right" to end based on > operator precedence rules. Parentheses are used to control operator > precedence. It would surprise people *greatly* if they didn't bound > the effect of the question mark. That's exactly my point. I was just responding to Random's suggestion that ?. could/should/might reach outside of the parens. I didn't pluck this idea out of thin air only to object to it. -- Steve From rosuav at gmail.com Wed Nov 2 07:34:46 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 2 Nov 2016 22:34:46 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161102101823.GK3365@ando.pearwood.info> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1477950247.1280228.773133201.5A3B88B6@webmail.messagingengine.com> <20161102000920.GI3365@ando.pearwood.info> <20161102101823.GK3365@ando.pearwood.info> Message-ID: On Wed, Nov 2, 2016 at 9:18 PM, Steven D'Aprano wrote: > On Wed, Nov 02, 2016 at 02:09:24PM +1100, Chris Angelico wrote: >> We already expect "to the left" and "to the right" to end based on >> operator precedence rules. Parentheses are used to control operator >> precedence. It would surprise people *greatly* if they didn't bound >> the effect of the question mark. > > That's exactly my point. I was just responding to Random's suggestion > that ?. could/should/might reach outside of the parens. I didn't pluck > this idea out of thin air only to object to it. I know. Sorry, I should have included some of Random's text in my quote. ChrisA From ncoghlan at gmail.com Wed Nov 2 10:59:39 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Nov 2016 00:59:39 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 2 November 2016 at 00:46, Guido van Rossum wrote: > I personally find the ?keyword pattern has less appeal than ?, ?? or ?. . Good to know :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Wed Nov 2 11:46:54 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 2 Nov 2016 08:46:54 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: But I also recall learning CoffeeScript via cargo-culting a large existing codebase and having not the foggiest ideas when it made sense to use ?. and when plain . was enough. So I think this feature is not very new-user-friendly and I still expect that in the end we'll have two rejected PEPs. But first we need to agree on what even the right definition of ?. is. It's been frighteningly difficult to explain this even on this list, even though I have a very clear view in my head, and PEP 505 also has the same semantics and explains well why those are the right semantics. So that's another point against this. It's syntactic sugar causing lots of cavities. On Wed, Nov 2, 2016 at 7:59 AM, Nick Coghlan wrote: > On 2 November 2016 at 00:46, Guido van Rossum wrote: > > I personally find the ?keyword pattern has less appeal than ?, ?? or ?. . > > Good to know :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 2 12:17:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Nov 2016 02:17:14 +1000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 3 November 2016 at 01:46, Guido van Rossum wrote: > But I also recall learning CoffeeScript via cargo-culting a large existing > codebase and having not the foggiest ideas when it made sense to use ?. and > when plain . was enough. So I think this feature is not very > new-user-friendly and I still expect that in the end we'll have two rejected > PEPs. But first we need to agree on what even the right definition of ?. is. > It's been frighteningly difficult to explain this even on this list, even > though I have a very clear view in my head, and PEP 505 also has the same > semantics and explains well why those are the right semantics. So that's > another point against this. It's syntactic sugar causing lots of cavities. Yeah, and so far the protocol based alternative I'm working on hasn't been any less headache-inducing (Mark has been reviewing some early iterations and had to draw a diagram to try to follow the proposed control flow). I think I have a way to simplify that idea further though, and if that works out I should have something I'm willing to share with a wider audience. The gist is that rather than writing the bare: target = expr1 ?? expr2 ?? expr3 You'd instead write None-coalescing as: target = exists(expr1) ?? exists(expr2) ?? expr3 and None-propagating as: target = missing(expr1) ?? missing(expr2) ?? expr3 with ?? being a protocol-driven short-circuiting binary operator controlled by the left operand rather than defining any particular semantics of its own. The "obj?." and "obj?[]" would then be shorthand for particular uses of "missing(obj) ?? ..." that avoid duplicate evaluation of the left operand (as well as bypassing the overhead of actually creating a "missing" instance). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From srkunze at mail.de Wed Nov 2 13:32:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 2 Nov 2016 18:32:22 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: <49d8716e-de3c-9481-ba40-927da6a8f769@mail.de> On 02.11.2016 17:17, Nick Coghlan wrote: > The gist is that rather than writing the bare: > > target = expr1 ?? expr2 ?? expr3 > > You'd instead write None-coalescing as: > > target = exists(expr1) ?? exists(expr2) ?? expr3 > > and None-propagating as: > > target = missing(expr1) ?? missing(expr2) ?? expr3 > > with ?? being a protocol-driven short-circuiting binary operator > controlled by the left operand rather than defining any particular > semantics of its own. I am sorry that I had to read this twice to get what you mean. > > The "obj?." and "obj?[]" would then be shorthand for particular uses > of "missing(obj) ?? ..." that avoid duplicate evaluation of the left > operand (as well as bypassing the overhead of actually creating a > "missing" instance). That could work if we accept that ?. and ?[] is only about the left hand-side. However, as the ? visually applies also to the attribute/index access on the right, which means it could be a more readable way of doing getattr(x, 'attr', None) or x[0] if len(x) > 0 else None Imagine you need to work with incomplete data and do some chaining with the new syntax. As soon as one single attribute isn't there, we hit an exception. The same for index. If the ?. and ?[] are introduced, I think applying the "non-existence" property also the right hand make sense here as well for a wide range of applications. Best, Sven From chris.barker at noaa.gov Wed Nov 2 14:13:37 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 2 Nov 2016 11:13:37 -0700 Subject: [Python-ideas] More user-friendly version for string.translate() In-Reply-To: <22552.16681.544472.5900@turnbull.sk.tsukuba.ac.jp> References: <20161025023704.GD15983@ando.pearwood.info> <22543.37392.571089.528253@turnbull.sk.tsukuba.ac.jp> <22544.64697.310751.462520@turnbull.sk.tsukuba.ac.jp> <22552.16681.544472.5900@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Nov 1, 2016 at 12:15 AM, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > > pretty slick -- but any hope of it being as fast as a C implemented > method? > > I would expect not in CPython, but if "fast" matters, why are you > using CPython rather than PyPy or Cython? oh come on! > If it matters *that* much, > you can afford to write your own C implementation. This is about a possible addition to the stdlib -- me writing my own C implementation has nothing to do with it. > But I doubt that > fast matters "that much" often enough to be worth maintaining yet > another string method in Python. This could be said about every string method in Python -- I understand that every addition is more code to maintain. But somehow we are adding all kinds of stuff like yet another string formatting method, talking about null coalescing operators and who knows what -- those are all a MUCH larger burden -- not just for maintaining the interpreter, but for everyone using python having more to remember and understand. On the other hand, powerful and performant string methods are a major plus for Python -- a good reason to us it over Perl :-) So an new one that provides, as I write before: > 1) single method call to do a common thing > > > > 2) nice fast, pure C performance > would fit right into to Python, and indeed, would be a similar implementation to existing methods -- so the maintenance burden would be a small addition (i.e if the internal representation for strings changed, all those methods would need re-visiting and similar changes) So the only key question is -- is the a common enough use case? > so I think a "keep these" method would help with both of these > > goals. > > Sure, but the translate method already gives you that, and a lot more. > yes but only with the fairly esoteric use of defaultdict. which brings me back to the above: 1) single method call to do a common thing the nice thing about a single method call is discoverability -- no newbie is going to figure out the .translate + defaultdict approach. > Note that when you're talking about working with Unicode characters, > no natural language activity I can imagine (not even translating > Buddhist texts, which involves a couple of Indian scripts as well as > Han ideographs) uses more than a fraction of defined characters. > which is why you may want to remove all the others :-) So really translate with defaultdict is a specialized loop that > marries an algorithmic body (which could do things like look up the > original script or other character properties to decide on the > replacement for the generic case) with a (usually "small") table of > exceptions. That seems like inspired design to me. > indeed -- .translate() itself is remarkably flexible -- you could even pas in a custom class that does all sorts of logic. and adding the defaultdict is an easy way to add a useful feature. But again, advanced usage and not very discoverable. Maybe that means we need some more docs and/or perhaps recipes instead. Anyway, I joined this thread to clarify what might be on the table -- but while I think it's a good idea, I dont have the bandwidth to move it through the process -- so unless someone steps up that does, we're done. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From danilo.bellini at gmail.com Wed Nov 2 14:14:16 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Wed, 2 Nov 2016 16:14:16 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> Message-ID: 2016-10-25 12:02 GMT-02:00 Stephen J. Turnbull : > My point is that although new syntax may be useful for simple cases, > serious applications will worry about computational accuracy and > likely will provide packages that handle general cases that nest these > simple cases. Given they exist, most modelers will prefer using those > packages to writing their own comprehensions. That may not apply in > other fields, but AFAICS it does apply in economics. So if you can't > handle the complex cases, the syntax is just cognitive overhead: > TOOWTDI will be the high-quality general packages even where they're > theoretically overkill. > [...] Sure, you *can* express a second-order > difference equation as two first-order equations, and perhaps you > would actually calculate it that way. But in economics we normally > express a second-order diff eq as a second-order diff eq. If the > input is a second-order equation that needs to be reformulated as a > system of first-order equations, then with the code required to > implement such transformations, it is not obvious to me that having > syntax for first-order equations is going to be worth the extra syntax > in the language. Most of the complexity is going to be in the > transformation which AFAICS is likely to be problem-specific, and > therefore unavoidable by the modeler. OTOH, as PyScanPrev shows, the > complexity of recursion can be hidden in a decorator, which the > modeler can cargo-cult. > I'm not talking about "simple cases", but about quite general cases, general in the sense that it allows modeling time varying models (and the proposal is about recursion, whose computational power should be obvious). What would be more general than a linear MIMO (multiple input, multiple output) time varying state-space model? It's not so common to find some package that works with time varying models. Which ones do you know? Actually, the state-space equations (as used in a PyScanPrev example) is a "modern" way of doing general [linear time varying] control engineering using first-order equations and a n-dimensional state vector, as opposed to the "classical" difference equations and its Z transform, which are the core of my other package, AudioLazy: https://pypi.python.org/pypi/audiolazy You're right when you say that "serious applications will worry about computational accuracy", indeed that was a reason I wrote the gammatone filters in AudioLazy, the cascading filter model was a requirement to avoid getting just amplified floating point quantization garbage as a result (as used to happen when using a single call to scipy.signal.lfilter). A "rule of thumb" in signal processing is to use cascading/parallel biquads instead of higher order [recursive] filters directly due to the computational accuracy (but that doesn't apply to comb filters, for example). Anyway, that makes this talking about computational accuracy sound like an argument for my proposal here, not against it. I STRONGLY disagree with "will provide packages that handle general cases", it simply doesn't make sense. At best, a package would just mirror what happens in math: is there any general solution for general ODE? Packages handles specific things, while syntax deals with the general stuff: once you find something that a package can't do, you need to use something different, using only what the language has (or another package, if there's any available). There are packages that include converters of difference equations from/to state-space equations, but these couldn't be used when you're talking about time varying models. You can't use LTI tools for something that isn't time-invariant. 2016-10-25 12:02 GMT-02:00 Stephen J. Turnbull : > Fine, but I only questioned economics. I'm not a rocket scientist, > I'll let the rocket scientists question that. If they don't, *I* > certainly will concede you have a point in those other fields. Ok. and I'm from a digital signal processing background (EE and a MSCS on it), mainly for audio applications, though I work today in a fintech. But we're not here to give just ad hominem / ad verecundiam pseudo-arguments (I mean, we're not talking about me or you). The simplest examples I've ever seen in digital control theory were about economics, mainly because these models are useful and don't need any kind of "continuous time to discrete time" conversion, as the time is already discrete in the first place. I've created from scratch the leaking bucket example mainly to avoid copying an economics example from the Wilson J. Rugh "Linear Systems Theory" book in the PyScanPrev. If I understood you correctly, you're saying that economics is simpler than I thought and doesn't require anything beyond a basic LTI difference equation, and for that restrict scenario there are packages that already do the job. OTOH, if your point is that the scan higher order function / PyScanPrev and stuff alike aren't enough for properly modeling every economical model there is without some undesired adaptation for a subset of these models, that makes some sense. I'm just saying it's useful for a lot of things, including applications in economics. Does it need to be the Harry Potter wand to be "accepted"? 2016-10-25 12:02 GMT-02:00 Stephen J. Turnbull : > > This maillist isn't very inviting... > > Why do you say that? Because people aren't falling over themselves to > accept your proposal? [...] > You're perfectly willing to tell other people what to read, though. I > realize times have changed since 1983, but as long as I've been on the > 'net, it's always been considered polite to investigate previous > discussions, and AFAIK it still is. > In the second part of this citation, you seem to realize the reason for what I said. Do you trust anyone who gives a value judgement about something he/she didn't read about without at least disclaiming about that lack of reading? It's not me who is telling other people to judge, but it's an idea I wrote about, and that's the reason this maillist exists, isn't it? I firmly believe "list comprehension" wouldn't be "accepted" today if it wasn't in Python syntax yet. Is there any previous discussion on this topic or something related to it? Where's the link? (I asked it before) It's always been considered polite to investigate before judging. 2016-10-25 12:02 GMT-02:00 Stephen J. Turnbull : > > Perhaps there are people who prefer masochist rituals instead of > > using "reduce", who knows? Who cares? > > (1) There are. (2) Evidently you don't. (3) You should, because the > leading (literally) person who prefers writing code himself to using > "reduce" is Guido van Rossum. > Actually, that's an argument for this proposal, not against. I'm proposing a way to avoid itertools.accumulate and functools.reduce using an explicit syntax instead of the functions. Or does GvR prefer map/filter instead of list comprehensions? If you're right, then we should be consistent and propose the elimination of list comprehension and every syntax alike. The arguments you're giving against my proposal are the same. 2016-10-25 13:00 GMT-02:00 Steven D'Aprano : > > I don't know if that's a conclusion from any other thread, but that's > > wrong. The only extra "freedom" required a way to access the previous > > output (or "accumulator", "memory", "state"... you can use the name you > > prefer, but they're all the same). How many parameters does > itertools.scan > > have? And map/filter? > > I do not know which email thread is being talked about here, but the > conclusion is correct. In the most general case, you might want: > > - the current value of some running total or result; > - the previous value of the running result; > - the value of the running result before that ("previous previous"); > - and so on; > - more than one running calculation at the same time; > - the current sequence value (for x in [1, 2, 3]); > - the previous sequence value (previous value of x); > - the sequence value before that ("previous previous x"); > - etc. > Wrong. 1. The current value is already there, it's not something new. 2. The accumulator (previous result) is the only thing I'm talking about, anything beyond that is also beyond my new syntax proposal 3. The "previous previous" and so on are just lag/windowing, I'm not proposing a way to solve these, but my examples did that. Complaining about that is a proof that my examples weren't read. We can talk about a syntax to solve these as well but that's not what my proposal is about. That includes the "previous sequence value" history. 2016-10-25 13:00 GMT-02:00 Steven D'Aprano : > > Recurrence relations are not all linear, and they don't always involve > only the previous value. Fibonacci numbers need to track two previous > running values: > > F[n] = F[n-1] + F[n-2] Did you know Fibonacci is an example in PyScanPrev since long ago? And not all recurrence relations are time invariant, I gave an example of a time varying recurrence relation. Do you have any example of non-linear recurrence relation? Do you know how to linearize it, or do you have a proof that it can't be linearized as a time varying model? Why does it make any difference for this proposal? 2016-10-25 13:00 GMT-02:00 Steven D'Aprano : > And perhaps there are people who think that using "reduce" and other > functional idioms instead of good, clean, easy-to-understand, easy-to- > debug imperative code is a "masochist ritual". If you're talking about people who are misusing "reduce", then you're arguing for alternatives, and this proposal has an alternative as an obvious corollary. Or do you think everyone should prefer nested code instead of something more expression-oriented, though that's again something in PEP20? If you don't like "reduce", go ahead with the triple-for-sections. Good luck on that. But that's not the point of my proposal, I just found "reduce" is taboo, and that reminds me when I said all that matters here is social status. 2016-10-25 13:00 GMT-02:00 Steven D'Aprano : > why do we need ANOTHER way of > solving these problems when there are already so many? More use-cases > doesn't answer that question I already said: it's explicit and cleaner. Easier to memorize. Lower McCabe complexity. Less tokens (not characters). Less repeated tokens when compared with the triple-for-section comprehensions. And how about the AST tree height, wouldn't it have the same of a common list comprehension? But what would answer that question for plain common list comprehensions? Does the same complaining apply, or you think that attacking map/filter are the same as attacking list comprehensions, as you did here with my proposal and reduce? I think one should then propose that list comprehensions have to be removed, just the keep the consistency. Oh, it won't be due to backwards compatibility, I know. 2016-10-25 23:36 GMT-02:00 David Mertz : > On Tue, Oct 25, 2016 at 1:55 PM, Rob Cliffe > wrote: >> >> >>> [prev * k for k in [5, 2, 4, 3] from prev = 1] >> [1, 5, 10, 40, 120] >> >> That makes sense for me, and seem simpler than: >> >> >>> from itertools import accumulate, chain >> >>> list(accumulate(chain([1], [5, 2, 4, 3]), lambda prev, k: prev * k)) >> [1, 5, 10, 40, 120] >> >> Well, if you want an instant reaction from someone skimming this thread: >> I looked at the first example and couldn't understand it. >> > > After reading every post in the thread, I still don't understand the > proposed new syntax really. > > How does 'prev' get bound in the loop? Is it a new magic keyword for "last > accumulated element?" Does the binding in the "from" clause magically say > that it should get rebound in the loop where it's no longer mentioned? Why > is the word `from` used here when there's no obvious relation to other uses? > > The alternate version looks much better if you don't try so hard to make > it look bad. The much more obvious spelling is: > > from operator import mul > > from itertools import accumulate, chain > > accumulate(chain([1], nums), mul) > > > If you give up a fear of using `import` and stop arbitrarily converting a > possibly infinite iterator to a concrete list, this form is extremely short > and obvious. Moreover, in the example, it is extra strange that the > multiplicative identity is added into the front of the iterator. This is > exactly the same thing as simply spelling it: > > accumulate(nums, mul) > > Which is even shorter. It's feels very artificially contrived to insist > that the initial element must live somewhere other than the iterator > itself. But it would be easy enough to write a wrapper to massage an > iterator for this special case: > > def prepend(item, it): > > return itertools.chain([item], it) > > > Doesn't save any characters, but the name might emphasize the intent. > Maybe this implementation forces the point even more: > > def prepend(item, it): > > yield item > > yield from it > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From schesis at gmail.com Wed Nov 2 14:30:38 2016 From: schesis at gmail.com (Zero Piraeus) Date: Wed, 02 Nov 2016 15:30:38 -0300 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: <1478111438.1838.88.camel@gmail.com> : Disclaimer: I haven't followed all of this discussion, so some or all of the following may already have been expressed better (and perhaps refuted better still). On Wed, 2016-11-02 at 08:46 -0700, Guido van Rossum wrote: > [...] we need to agree on what even the right definition of ?. is. It's > been frighteningly difficult to explain this even on this list, even > though I have a very clear view in my head, and PEP 505 also has the > same semantics and explains well why those are the right semantics. I think the proposed semantics for ?. are confusing (and the operator itself dangerous, but more on that later). If I write something like obj.attr, the failure mode I care about is that obj has no attribute attr, rather than that obj is specifically None (or one of a defined group of somewhat Nonelike objects). Clearly, in such a circumstance, obj is not what I expected it to be, because I thought it was going to have an attribute attr, and it doesn't.? Whether that's because it's None, some other Nonelike object, False, or some other type of object altogether may or may not be relevant, but is certainly secondary to the fact that there's no attr for me to work with. Most often I will want the default behaviour that an AttributeError is raised, but often it would be useful to say I'd like some kind of default value back with e.g. getattr(obj, 'attr', default), and sometimes it would be useful for that default value to propagate through a chain of attribute accesses without having to write a convoluted mess of nested getattr calls. In the latter case it usually does not matter to me what type obj is, except that if there *were* some syntax in Python that allowed me to propagate a default value through a chain of failed attribute accesses, it would be quite frustrating to find that I couldn't use it because obj happened not to be one of an anointed cohort of Nonelike objects. I'm sure it's obvious by now that I think the intuitive behaviour of any ?. operator is to suppress AttributeError on the right hand side, rather than check for nullity on the left. Maybe this reveals na?vety on my part regarding some inherent design or implementation difficulty with suppressing errors on the right rather than checking nullity on the left. However, I think "my" semantics are at least as likely to be assumed by a non-expert as the alternative proposed in this thread, and that's going to cause a lot of confusion. ... which leads to a second point: this is dangerous. I'm going to try to be tactful here, bearing in mind Nick Coghlan's well-argued imprecations against denigrating other languages, but: I recently spent a few months writing Ruby for a living. Ruby has a &. operator with similar semantics to those proposed, as well as (in ActiveSupport) an obj.try(:attr) method closer to the exception-catching model described above. It has both of those things, as far as I can tell, because of an IMO misguided preference in the Rails world for returning nil rather than raising exceptions, which tends to cause errors to materialize far from their source. When errors materialize far from their source, they are harder to debug, and it can be very tempting to use the likes of &. and try() as band- aids, rather than thoroughly investigate the causes of such errors. The end result of this tendency is that errors materialize further still from their source, and code gets progressively harder to debug over time. I worry that introducing such band-aids into Python will encourage similar behaviour, and for that reason I'm opposed to introducing an operator with either null-coalescing or exception-suppressing behaviour.? Further, if a library encourages chained attribute accesses which may fail mid-chain, then I would rather that library take responsibility for the consequences of its design at the cost of some magic (e.g. by catching and handling AttributeError) than introduce confusing new syntax into the language. However, I do think an alternative is possible: extend getattr(). ? ? getattr(obj, ('chain', 'of', 'attributes'), default) This is more verbose than an operator, but I think its intent is clearer, and its verbosity makes it less likely to be misused. If extending getattr is unacceptable, a new dig() function (either as a builtin or part of the stdlib) with the same semantics would work just as well. If this has already been proposed here, I apologise (and would appreciate a pointer to arguments against it, if anyone can find them without too much effort). Incidentally, Chris Angelico's PEP 463 "Exception-catching expressions" could result in a clearer-still idiom: ? ? (obj.chain.of.attributes except AttributeError: default) ... but I suppose that ship has sailed. ?-[]z. From benhoyt at gmail.com Wed Nov 2 14:30:49 2016 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 2 Nov 2016 14:30:49 -0400 Subject: [Python-ideas] Small improvements to the profile/cProfile API Message-ID: Hi folks, Every time I do some Python profiling (with cProfile) the API feels kinda baroque, and I have to write ~10 line helper functions to do what I want. For example, if I want the return value of running some function and the profiling output as a string (e.g., to send as part of a web response), I have to do something like this: import cProfile import io import pstats def profile(func, *args, **kwargs): profiler = cProfile.Profile() result = profiler.runcall(func, *args, **kwargs) stream = io.StringIO() stats = pstats.Stats(profiler, stream=stream) stats.sort_stats('cumulative') stats.print_stats(strip_dirs=True) return (result, stream.getvalue()) Something like that might be a useful function in its own right, but I took a look at the current API, and also an open issue that addresses some of this (https://bugs.python.org/issue9285 last touched almost 4 years ago), and came up with the following: 1) Add a format_stats() function to Profile to give the profiling results as a string (kind of parallel to format_* vs print_* functions in the "traceback" module). Signature would format_stats(self, *restrictions, sort='stdname', strip_dirs=False). 2) Add __enter__ and __exit__ to Profile so you can use it in a "with" statement. 3) Add a top-level runcall() function to the cProfile (and profile) modules. This isn't particularly useful for me, but it'd make the module easier to use from the command line, and it's one of the API improvements over at issue 9285. Other things in issue 9285 that I don't think are a good idea: the separate runblock() context manager (there should be only one way to do it, and I think "with Profile()" is great), and the @profile decorator (I really don't see the use of this -- you don't always want to profile a function when calling it, only it certain contexts). My code implementing the above three things as a separate module (subclassing Profile) is copied below. I think the above additions are non-controversial improvements -- do you think I should make a patch to get this into the standard library? New issue or add it to 9285? Another feature that I wanted and would be useful for a lot of folks, I think, would be some way to fetch the results as proper Python objects, rather than as a file/string. Maybe a Profile.get_results() function that returns a ProfileResult namedtuple which has total time and number of calls etc, as well as a list of ProfileEntry namedtuples that have the data for each function. Thoughts on that (before any bike-shedding on the exact API)? Currently folks who need this data have to use undocumented parts of Profile like .stats and .fcn_list, for example the Pyramid debug toolbar extracts this data so it can render it in an HTML web page: https://github.com/Pylons/pyramid_debugtoolbar/blob/ed406d7f3c8581458c2e7bdf25e11e9ee8e3d489/pyramid_debugtoolbar/panels/performance.py#L93 Thanks! Ben # BELOW is my profileutils.py module """Code profiling utilities. >>> def foo(n): ... return sum(sum(range(i)) for i in range(n)) >>> with Profile() as profiler: ... result = foo(5) >>> print(profiler.format_stats(3, sort='nfl')) # doctest: +ELLIPSIS 15 function calls (10 primitive calls) in 0.000 seconds Ordered by: name/file/line List reduced from 5 to 3 due to restriction <3> ncalls tottime percall cumtime percall filename:lineno(function) 6/1 0.000 0.000 0.000 0.000 {built-in method builtins.sum} 6 0.000 0.000 0.000 0.000 <...>:2() 1 0.000 0.000 0.000 0.000 {method 'disable' ...} >>> result 10 >>> result = runcall(foo, 10) # doctest: +ELLIPSIS 24 function calls (14 primitive calls) in 0.000 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 <...>:1(foo) 11 0.000 0.000 0.000 0.000 <...>:2() 11/1 0.000 0.000 0.000 0.000 {built-in method builtins.sum} 1 0.000 0.000 0.000 0.000 {method 'disable' ...} >>> result 120 """ import cProfile import io import pstats VALID_SORTS = [ 'calls', 'cumtime', 'cumulative', 'file', 'filename', 'line', 'module', 'name', 'ncalls', 'nfl', 'pcalls', 'stdname', 'time', 'tottime', ] class Profile(cProfile.Profile): def format_stats(self, *restrictions, sort='stdname', strip_dirs=False): """Format stats report but return as string instead of printing to stdout. "restrictions" are as per Stats.print_stats(). Entries are sorted as per Stats.sort_stats(sort), and directories are stripped from filenames if strip_dirs is True. """ sort_keys = (sort,) if isinstance(sort, (int, str)) else sort stream = io.StringIO() stats = pstats.Stats(self, stream=stream) stats.sort_stats(*sort_keys) if strip_dirs: stats.strip_dirs() stats.print_stats(*restrictions) return stream.getvalue() # Define __enter__ and __exit__ to enable "with statement" support def __enter__(self): self.enable() return self def __exit__(self, exc_type, exc_value, exc_traceback): self.disable() def runcall(func, *args, **kwargs): """Profile running func(*args, **kwargs) and print stats to stdout.""" profiler = cProfile.Profile() result = profiler.runcall(func, *args, **kwargs) profiler.print_stats() return result if __name__ == '__main__': import doctest doctest.testmod() -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Wed Nov 2 14:34:14 2016 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 2 Nov 2016 18:34:14 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 2016-11-02 16:17, Nick Coghlan wrote: [snip] > Yeah, and so far the protocol based alternative I'm working on hasn't > been any less headache-inducing (Mark has been reviewing some early > iterations and had to draw a diagram to try to follow the proposed > control flow). > > I think I have a way to simplify that idea further though, and if that > works out I should have something I'm willing to share with a wider > audience. > > The gist is that rather than writing the bare: > > target = expr1 ?? expr2 ?? expr3 > > You'd instead write None-coalescing as: > > target = exists(expr1) ?? exists(expr2) ?? expr3 > > and None-propagating as: > > target = missing(expr1) ?? missing(expr2) ?? expr3 > > with ?? being a protocol-driven short-circuiting binary operator > controlled by the left operand rather than defining any particular > semantics of its own. > > The "obj?." and "obj?[]" would then be shorthand for particular uses > of "missing(obj) ?? ..." that avoid duplicate evaluation of the left > operand (as well as bypassing the overhead of actually creating a > "missing" instance). > How about borrowing from C: target = expr1 || expr2 || expr3 target = expr1 && expr2 && expr3 except that only None would be considered falsey? Or would that be confusing? From danilo.bellini at gmail.com Wed Nov 2 14:43:21 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Wed, 2 Nov 2016 16:43:21 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22542.49891.835655.410788@turnbull.sk.tsukuba.ac.jp> <22543.26081.957166.331323@turnbull.sk.tsukuba.ac.jp> Message-ID: 2016-10-25 12:29 GMT-02:00 Paul Moore : > On the other hand, the *ideas* are really interesting and valuable. > I'm certainly planning on looking at PyScanPrev when I get the chance. > And the discussions can frequently make people rethink their beliefs. > > So people posting ideas here should expect pushback - and should be > prepared to learn how to think about the wider context in which > changes to Python need to exist. That pushback won't[1] be hostile or > negative, although it can feel that way to newcomers. But if a poster > is inclined to take challenges to their idea personally, and even more > so if they respond negatively, things can get tense. So please don't > :-) > > So, bringing this back on topic - Danilo, what is your justification > for suggesting that this technique should be language syntax, as > opposed to simply being a 3rd party module (which you've already > written, which is great)? Do you know what sorts of things would be > viewed as evidence in favour of promoting this to syntax, or can we > help in clarifying the sort of evidence you'd need to collect? Are the > relevant design guidelines (things like "there should be one obvious > way to do it" that frequently get quoted around here without much > explanation) clear to you, or do you have questions? > > Hopefully we can change your mind about how inviting you find us :-) > Thanks for your message, Paul. When I said the list isn't inviting, I'm not talking about everyone, but about quite few people that disturbs. I'm just trying to help improving the language. I'm waiting for your opinion about PyScanPrev. =) 2016-10-25 16:01 GMT-02:00 Brendan Barnwell : > Recurrence relations are much more general than just "have access to the > previous value". They may have access to any of the earlier values, and/or > multiple earlier values. So if what we wanted was to able to use > recurrence relations, your proposal would be insufficient. Brendan, please see the PyScanPrev examples, mainly the Fibonacci and the State-space model examples. Recursion is enough to give you that. The proposal isn't about lag and windowing, but if you've got an idea to improve that, I'd like to know. 2016-10-25 15:55 GMT-02:00 Rob Cliffe : > On 24/10/2016 06:11, Danilo J. S. Bellini wrote: > >> >>> [prev * k for k in [5, 2, 4, 3] from prev = 1] >> [1, 5, 10, 40, 120] >> >> That makes sense for me, and seem simpler than: >> >> >>> from itertools import accumulate, chain >> >>> list(accumulate(chain([1], [5, 2, 4, 3]), lambda prev, k: prev * k)) >> [1, 5, 10, 40, 120] >> > Well, if you want an instant reaction from someone skimming this thread: I > looked at the first example and couldn't understand it. Then I looked at > the second one, and could understand it (even though I may never have used > "chain" or heard of "accumulate"). Obviously your mileage varies. > Rob, would it be the same if it was the other way around? I'm also somewhat familiar with map/filter, but I think you would need more time (and documentation) to understand what the second does if you hadn't seen the first. What do you think? 2016-10-25 23:36 GMT-02:00 David Mertz : > After reading every post in the thread, I still don't understand the > proposed new syntax really. > The proposal is to create a new syntax for recursive set/list/dict comprehensions and generator expressions. And that's all. It could be a new magic keyword as you said, but there are alternatives like the "from" example I gave. 2016-10-25 23:36 GMT-02:00 David Mertz : > If you give up a fear of using `import` and stop arbitrarily converting a > possibly infinite iterator to a concrete list > Ad hominem, and that's just offensive. Who said I "fear" an import? Who is "converting a possibly infinite iterator to a concrete list"?! You know that "it was a built-in, now it's in a module" is meaningful, and when it comes to what happened to "reduce", it's an argument for my proposal, not against it. Why should I fear? -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Wed Nov 2 15:02:47 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 2 Nov 2016 20:02:47 +0100 Subject: [Python-ideas] More user-friendly version for string.translate() In-Reply-To: References: <20161025023704.GD15983@ando.pearwood.info> <22543.37392.571089.528253@turnbull.sk.tsukuba.ac.jp> <22544.64697.310751.462520@turnbull.sk.tsukuba.ac.jp> Message-ID: On 27 October 2016 at 00:17, Chris Barker wrote: > 1) an easy way to spell "remove all the characters other than these" > > I think that's a good idea. What with unicode having an enormous number > of code points, it really does make sense to have a way to specify > only what you >want, rather than what you don't want. > > Back in the good old days of 1-byte chars, it wasn't hard to build up > a full 256 element translate table -- not so much anymore. > And one of the whole points of str.translate() is good performance. Actually even with ASCII (read for python 2.7) I would also be happy to have such function: say I just want to keep only digits so I write: digits = "0123456789" newstring = somestring.keep(digits) Despite I can do it other way, this would be much simpler and clearer way to do it. And I suppose it is quite common task not only for me. Currently 99% of my programs are in python 2.7. And I started to use python 3 only for tasks when I want to process unicode strings (ironically only to get rid of unicode). Mikhail From Nikolaus at rath.org Wed Nov 2 15:32:50 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 02 Nov 2016 12:32:50 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1478111438.1838.88.camel@gmail.com> (Zero Piraeus's message of "Wed, 02 Nov 2016 15:30:38 -0300") References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> Message-ID: <87shr9r82l.fsf@thinkpad.rath.org> On Nov 02 2016, Zero Piraeus wrote: > On Wed, 2016-11-02 at 08:46 -0700, Guido van Rossum wrote: >> [...] we need to agree on what even the right definition of ?. is. It's >> been frighteningly difficult to explain this even on this list, even >> though I have a very clear view in my head, and PEP 505 also has the >> same semantics and explains well why those are the right semantics. > > I think the proposed semantics for ?. are confusing (and the operator > itself dangerous, but more on that later). > > If I write something like obj.attr, the failure mode I care about is that > obj has no attribute attr, rather than that obj is specifically None (or > one of a defined group of somewhat Nonelike objects). > > Clearly, in such a circumstance, obj is not what I expected it to be, > because I thought it was going to have an attribute attr, and it > doesn't.? That means that you do not need null coalescing operators. They're not intended for your use-case, and you are not the target audience. Criticizing the proposal on this basis is like critizing it for not helping with asynchronous I/O. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From mikhailwas at gmail.com Wed Nov 2 15:38:42 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 2 Nov 2016 20:38:42 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 2 November 2016 at 19:34, MRAB wrote: > How about borrowing from C: > > target = expr1 || expr2 || expr3 > target = expr1 && expr2 && expr3 > > except that only None would be considered falsey? > > Or would that be confusing? Sorry for intruding into discussion and off-topic again, Such things like result = a > b ? x : y is IMHO a syntactical herecy. Such things disgust me from programming. Why on earth one cannot just wrap it in function if (a > b) { return x; } else { return y; } and write: c = nicefunc(a,b) Which will be WAY more readable and compact, if one wants to be compact for some reason. Mikhail From ethan at stoneleaf.us Wed Nov 2 16:21:49 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 02 Nov 2016 13:21:49 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <87shr9r82l.fsf@thinkpad.rath.org> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> <87shr9r82l.fsf@thinkpad.rath.org> Message-ID: <581A4ADD.9060606@stoneleaf.us> On 11/02/2016 12:32 PM, Nikolaus Rath wrote: > On Nov 02 2016, Zero Piraeus wrote: >> On Wed, 2016-11-02 at 08:46 -0700, Guido van Rossum wrote: >>> [...] we need to agree on what even the right definition of ?. is. It's >>> been frighteningly difficult to explain this even on this list, even >>> though I have a very clear view in my head, and PEP 505 also has the >>> same semantics and explains well why those are the right semantics. >> >> I think the proposed semantics for ?. are confusing (and the operator >> itself dangerous, but more on that later). >> >> If I write something like obj.attr, the failure mode I care about is that >> obj has no attribute attr, rather than that obj is specifically None (or >> one of a defined group of somewhat Nonelike objects). >> >> Clearly, in such a circumstance, obj is not what I expected it to be, >> because I thought it was going to have an attribute attr, and it >> doesn't. > > That means that you do not need null coalescing operators. They're not > intended for your use-case, and you are not the target audience. However, it's definitely a point for the confusiness of it all. I like the idea of null-(coalescing|negation|borrowing|what-have-you), but I also very much appreciate the keyword approach used in Python. I can already feel the headache coming on from trying to read and decipher the various ? accesses... -- ~Ethan~ From mehaase at gmail.com Wed Nov 2 16:40:57 2016 From: mehaase at gmail.com (Mark E. Haase) Date: Wed, 2 Nov 2016 16:40:57 -0400 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1478111438.1838.88.camel@gmail.com> References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> Message-ID: Zero Piraeus writes: > If I write something like obj.attr, the failure mode I care about is that > obj has no attribute attr, rather than that obj is specifically None (or > one of a defined group of somewhat Nonelike objects). > > Clearly, in such a circumstance, obj is not what I expected it to be, > because I thought it was going to have an attribute attr, and it > doesn't. > That's what you care about because you're working with a data model where obj should not be None, and so you want an AttributeError to be raised when your assumption is violated. That's fine. But other people have other needs. PEP-505 includes a script that detects examples of null-coalescing and safe navigation syntax so that we can run it on real projects. Rather than debating if None should be used in this way, I prefer to measure how frequently None is already being used this way. I'm sure it's obvious by now that I think the intuitive behaviour of any > ?. operator is to suppress AttributeError on the right hand side, rather > than check for nullity on the left. > I can't emphasize strongly enough (and the same is said in PEP-505): this is not an error handling syntax. It does not suppress any exceptions. It is a syntax for handling common patterns with None. These are patterns that demonstrably exist in real Python code, including the standard library. Rewriting that code to use this new syntax would not change the safety, semantics, or error handling of that code. > However, I do think an alternative is possible: extend getattr(). > > getattr(obj, ('chain', 'of', 'attributes'), default) > > This is more verbose than an operator, but I think its intent is clearer, > and its verbosity makes it less likely to be misused. > How do your proposed built-in handle this case? >>> my_date?.replace(microsecond=0).isoformat() Incidentally, Chris Angelico's PEP 463 "Exception-catching expressions" > could result in a clearer-still idiom: > > (obj.chain.of.attributes except AttributeError: default) > > ... but I suppose that ship has sailed. > That doesn't have the same semantics as obj?.chain.of.attributes. If you haven't read the PEP, all of these issues are covered in great detail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Nov 2 16:50:14 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 2 Nov 2016 13:50:14 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: Even though I really don't want new null-coalescing operators, I really appreciate the ternary operator in Python (or in C). On Wed, Nov 2, 2016 at 12:38 PM, Mikhail V wrote: > result = a > b ? x : y > > is IMHO a syntactical herecy. Such things disgust me from programming. > Why on earth one cannot just wrap it in function > c = nicefunc(a,b) > The problem here is that the general form isn't ONLY to return 'x' or 'y' but the decide between arbitrary values. Hard-coding the variables into the function loses 90%+ of the point. So the general function would need a signature like: c = nicefunc(a, b, x, y) The problem here is that this call might be: c = nicefunc(a, b, run_for_hours(), has_side_effects()) We only want ONE of 'x' and 'y' to eagerly evaluate. In the C or Python ternary we get exactly that. Obviously, that also happens in your fully spelled out if/else block too, but that's multiline and needs to setup variables not just be used as an expression. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.mitchell at leapfrog3d.com Wed Nov 2 16:52:16 2016 From: tim.mitchell at leapfrog3d.com (Tim Mitchell) Date: Thu, 3 Nov 2016 09:52:16 +1300 Subject: [Python-ideas] Small improvements to the profile/cProfile API In-Reply-To: References: Message-ID: I use an @profile() decorator for almost all my profiling. If you want to profile function foo you just decorate it and re-run the program. With a with block you have to find the places where foo is called and put with statements around the calls. I think both approaches are equally valid and useful. On 3 November 2016 at 07:30, Ben Hoyt wrote: > Hi folks, > > Every time I do some Python profiling (with cProfile) the API feels kinda > baroque, and I have to write ~10 line helper functions to do what I want. > For example, if I want the return value of running some function and the > profiling output as a string (e.g., to send as part of a web response), I > have to do something like this: > > import cProfile > import io > import pstats > > def profile(func, *args, **kwargs): > profiler = cProfile.Profile() > result = profiler.runcall(func, *args, **kwargs) > stream = io.StringIO() > stats = pstats.Stats(profiler, stream=stream) > stats.sort_stats('cumulative') > stats.print_stats(strip_dirs=True) > return (result, stream.getvalue()) > > Something like that might be a useful function in its own right, but I > took a look at the current API, and also an open issue that addresses some > of this (https://bugs.python.org/issue9285 last touched almost 4 years > ago), and came up with the following: > > 1) Add a format_stats() function to Profile to give the profiling results > as a string (kind of parallel to format_* vs print_* functions in the > "traceback" module). Signature would format_stats(self, *restrictions, > sort='stdname', strip_dirs=False). > > 2) Add __enter__ and __exit__ to Profile so you can use it in a "with" > statement. > > 3) Add a top-level runcall() function to the cProfile (and profile) > modules. This isn't particularly useful for me, but it'd make the module > easier to use from the command line, and it's one of the API improvements > over at issue 9285. > > Other things in issue 9285 that I don't think are a good idea: the > separate runblock() context manager (there should be only one way to do it, > and I think "with Profile()" is great), and the @profile decorator (I > really don't see the use of this -- you don't always want to profile a > function when calling it, only it certain contexts). > > My code implementing the above three things as a separate module > (subclassing Profile) is copied below. > > I think the above additions are non-controversial improvements -- do you > think I should make a patch to get this into the standard library? New > issue or add it to 9285? > > Another feature that I wanted and would be useful for a lot of folks, I > think, would be some way to fetch the results as proper Python objects, > rather than as a file/string. Maybe a Profile.get_results() function that > returns a ProfileResult namedtuple which has total time and number of calls > etc, as well as a list of ProfileEntry namedtuples that have the data for > each function. Thoughts on that (before any bike-shedding on the exact API)? > > Currently folks who need this data have to use undocumented parts of > Profile like .stats and .fcn_list, for example the Pyramid debug toolbar > extracts this data so it can render it in an HTML web page: > https://github.com/Pylons/pyramid_debugtoolbar/blob/ > ed406d7f3c8581458c2e7bdf25e11e9ee8e3d489/pyramid_debugtoolbar/panels/ > performance.py#L93 > > Thanks! > Ben > > # BELOW is my profileutils.py module > > """Code profiling utilities. > > >>> def foo(n): > ... return sum(sum(range(i)) for i in range(n)) > > >>> with Profile() as profiler: > ... result = foo(5) > >>> print(profiler.format_stats(3, sort='nfl')) # doctest: +ELLIPSIS > 15 function calls (10 primitive calls) in 0.000 seconds > > Ordered by: name/file/line > List reduced from 5 to 3 due to restriction <3> > > ncalls tottime percall cumtime percall filename:lineno(function) > 6/1 0.000 0.000 0.000 0.000 {built-in method > builtins.sum} > 6 0.000 0.000 0.000 0.000 <...>:2() > 1 0.000 0.000 0.000 0.000 {method 'disable' ...} > > > > >>> result > 10 > > >>> result = runcall(foo, 10) # doctest: +ELLIPSIS > 24 function calls (14 primitive calls) in 0.000 seconds > > Ordered by: standard name > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.000 0.000 0.000 0.000 <...>:1(foo) > 11 0.000 0.000 0.000 0.000 <...>:2() > 11/1 0.000 0.000 0.000 0.000 {built-in method > builtins.sum} > 1 0.000 0.000 0.000 0.000 {method 'disable' ...} > > > >>> result > 120 > """ > > import cProfile > import io > import pstats > > > VALID_SORTS = [ > 'calls', 'cumtime', 'cumulative', 'file', 'filename', > 'line', 'module', 'name', 'ncalls', 'nfl', > 'pcalls', 'stdname', 'time', 'tottime', > ] > > > class Profile(cProfile.Profile): > def format_stats(self, *restrictions, sort='stdname', > strip_dirs=False): > """Format stats report but return as string instead of printing to > stdout. "restrictions" are as per Stats.print_stats(). Entries are > sorted as per Stats.sort_stats(sort), and directories are stripped > from filenames if strip_dirs is True. > """ > sort_keys = (sort,) if isinstance(sort, (int, str)) else sort > stream = io.StringIO() > stats = pstats.Stats(self, stream=stream) > stats.sort_stats(*sort_keys) > if strip_dirs: > stats.strip_dirs() > stats.print_stats(*restrictions) > return stream.getvalue() > > # Define __enter__ and __exit__ to enable "with statement" support > > def __enter__(self): > self.enable() > return self > > def __exit__(self, exc_type, exc_value, exc_traceback): > self.disable() > > > def runcall(func, *args, **kwargs): > """Profile running func(*args, **kwargs) and print stats to stdout.""" > profiler = cProfile.Profile() > result = profiler.runcall(func, *args, **kwargs) > profiler.print_stats() > return result > > > if __name__ == '__main__': > import doctest > doctest.testmod() > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- *Tim Mitchell* Senior Software Developer *phone:* +64 3 961 1031 ext. 217 *email:* tim.mitchell at leapfrog3d.com *skype: *tim.mitchell.leapfrog3d *address:* 41 Leslie Hills Drive, Riccarton, Christchurch, 8011, New Zealand www.leapfrog3d.com An ARANZ Geo Limited Product ------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Wed Nov 2 16:58:09 2016 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 2 Nov 2016 16:58:09 -0400 Subject: [Python-ideas] Small improvements to the profile/cProfile API In-Reply-To: References: Message-ID: Okay, got it, that sounds fair enough. With your @profile decorator how do you tell it when and where to print the output? Can you post the source for your decorator? On Wed, Nov 2, 2016 at 4:52 PM, Tim Mitchell wrote: > I use an @profile() decorator for almost all my profiling. If you want to > profile function foo you just decorate it and re-run the program. > With a with block you have to find the places where foo is called and put > with statements around the calls. > I think both approaches are equally valid and useful. > > On 3 November 2016 at 07:30, Ben Hoyt wrote: > >> Hi folks, >> >> Every time I do some Python profiling (with cProfile) the API feels kinda >> baroque, and I have to write ~10 line helper functions to do what I want. >> For example, if I want the return value of running some function and the >> profiling output as a string (e.g., to send as part of a web response), I >> have to do something like this: >> >> import cProfile >> import io >> import pstats >> >> def profile(func, *args, **kwargs): >> profiler = cProfile.Profile() >> result = profiler.runcall(func, *args, **kwargs) >> stream = io.StringIO() >> stats = pstats.Stats(profiler, stream=stream) >> stats.sort_stats('cumulative') >> stats.print_stats(strip_dirs=True) >> return (result, stream.getvalue()) >> >> Something like that might be a useful function in its own right, but I >> took a look at the current API, and also an open issue that addresses some >> of this (https://bugs.python.org/issue9285 last touched almost 4 years >> ago), and came up with the following: >> >> 1) Add a format_stats() function to Profile to give the profiling results >> as a string (kind of parallel to format_* vs print_* functions in the >> "traceback" module). Signature would format_stats(self, *restrictions, >> sort='stdname', strip_dirs=False). >> >> 2) Add __enter__ and __exit__ to Profile so you can use it in a "with" >> statement. >> >> 3) Add a top-level runcall() function to the cProfile (and profile) >> modules. This isn't particularly useful for me, but it'd make the module >> easier to use from the command line, and it's one of the API improvements >> over at issue 9285. >> >> Other things in issue 9285 that I don't think are a good idea: the >> separate runblock() context manager (there should be only one way to do it, >> and I think "with Profile()" is great), and the @profile decorator (I >> really don't see the use of this -- you don't always want to profile a >> function when calling it, only it certain contexts). >> >> My code implementing the above three things as a separate module >> (subclassing Profile) is copied below. >> >> I think the above additions are non-controversial improvements -- do you >> think I should make a patch to get this into the standard library? New >> issue or add it to 9285? >> >> Another feature that I wanted and would be useful for a lot of folks, I >> think, would be some way to fetch the results as proper Python objects, >> rather than as a file/string. Maybe a Profile.get_results() function that >> returns a ProfileResult namedtuple which has total time and number of calls >> etc, as well as a list of ProfileEntry namedtuples that have the data for >> each function. Thoughts on that (before any bike-shedding on the exact API)? >> >> Currently folks who need this data have to use undocumented parts of >> Profile like .stats and .fcn_list, for example the Pyramid debug toolbar >> extracts this data so it can render it in an HTML web page: >> https://github.com/Pylons/pyramid_debugtoolbar/blob/ed406d7f >> 3c8581458c2e7bdf25e11e9ee8e3d489/pyramid_debugtoolbar/ >> panels/performance.py#L93 >> >> Thanks! >> Ben >> >> # BELOW is my profileutils.py module >> >> """Code profiling utilities. >> >> >>> def foo(n): >> ... return sum(sum(range(i)) for i in range(n)) >> >> >>> with Profile() as profiler: >> ... result = foo(5) >> >>> print(profiler.format_stats(3, sort='nfl')) # doctest: +ELLIPSIS >> 15 function calls (10 primitive calls) in 0.000 seconds >> >> Ordered by: name/file/line >> List reduced from 5 to 3 due to restriction <3> >> >> ncalls tottime percall cumtime percall filename:lineno(function) >> 6/1 0.000 0.000 0.000 0.000 {built-in method >> builtins.sum} >> 6 0.000 0.000 0.000 0.000 <...>:2() >> 1 0.000 0.000 0.000 0.000 {method 'disable' ...} >> >> >> >> >>> result >> 10 >> >> >>> result = runcall(foo, 10) # doctest: +ELLIPSIS >> 24 function calls (14 primitive calls) in 0.000 seconds >> >> Ordered by: standard name >> >> ncalls tottime percall cumtime percall filename:lineno(function) >> 1 0.000 0.000 0.000 0.000 <...>:1(foo) >> 11 0.000 0.000 0.000 0.000 <...>:2() >> 11/1 0.000 0.000 0.000 0.000 {built-in method >> builtins.sum} >> 1 0.000 0.000 0.000 0.000 {method 'disable' ...} >> >> >> >>> result >> 120 >> """ >> >> import cProfile >> import io >> import pstats >> >> >> VALID_SORTS = [ >> 'calls', 'cumtime', 'cumulative', 'file', 'filename', >> 'line', 'module', 'name', 'ncalls', 'nfl', >> 'pcalls', 'stdname', 'time', 'tottime', >> ] >> >> >> class Profile(cProfile.Profile): >> def format_stats(self, *restrictions, sort='stdname', >> strip_dirs=False): >> """Format stats report but return as string instead of printing to >> stdout. "restrictions" are as per Stats.print_stats(). Entries are >> sorted as per Stats.sort_stats(sort), and directories are stripped >> from filenames if strip_dirs is True. >> """ >> sort_keys = (sort,) if isinstance(sort, (int, str)) else sort >> stream = io.StringIO() >> stats = pstats.Stats(self, stream=stream) >> stats.sort_stats(*sort_keys) >> if strip_dirs: >> stats.strip_dirs() >> stats.print_stats(*restrictions) >> return stream.getvalue() >> >> # Define __enter__ and __exit__ to enable "with statement" support >> >> def __enter__(self): >> self.enable() >> return self >> >> def __exit__(self, exc_type, exc_value, exc_traceback): >> self.disable() >> >> >> def runcall(func, *args, **kwargs): >> """Profile running func(*args, **kwargs) and print stats to stdout.""" >> profiler = cProfile.Profile() >> result = profiler.runcall(func, *args, **kwargs) >> profiler.print_stats() >> return result >> >> >> if __name__ == '__main__': >> import doctest >> doctest.testmod() >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > > *Tim Mitchell* > Senior Software Developer > > > > *phone:* +64 3 961 1031 ext. 217 > *email:* tim.mitchell at leapfrog3d.com > *skype: *tim.mitchell.leapfrog3d > *address:* 41 Leslie Hills Drive, Riccarton, Christchurch, 8011, New > Zealand > www.leapfrog3d.com > > An ARANZ Geo Limited Product > > > > ------------------------------ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.mitchell at leapfrog3d.com Wed Nov 2 17:45:40 2016 From: tim.mitchell at leapfrog3d.com (Tim Mitchell) Date: Thu, 3 Nov 2016 10:45:40 +1300 Subject: [Python-ideas] Small improvements to the profile/cProfile API In-Reply-To: References: Message-ID: Hi Ben, Mostly I just print to stdout, I imagine more flexibility would be needed in general. This is for python 2.7 - don't know if it works for 3. def profile(sort='time', restriction=(), callers=None, callees=None, filename=None): def _profileDecorator(func): "print profile stats for decorated function" def wrapper(*args, **kwargs): print 'Profile for:', func.__name__ prof = cProfile.Profile() result = prof.runcall(func, *args, **kwargs) _, statsFileName = tempfile.mkstemp() prof.dump_stats(statsFileName) if filename is None: stats = pstats.Stats(statsFileName) else: stats = pstats.Stats(statsFileName, stream=open(filename, 'w')) if isinstance(sort, basestring): stats.sort_stats(sort) else: stats.sort_stats(*sort) if isinstance(restriction, (tuple, list)): stats.print_stats(*restriction) else: stats.print_stats(restriction) if callers is not None: if isinstance(callers, basestring): stats.print_callers(callers) else: stats.print_callers(*callers) if callees is not None: if isinstance(callees, basestring): stats.print_callees(callees) else: stats.print_callees(*callees) return result return wrapper return _profileDecorator Cheers Tim On 3 November 2016 at 09:58, Ben Hoyt wrote: > Okay, got it, that sounds fair enough. With your @profile decorator how do > you tell it when and where to print the output? Can you post the source for > your decorator? > > On Wed, Nov 2, 2016 at 4:52 PM, Tim Mitchell > wrote: > >> I use an @profile() decorator for almost all my profiling. If you want >> to profile function foo you just decorate it and re-run the program. >> With a with block you have to find the places where foo is called and put >> with statements around the calls. >> I think both approaches are equally valid and useful. >> >>> conduct/ >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Nov 2 17:45:17 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 2 Nov 2016 14:45:17 -0700 Subject: [Python-ideas] More user-friendly version for string.translate() In-Reply-To: References: <20161025023704.GD15983@ando.pearwood.info> <22543.37392.571089.528253@turnbull.sk.tsukuba.ac.jp> <22544.64697.310751.462520@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Nov 2, 2016 at 12:02 PM, Mikhail V wrote: > Actually even with ASCII (read for python 2.7) I would also be happy > to have such function: say I just want to keep only digits so I write: > > digits = "0123456789" > newstring = somestring.keep(digits) > well, with ascii, it's not too hard to make a translation table: digits = "0123456789" table = [(o if chr(o) in digits else None )for o in range(256)] s = "some stuff and some 456 23 numbers 888" s.translate(table) '45623888' but then there is the defaultdict way: s.translate(defaultdict(*lambda*: *None*, {ord(c):c *for* c *in* digits}.items())) '45623888' wasn't that easy? Granted, if you need to do this, you'd wrap it in a function like Chris A. Suggested. But this really isn't easy or discoverable -- it took me a fair bit of fidlding to get right, and I knew I was looking for a defaultdict implementation. Also: In [43]: table Out[43]: defaultdict(>, {48: '0', 49: '1', 50: '2', 51: '3', 52: '4', 53: '5', 54: '6', 55: '7', 56: '8', 57: '9'}) In [44]: s.translate(table) Out[44]: '45623888' In [45]: table Out[45]: defaultdict(>, {32: None, 48: '0', 49: '1', 50: '2', 51: '3', 52: '4', 53: '5', 54: '6', 55: '7', 56: '8', 57: '9', 97: None, 98: None, 100: None, 101: None, 102: None, 109: None, 110: None, 111: None, 114: None, 115: None, 116: None, 117: None}) defaultdict puts an entry in for every ordinal checked -- this could get big -- granted, probaly nt a big deal with modern computer memory, but still... it might even be worth making a NoneDict for this: class NoneDict(dict): """ Dictionary implementation that always returns None when a key is not in the dict, rather than raising a KeyError """ def __getitem__(self, key): try: val = dict.__getitem__(self, key) except KeyError: val = None return val (see enclosed -- it works fine with translate) (OK, that was fun, but no, not really that useful) Despite I can do it other way, this would be much simpler and clearer > way to do it. And I suppose it is quite common task not only for me. > That's the key question -- is this a common task? If so, then whie there are ways to do it, they're not easy nor discoverable. And while some of the guiding principles of this list are: "not every two line function needs to be in the standard lib" and "put it up on PYPi, and see if a lot of people find it useful" It's actually kind of silly to put a single function up as a PyPi package -- and I doubt many people will find it if you did. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: none_dict.py Type: text/x-python-script Size: 1310 bytes Desc: not available URL: From greg.ewing at canterbury.ac.nz Wed Nov 2 18:22:45 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Nov 2016 11:22:45 +1300 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> Message-ID: <581A6735.9040503@canterbury.ac.nz> Zero Piraeus writes: > > If I write something like obj.attr, the failure mode I care about is > that > obj has no attribute attr, rather than that obj is specifically None (or > one of a defined group of somewhat Nonelike objects). > > Clearly, in such a circumstance, obj is not what I expected it to be, > because I thought it was going to have an attribute attr, and it > doesn't. If it's an error, you shouldn't be trying to do anything about it, just let the exception happen. The proposed .? syntax is designed for cases where it's *not* an error for the object to be missing the attribute, *and* the correct action in that situation is to skip whatever you would have done otherwise. What needs to be decided is whether such use cases are frequent enough to justify special syntax. -- Greg From pavol.lisy at gmail.com Wed Nov 2 18:59:41 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Wed, 2 Nov 2016 23:59:41 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> Message-ID: On 10/15/16, Nick Coghlan wrote: > * None-coalescing operator: x ?or y > * None-severing operator: x ?and y > * None-coalescing augmented assignment: x ?= y > * None-severing attribute access: x?.attr > * None-severing subscript lookup: x?[expr] Please don't be too harsh to me for my next words! :) Just to trying to find something more cool (at least for young people :P ) I was thinking about emoticons... :( (means I am not happy because left is None) But parenthesis would be too confusing with "real" parenthesis. Example: (x :( (a + b)) # and editor would have problem to find corresponding pairs of parenthesis! So I am just trying to replace parenthesis with <> and playing with it * None-coalescing operator: x :< y * None-severing operator: x :> y * None-coalescing augmented assignment: x =< y * None-severing attribute access: x.>attr * None-severing subscript lookup: x[>expr] # ok this one seems a little weird to me too alternatively: x:>[expr] With this syntax/paradigm we could have also: * None-severing augmented assignment: x => y About spelling: we could say None-coalescing and None-severing operator in theory and sad and happy operator in real life. :) PL. From matt at getpattern.com Wed Nov 2 18:59:54 2016 From: matt at getpattern.com (Matt Gilson) Date: Wed, 2 Nov 2016 15:59:54 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <581A6735.9040503@canterbury.ac.nz> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> <581A6735.9040503@canterbury.ac.nz> Message-ID: I actually think that Zero's point here is quite valid... At some earlier point in the thread, I believe that Nick Coughlin was saying that we should be asking ourselves _why_ we want to do something like this and the result of that discussion was because there is pain when working with "pseudo-structured" responses from various APIs. This use-case resonates with me as I work with JSON responses quite frequently. The thing about "pseudo-structured" data is that there isn't an agreed upon way to represent it. While one API might send a field with a `null` value in some cases, another API might send a field with no data (after all, why bother putting it in the response if it's going to be `null`? You're just wasting bytes on the wire). As a concrete case where fields are truly missing -- Most of the Google APIs accept a "fields" parameter that allows you to pair down what is actually included in the response (If you don't believe me, feel free to play around with it in their API explorer -- https://developers.google.com/google-apps/calendar/v3/reference/calendars/get ). So, while this proposal is specifically about a "Null coalescing operator", my guess is that users will think of it as a "Get the field if it exists, else short-circuit and return `None`". And it might take a lot of education to try to get everyone aligned on the same page around what the "if it exists" actually means. At first, I was thinking that perhaps _adding_ an operator would help the community to standardize around "If the field is missing, then put a `None` in there -- But unfortunately, I think that the JSON API example demonstrates that a really big use-case for this operator is in working with APIs that are likely written in different languages whose communities frequently don't follow the same norms that python has embraced. I suppose that the suggestion would be to write: foo.get('bar')?['baz'] if the `foo` dict may or may not have ` bar` key? On Wed, Nov 2, 2016 at 3:22 PM, Greg Ewing wrote: > Zero Piraeus writes: > >> >> If I write something like obj.attr, the failure mode I care about is >> that >> obj has no attribute attr, rather than that obj is specifically None >> (or >> one of a defined group of somewhat Nonelike objects). >> >> Clearly, in such a circumstance, obj is not what I expected it to be, >> because I thought it was going to have an attribute attr, and it >> doesn't. >> > > If it's an error, you shouldn't be trying to do anything > about it, just let the exception happen. > > The proposed .? syntax is designed for cases where it's *not* > an error for the object to be missing the attribute, *and* > the correct action in that situation is to skip whatever > you would have done otherwise. > > What needs to be decided is whether such use cases are frequent > enough to justify special syntax. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt at getpattern.com // P: 603.892.7736 We?re looking for beta testers. Go here to sign up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavol.lisy at gmail.com Wed Nov 2 19:48:33 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Thu, 3 Nov 2016 00:48:33 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20160910002719.GG22471@ando.pearwood.info> <22484.14715.118314.556074@turnbull.sk.tsukuba.ac.jp> <1473527686.2634873.721679809.56D51639@webmail.messagingengine.com> <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 10/31/16, Guido van Rossum wrote: > For "everything to the right" it would seem we have some freedom: e.g. if > we have "foo.bar?.baz(bletch)" is the call included? The answer is yes -- > the concept we're after here is named "trailer" in the Grammar file in the > source code ( > https://github.com/python/cpython/blob/master/Grammar/Grammar#L119), and > "primary" in the reference manual ( > https://docs.python.org/3/reference/expressions.html#primaries). This means > all attribute references ("x.y"), index/slice operations ("x[...]"), and > calls ("x(...)"). > > Note that in almost all cases the "?." operator will be used in an context > where there is no other operator of lower precedence before or after it -- > given the above meaning, it doesn't make a lot of sense to write "1 + x?.a" > because "1 + None" is always an error (and ditto for "x?.a + 1"). However > it still makes sense to assign such an expression to a variable or pass it > as an argument to a function. > > So you can ignore the preceding four paragraphs: just remember the > simplified rule (indented and in bold, depending on your email client) and > let your intuition do the rest. Maybe it can even be simplified more: > > > *The "?." operator splits the expression in two parts; the second part is > skipped if the first part is None.* > > Eventually this *will* become intuitive. The various constraints are all > naturally imposed by the grammar so you won't have to think about them > consciously. > > --Guido If we skip function call then we also skip argument evaluation? def fnc(): print('I am here') None(fnc()) # behavior similar to this? None()[fnc()] # or to this? PL From greg.ewing at canterbury.ac.nz Wed Nov 2 17:57:57 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Nov 2016 10:57:57 +1300 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: <581A6165.1040809@canterbury.ac.nz> MRAB wrote: > target = expr1 || expr2 || expr3 > target = expr1 && expr2 && expr3 > > except that only None would be considered falsey? > > Or would that be confusing? Yes, I think that borrowing an operator from C but giving it subtly different semantics would be *very* confusing, especially to people more familiar with C than Python. They're going to look at it and *think* they know what it means, except they don't. -- Greg From steve at pearwood.info Wed Nov 2 20:19:54 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 Nov 2016 11:19:54 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: <20161103001954.GM3365@ando.pearwood.info> On Wed, Nov 02, 2016 at 08:46:54AM -0700, Guido van Rossum wrote: > But first we need to agree on what even the right definition > of ?. is. It's been frighteningly difficult to explain this even on this > list, even though I have a very clear view in my head, This is Python-Ideas and with respect to the many fine people on this list, the culture of the list does tend strongly towards over- generalisation which can lead to confusion: it may be hard to understand a proposal when multiple competing/alternative proposals are flying past all at once. That should not necessarily be taken as an argument against the feature itself: once details are locked down, and alternatives dismissed, the feature may not be so hard to understand, even for beginners. Here's my attempt at an explanation. spam?.eggs is simply syntactic sugar for: None if spam is None else spam.eggs It is not a generalised way of catching AttributeError. Nor is it a generalised way of detecting arbitrary empty or missing values. It is specific to None. "spam" can be an expression, not just a name, in which case it is only evaluated once: database.lookup[expensive+calculation]?.attribute only performs the expensive calculation and lookup once. It makes sense to allow ?. to bind "all the way to the right" so that: spam?.eggs.cheese.tomato.aardvark is sugar for: None if spam is None else spam.eggs.cheese.tomato.aardvark The ?[] operator extends this to item lookup: spam?[eggs] is sugar for: None if spam is None else spam[eggs] and the ?? "None-coalescing" operator extends this to general expressions: spam ?? eggs being sugar for: eggs if spam is None else spam Again, spam is only evaluated once. -- Steve From mikhailwas at gmail.com Wed Nov 2 20:33:40 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 3 Nov 2016 01:33:40 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On 2 November 2016 at 21:50, David Mertz wrote: > Even though I really don't want new null-coalescing operators, I really > appreciate the ternary operator in Python (or in C). > > On Wed, Nov 2, 2016 at 12:38 PM, Mikhail V wrote: >> >> result = a > b ? x : y >> >> is IMHO a syntactical herecy. Such things disgust me from programming. >> Why on earth one cannot just wrap it in function >> c = nicefunc(a,b) > > > The problem here is that the general form isn't ONLY to return 'x' or 'y' > but the decide between arbitrary values. Hard-coding the variables into the > function loses 90%+ of the point. > > So the general function would need a signature like: > > c = nicefunc(a, b, x, y) > > The problem here is that this call might be: > > c = nicefunc(a, b, run_for_hours(), has_side_effects()) > > We only want ONE of 'x' and 'y' to eagerly evaluate. In the C or Python > ternary we get exactly that. Obviously, that also happens in your fully > spelled out if/else block too, but that's multiline and needs to setup > variables not just be used as an expression. [Apologies for off-topic one more time] So you say "that's multiline" just as if a multiner is something bad. All evil comes from the wish to write things more compactly. Also what to evaluate or not evaluate lies solely on the compiler/interpreter so that is not what the user must think of. Anyway, I am too far from doing chain attribute selections, but for the above example, a one-liner: if (a > b) : run_for_hours() else has_side_effects() Is this different from above? For me it is way more readable and no need to learn/memorise new operators. Also, *anything* of less then 3 characters as an operator is most likely to be initially a fail, simply because it is hard to *see* it in a mess of code, unlike kewords and well formatted multiliner. Or for example to return non-null of multiple operands: value = not_a_function_but_selector ( a, b, c, d ) Cannot this principle be exposed to attribute selector? As said I am too far from these problematics, still I'll try, though it may be total nonsense I am telling here: with seek_chain(foo.bar.bee.buzz) : name = "got it" I mean do you really need to stick something in the middle of chain, why not leave the whole thing there and let the "seek_chain" do the job (whatever this must be, I don't have an idea) Mikhail From steve at pearwood.info Wed Nov 2 20:50:53 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 Nov 2016 11:50:53 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: <20161103005051.GN3365@ando.pearwood.info> On Thu, Nov 03, 2016 at 02:17:14AM +1000, Nick Coghlan wrote: > Yeah, and so far the protocol based alternative I'm working on hasn't > been any less headache-inducing (Mark has been reviewing some early > iterations and had to draw a diagram to try to follow the proposed > control flow). Even if your protocol idea pans out and is a good idea, it doesn't solve the use-cases that PEP 505 is intended to solve. PEP 505 is specifically for the cases where None *is* special, where you *don't* want "None or something like None", but specifically None and nothing else. Your protocol idea is not an alternative to 505, it is independent of 505: it wants a second generalised concept of "None or something else" (distinct from falsey values), while 505 is specifically about "None and nothing else". -- Steve From python at mrabarnett.plus.com Wed Nov 2 21:06:12 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 3 Nov 2016 01:06:12 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <581A6165.1040809@canterbury.ac.nz> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <581A6165.1040809@canterbury.ac.nz> Message-ID: <61e319c2-8af5-40eb-03ac-190e1d51fb1e@mrabarnett.plus.com> On 2016-11-02 21:57, Greg Ewing wrote: > MRAB wrote: >> target = expr1 || expr2 || expr3 >> target = expr1 && expr2 && expr3 >> >> except that only None would be considered falsey? >> >> Or would that be confusing? > > Yes, I think that borrowing an operator from C but giving > it subtly different semantics would be *very* confusing, > especially to people more familiar with C than Python. > They're going to look at it and *think* they know what it > means, except they don't. > OK, if we're going to have ?. and ?[, then: target = expr1 ?| expr2 ?| expr3 target = expr1 ?& expr2 ?& expr3 A disadvantage is that they look a little like | and &, which don't short-circuit. From steve at pearwood.info Wed Nov 2 21:10:46 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 Nov 2016 12:10:46 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <581A6735.9040503@canterbury.ac.nz> References: <1478111438.1838.88.camel@gmail.com> <581A6735.9040503@canterbury.ac.nz> Message-ID: <20161103011046.GO3365@ando.pearwood.info> On Thu, Nov 03, 2016 at 11:22:45AM +1300, Greg Ewing wrote: > The proposed .? syntax is designed for cases where it's *not* > an error for the object to be missing the attribute, No it is not. That is absolutely not what the syntax means. I'm sorry to single you out Greg, but have you read the PEP? Or even the *title* of the PEP? Or even the subject line of this email thread??? Okay maybe the subject line of this thread isn't too clear (what's Null?). But the PEP is absolutely clear that this is specifically for dealing with None, not arbitrary missing attributes. If you write: obj = [] # oops I meant {} obj?.update(mapping) you will still get an AttributeError, because lists don't have an update method. > *and* > the correct action in that situation is to skip whatever > you would have done otherwise. > > What needs to be decided is whether such use cases are frequent > enough to justify special syntax. I think that the PEP does a good job of demonstrating that use-cases for these None-aware operators are common. -- Steve From klahnakoski at mozilla.com Wed Nov 2 22:22:56 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 2 Nov 2016 22:22:56 -0400 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <1478111438.1838.88.camel@gmail.com> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> Message-ID: <6a17a21a-bd59-c804-82e0-43b697d1d304@mozilla.com> On 11/2/2016 2:30 PM, Zero Piraeus wrote: > > If I write something like obj.attr, the failure mode I care about is that > obj has no attribute attr, rather than that obj is specifically None (or > one of a defined group of somewhat Nonelike objects). > I agree with this understanding. The problem with None-coalescing is it doesn't consider where None came from. I suspect enumerating the source of None values will reveal the majority of them are a result of `getattr(obj, attr)` returning None (or obj.get(attr) returning None). If your code deals with annotated objects, rather than strictly typed objects, you will have many instances of attributes resolving to None. Making specific classes for each of the annotations, or combinations of annotations, is prohibitive, so you don't. Rather, you use a few general types and set some attributes to None to indicate a property is not-relevant-for-this-object. None checks are type checks. They are late type checks . From pavel.velikhov at gmail.com Wed Nov 2 22:43:57 2016 From: pavel.velikhov at gmail.com (Pavel Velikhov) Date: Thu, 3 Nov 2016 08:43:57 +0600 Subject: [Python-ideas] Query Language extension to Python In-Reply-To: <684fa192-bf32-4196-ba62-db4efe988d80@googlegroups.com> References: <4233391F-9872-455B-8B29-CF7EFC60D1C6@gmail.com> <684fa192-bf32-4196-ba62-db4efe988d80@googlegroups.com> Message-ID: <99CD9B77-F489-428D-952E-B68C1ED8D1D0@gmail.com> > 3 ????. 2016 ?., ? 0:28, Denis Akhiyarov ???????(?): > > How does this compare to LINQ in .NET? Denis, PythonQL is very similar to LINQ actually. The syntax is a little different - more Pythonic vs SQL, the clauses can be composed arbitrarily, we have named tuples, that make nested queries easier to write and a few minor bells and whistles. > >> On Tuesday, November 1, 2016 at 2:03:44 PM UTC-5, Pavel Velikhov wrote: >> Hi David! >> >> I haven?t used blaze, but its looks quite similar to pandas, at least conceptually. Thanks for >> the reference! >> >> The big difference with PythonQL is that we actually extend the syntax of Python with a few >> constructs that are typically used in query languages (group by, order by, window, let clause). >> The language extension is quite small and easy to grasp, but its very powerful: you can use >> this language to easily formulate pretty complex queries is a rather simple way. >> >> So traditionally - a query language (PythonQL) is good at expressing complex things easily, but >> then you need a lot of work from the optimizer and the database to turn it into an efficient plan. A >> library like blaze or pandas is more of an ?algebra? - its really a plan specification. It will usually >> take much longer to memorize all the operators and ways of doing things in such a library and typically >> you have to go back to the documentation to do things that differ slightly from what you typically do. >> >> Oh yeah, so far our execution engine is pretty simple and not too efficient, but we plan to fix this >> in the future and be at least comparable to pandas performance (need to look at what? s under the >> hood in blaze). >> >> Of course this is my take (although I heard a few similar things from our early users). It would be >> interesting to see how other folks compare the two approaches. >> >> Btw. we have built a library for working with pandas Dataframes, we could do it for blaze too, I suppose. >> >> >> >>> On 1 Nov 2016, at 21:17, David Mertz wrote: >>> >>> How do you see this as different from Blaze (http://blaze.readthedocs.io/en/latest/index.html)? >>> A >>> >>> >>>> On Nov 1, 2016 1:34 AM, "Pavel Velikhov" wrote: >>>> Hi Folks, >>>> >>>> We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language, >>>> drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! >>>> >>>> The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have >>>> an interactive interpreter and Jupyter support planned. >>>> >>>> Best regards! >>>> PythonQL team >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python... at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 3 00:46:35 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Nov 2016 17:46:35 +1300 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161103011046.GO3365@ando.pearwood.info> References: <1478111438.1838.88.camel@gmail.com> <581A6735.9040503@canterbury.ac.nz> <20161103011046.GO3365@ando.pearwood.info> Message-ID: <581AC12B.9050102@canterbury.ac.nz> Steven D'Aprano wrote: > Or even the subject line of this email thread??? Sorry, crossed over discussions. But I think it's also true that the null-coalescing idea is for cases where it's not an error for something to be None. -- Greg From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 3 06:10:20 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 3 Nov 2016 19:10:20 +0900 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> Message-ID: <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> Danilo J. S. Bellini writes: > I'm not talking about "simple cases", but about quite general > cases, general in the sense that it allows modeling time varying > models (and the proposal is about recursion, whose computational > power should be obvious). I think I admitted that in the text you snipped. If not, consider it stipulated here. My point is that for me functions like accumulate are sufficient without new syntax, and if you are going to do time- varying stuff in the kind of economics I studied and research, you are going to need to carry around so much state that the relative amount of simplicity you can gain with syntax is very small. As for recursion, the syntax you proposed doesn't say "recursion" to me, it says "let's add more complexity to a syntax that already is hard to fit on a line so we can turn a sequence into a series." > Anyway, that makes this talking about computational accuracy sound > like an argument for my proposal here, not against it. Not as I see it. My point is that functions like accumulate already get me as much of your proposal as I can see being useful in my own applications, and so I'd rather spend effort on the inherent complexity of accurate computation than on learning new syntax which as far as I can see buys me no extra simplicity. Consider the existing comprehension syntax. I use it all the time because it's very expressive, it "looks like" what I would write in a declarative mathematical definition, and the constructive alternative is not a function call, it's a loop, in many cases with a nested if test as well. However, when you try to express more than one loop with a comprehension, you end up with a counterintuitive ordering based on the current "this expands naturally to for loops and if conditionals according to this simple rule" syntax, along with a separation of the result expression from the loop where it's actually produced. It's not obvious to me that use of comprehension syntax beyond [ f(x) for x in y if g(x) ] is all that useful, but that very basic form is extremely useful by itself. If your proposal will help make more complex comprehensions easy to understand, then you've got something I'd like to have. But I don't yet see how it fixes that, and if not, I ain't gonna need it and I don't want to have to explain it to my students when they ain't gonna need it either. > Is there any previous discussion on this topic or something related > to it? Where's the link? (I asked it before) I don't have one off-hand (I really am interested only in the claimed uses in economics). As I wrote earlier, maybe David Mertz does. > Actually, that's an argument for this proposal, not against. I'm > proposing a way to avoid itertools.accumulate and functools.reduce > using an explicit syntax instead of the functions. Or does GvR > prefer map/filter instead of list comprehensions? itertools functions are widely used and generally well-thought-of. Avoiding accumulate is a non-goal AFAIK. AIUI, Guido does think that avoiding reduce is a goal, but I believe that's because he thinks the semantics are just plain hard to understand. If your syntax is in large part a substitute for reduce, I doubt he will like it more than reduce just because it's syntax. (I do not speak for Guido; I offer my observation in the hope it will help you prepare to present your idea to him in the most favorable light.) > If you're right, then we should be consistent and propose the elimination > of list comprehension and every syntax alike. The arguments you're giving > against my proposal are the same. "A foolish consistency is the hobgoblin of small minds." If Emerson hadn't lived a century before Tim, I'm sure it would be in the Zen. That is, comprehensions are clearly useful, and I use them in almost every program I write. I still don't see when I'd ever strongly prefer the syntax you propose to the itertools we already have, so I don't care if it's a consistent extension. That doesn't mean it's not useful to you or others, it just means I'm not convinced -- and so I will not join the proponents. That's all. From steve at pearwood.info Thu Nov 3 07:38:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 Nov 2016 22:38:18 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <6a17a21a-bd59-c804-82e0-43b697d1d304@mozilla.com> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> <6a17a21a-bd59-c804-82e0-43b697d1d304@mozilla.com> Message-ID: <20161103113817.GR3365@ando.pearwood.info> On Wed, Nov 02, 2016 at 10:22:56PM -0400, Kyle Lahnakoski wrote: > > On 11/2/2016 2:30 PM, Zero Piraeus wrote: > > > >If I write something like obj.attr, the failure mode I care about is that > >obj has no attribute attr, rather than that obj is specifically None (or > >one of a defined group of somewhat Nonelike objects). > > > > I agree with this understanding. The problem with None-coalescing is > it doesn't consider where None came from. I don't see why you think that's a problem. What does it matter where None comes from? It seems to me that the important factor is what you intend to do with it, not where it came from. If I write this code: None if obj is None else obj.attr what difference does it make where None came from? The important thing is that I'm treating obj == None as a known condition. Now if you *don't* want to treat None specially, then of course this feature will be of no interest to you. This feature isn't aimed at you, it is aimed at the many people who do treat None specially. If you prefer: # obj is known to be a specific type, but attr may be None None if obj.attr is None else obj.attr.flange() That would be written obj.attr?.flange() in the proposed new syntax, and obj.attr will only be evaluated once. But note that this is quite different from the situation that Zero is talking about. In Zero's code, he prefers for obj.attr to not exist at all rather than setting it to None. So rather than this: None if obj.attr is None else obj.attr.flange() he might write something similar to this: None if not hasattr(obj, 'attr') else obj.attr.flange() (or at least *I* might write that, under the circumstances Zero refers to.) And here's the good bit: under this proposal, we might simplify it to this: getattr(obj, 'attr', None)?.flange() which handles three cases for the price of one: - obj.attr doesn't exist at all, in which case return None; - obj.attr exists and is None, in which case return None; - obj.attr exists and is not None, in which case call its flange() method. To me, having a compact and expressive syntax to write that is very attractive. > I suspect enumerating the source > of None values will reveal the majority of them are a result of > `getattr(obj, attr)` returning None (or obj.get(attr) returning None). Have you read the PEP? If you do, you'll see that the author has already collected potential examples of use from the standard library: https://github.com/python/peps/blob/master/pep-0505/find-pep505.out Looking at those examples of code, I don't think it is likely that the majority (or even a large minority) are the result of getattr. But even if they are, what difference does it make? > If your code deals with annotated objects, rather than strictly typed > objects, you will have many instances of attributes resolving to None. > Making specific classes for each of the annotations, or combinations of > annotations, is prohibitive, so you don't. Rather, you use a few > general types and set some attributes to None to indicate a property is > not-relevant-for-this-object. I'm afraid I don't understand this. > None checks are type checks. They are late type checks . Call it a type check if you like, but type check or value check or "rhubarb", whatever you want to call it, any time you test for None: if expression is None ... there's a reasonable chance that the None-aware operators may help you write better, more expressive code without falling into the trap of using `or` to handle None. Not in *all* cases, but I think often enough. -- Steve From p.f.moore at gmail.com Thu Nov 3 08:36:09 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 3 Nov 2016 12:36:09 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161103113817.GR3365@ando.pearwood.info> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <1478111438.1838.88.camel@gmail.com> <6a17a21a-bd59-c804-82e0-43b697d1d304@mozilla.com> <20161103113817.GR3365@ando.pearwood.info> Message-ID: On 3 November 2016 at 11:38, Steven D'Aprano wrote: > Looking at those examples of code, I don't think it is likely that the > majority (or even a large minority) are the result of getattr. > > But even if they are, what difference does it make? It may have mattered, if a getattr(obj, attr, coalesce=true) function would provide the same effect with no syntax change. That approach was suggested earlier in the thread, and met with a lukewarm response, but if it had turned out to actually address the majority of use cases, that would be a point in its favour. It's a relatively minor point, but a valid one. Paul From srkunze at mail.de Thu Nov 3 12:00:49 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 3 Nov 2016 17:00:49 +0100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <61e319c2-8af5-40eb-03ac-190e1d51fb1e@mrabarnett.plus.com> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <581A6165.1040809@canterbury.ac.nz> <61e319c2-8af5-40eb-03ac-190e1d51fb1e@mrabarnett.plus.com> Message-ID: <100dba07-72bf-19e7-b966-e03ba028764a@mail.de> On 03.11.2016 02:06, MRAB wrote: > On 2016-11-02 21:57, Greg Ewing wrote: >> MRAB wrote: >>> target = expr1 || expr2 || expr3 >>> target = expr1 && expr2 && expr3 >>> >>> except that only None would be considered falsey? >>> >>> Or would that be confusing? >> >> Yes, I think that borrowing an operator from C but giving >> it subtly different semantics would be *very* confusing, >> especially to people more familiar with C than Python. >> They're going to look at it and *think* they know what it >> means, except they don't. >> > OK, if we're going to have ?. and ?[, then: > > target = expr1 ?| expr2 ?| expr3 > target = expr1 ?& expr2 ?& expr3 > > A disadvantage is that they look a little like | and &, which don't > short-circuit. Many recent proposals were out-right rejected by a majority of Python users and developers because of it's cryptic, Perl-like appearance. Don't you think ?., ?[..], ??, ?|, ?& etc. don't look Perlish and cryptic? Best, Sven From chris.barker at noaa.gov Thu Nov 3 14:36:55 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 3 Nov 2016 11:36:55 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161103001954.GM3365@ando.pearwood.info> References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <20161103001954.GM3365@ando.pearwood.info> Message-ID: Thanks Steven, this is great! so -- when all this started, I think one of the use cases was to clean up this really common idiom: self.an_arg = the_default if an_arg is None else an_arg so would that be: self.an_arg = the_default ?? an_arg That would be nice. Though the fact that I'm still not sure if that's correct makes me think this is not so intuitive! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Nov 3 15:00:03 2016 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 3 Nov 2016 19:00:03 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <20161103001954.GM3365@ando.pearwood.info> Message-ID: On 2016-11-03 18:36, Chris Barker wrote: > Thanks Steven, this is great! > > so -- when all this started, I think one of the use cases was to clean > up this really common idiom: > > self.an_arg = the_default if an_arg is None else an_arg > > so would that be: > > self.an_arg = the_default ?? an_arg > > That would be nice. > > Though the fact that I'm still not sure if that's correct makes me think > this is not so intuitive! > No, ?? is a bit like 'or', except that only None is falsey, so it would be: self.an_arg = an_arg ?? the_default From chris.barker at noaa.gov Thu Nov 3 15:35:07 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 3 Nov 2016 12:35:07 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> <20161103001954.GM3365@ando.pearwood.info> Message-ID: On Thu, Nov 3, 2016 at 12:00 PM, MRAB wrote: > self.an_arg = the_default if an_arg is None else an_arg > > > No, ?? is a bit like 'or', except that only None is falsey, so it would be: >> > > self.an_arg = an_arg ?? the_default thanks! and actually, that reads much better to me. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Nov 3 13:31:35 2016 From: brett at python.org (Brett Cannon) Date: Thu, 03 Nov 2016 17:31:35 +0000 Subject: [Python-ideas] Going off-topic (was: Null coalescing operator) In-Reply-To: References: <22548.828.996348.657305@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, 2 Nov 2016 at 17:34 Mikhail V wrote: > On 2 November 2016 at 21:50, David Mertz wrote: > > Even though I really don't want new null-coalescing operators, I really > > appreciate the ternary operator in Python (or in C). > > > > On Wed, Nov 2, 2016 at 12:38 PM, Mikhail V wrote: > >> > >> result = a > b ? x : y > >> > >> is IMHO a syntactical herecy. Such things disgust me from programming. > >> Why on earth one cannot just wrap it in function > >> c = nicefunc(a,b) > > > > > > The problem here is that the general form isn't ONLY to return 'x' or 'y' > > but the decide between arbitrary values. Hard-coding the variables into > the > > function loses 90%+ of the point. > > > > So the general function would need a signature like: > > > > c = nicefunc(a, b, x, y) > > > > The problem here is that this call might be: > > > > c = nicefunc(a, b, run_for_hours(), has_side_effects()) > > > > We only want ONE of 'x' and 'y' to eagerly evaluate. In the C or Python > > ternary we get exactly that. Obviously, that also happens in your fully > > spelled out if/else block too, but that's multiline and needs to setup > > variables not just be used as an expression. > > [Apologies for off-topic one more time] > A quick mailing list etiquette lesson: if you know you're going off-topic then please start a new thread with a new subject line (see the subject line for this new thread as an example). Since "off-topic" means "new thread of discussion", then please just start a new thread. You can't assume that people are reading email through an email client which does threading, meaning those of us not interested in the off-topic offshoot have to read your email, realize it's off-topic, and then get passed that email to get to the next email that is on-topic (and this is still a time sink even when your first line of your reply is "this is off-topic"). So please be mindful of when you stray off-topic, and when you do then start a new thread to make managing different discussions easier for everyone. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Nov 3 19:06:39 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 4 Nov 2016 10:06:39 +1100 Subject: [Python-ideas] Null coalescing operator In-Reply-To: References: <20161103001954.GM3365@ando.pearwood.info> Message-ID: <20161103230638.GT3365@ando.pearwood.info> On Thu, Nov 03, 2016 at 12:35:07PM -0700, Chris Barker wrote: > On Thu, Nov 3, 2016 at 12:00 PM, MRAB wrote: > > > self.an_arg = the_default if an_arg is None else an_arg > > > No, ?? is a bit like 'or', except that only None is falsey, so it would be: > > > > self.an_arg = an_arg ?? the_default > > > thanks! and actually, that reads much better to me. That suggests a possible different colour for this operator: `?or`. (Apologies if that's already been suggested and rejected earlier.) The None-aware "safe navigation" operators ?. and ?[] will be used where Python already uses punctuation: spam.eggs # ordinary attribute lookup spam?.eggs # None-aware attribute lookup spam[eggs] # ordinary item lookup spam?[eggs] # None-aware item lookup which is simple enough to remember: just prefix your usual operator with a question mark to make it None-aware. But one of the disadvantages of ?? as an operator is that it replaces a keyword with completely unrelated punctuation: spam or default # default only if spam is any Falsey value spam ?? default # default only if spam is None Compared to: spam ?or default gives us the same rule: prefix the `or` operator with ? to make it None- aware. -- Steve From chris.barker at noaa.gov Thu Nov 3 20:18:04 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 3 Nov 2016 17:18:04 -0700 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161103230638.GT3365@ando.pearwood.info> References: <20161103001954.GM3365@ando.pearwood.info> <20161103230638.GT3365@ando.pearwood.info> Message-ID: On Thu, Nov 3, 2016 at 4:06 PM, Steven D'Aprano wrote: > > > No, ?? is a bit like 'or', except that only None is falsey, so it > would be: > > > > > > self.an_arg = an_arg ?? the_default > > > > > > thanks! and actually, that reads much better to me. > > That suggests a possible different colour for this operator: `?or`. > I like it! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 5 05:50:44 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 5 Nov 2016 19:50:44 +1000 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol Message-ID: Hi folks, As promised, here's a follow-up to the withdrawn PEP 531 that focuses on coming up with a common protocol driven solution to conditional evaluation of subexpressions that also addresses the element-wise comparison chaining problem that Guido noted when rejecting PEP 335. I quite like how it came out, but see the "Risks & Concerns" section for a discussion of an internal inconsistency it would introduce into the language if accepted as currently written, and the potentially far-reaching consequences actually resolving that inconsistency might have on the way people write their Python code (if the PEP was subsequently approved). Regards, Nick. ================================= PEP: 532 Title: A circuit breaking operator and protocol Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Oct-2016 Python-Version: 3.7 Post-History: 5-Nov-2016 Abstract ======== Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP proposes the addition of a new protocol-driven circuit breaking operator to Python that allows the left operand to decide whether or not the expression should short circuit and return a result immediately, or else continue on with evaluation of the right operand:: exists(foo) else bar missing(foo) else foo.bar() These two expressions can be read as: * "the expression result is 'foo' if it exists, otherwise it is 'bar'" * "the expression result is 'foo' if it is missing, otherwise it is 'foo.bar()'" Execution of these expressions relies on a new circuit breaking protocol that implicitly avoids repeated evaluation of the left operand while letting that operand fully control the result of the expression, regardless of whether it skips evaluation of the right operand or not:: _lhs = LHS type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) To properly support logical negation of circuit breakers, a new ``__not__`` protocol method would also be introduced allowing objects to control the result of ``not obj`` expressions. As shown in the basic example above, the PEP further proposes the addition of builtin ``exists`` and ``missing`` circuit breakers that provide conditional branching based on whether or not an object is ``None``, but return the original object rather than the existence checking wrapper when the expression evaluation short circuits. In addition to being usable as simple boolean operators (e.g. as in ``assert all(exists, items)`` or ``if any(missing, items):``), these circuit breakers will allow existence checking fallback operations (aka None-coalescing operations) to be written as:: value = exists(expr1) else exists(expr2) else expr3 and existence checking precondition operations (aka None-propagating or None-severing operations) to be written as:: value = missing(obj) else obj.field.of.interest value = missing(obj) else obj["field"]["of"]["interest"] A change to the definition of chained comparisons is also proposed, where the comparison chaining will be updated to use the circuit breaking operator rather than the logical disjunction (``and``) operator if the left hand comparison returns a circuit breaker as its result. While there are some practical complexities arising from the current handling of single-valued arrays in NumPy, this change should be sufficient to allow elementwise chained comparison operations for matrices, where the result is a matrix of boolean values, rather than tautologically returning ``True`` or raising ``ValueError``. Relationship with other PEPs ============================ This PEP is a direct successor to PEP 531, replacing the existence checking protocol and the new ``?then`` and ``?else`` syntactic operators defined there with a single protocol driven ``else`` operator and adjustments to the ``not`` operator. The existence checking use cases are taken from that PEP. It is also a direct successor to PEP 335, which proposed the ability to overload the ``and`` and ``or`` operators directly, with the ability to overload the semantics of comparison chaining being one of the consequences of that change. The proposal in this PEP to instead handle the element-wise comparison use case by changing the semantic definition of comparison chaining is drawn from Guido's rejection of PEP 335. This PEP competes with the dedicated null-coalescing operator in PEP 505, proposing that improved support for null-coalescing operations be offered through a more general protocol-driven short circuiting operator and related builtins, rather than through a dedicated single-purpose operator. It doesn't compete with PEP 505's proposed shorthands for existence checking attribute access and subscripting, but instead offers an alternative underlying semantic framework for defining them: * ``EXPR?.attr`` would be syntactic sugar for ``missing(EXPR) else EXPR.attr`` * ``EXPR?[key]`` would be syntactic sugar for ``missing(EXPR) else EXPR[key]`` In both cases, the dedicated syntactic form could be optimised to avoid actually creating the circuit breaker instance. Specification ============= The circuit breaking operator (``else``) ---------------------------------------- Circuit breaking expressions would be written using ``else`` as a new binary operator, akin to the existing ``and`` and ``or`` logical operators:: LHS else RHS Ignoring the hidden variable assignment, this is semantically equivalent to:: _lhs = LHS type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) The key difference relative to the existing ``or`` operator is that the value determining which branch of the conditional expression gets executed *also* gets a chance to postprocess the results of the expressions on each of the branches. As part of the short-circuiting behaviour, interpreter implementations are expected to access only the protocol method needed for the branch that is actually executed, but it is still recommended that circuit breaker authors that always return ``True`` or always return ``False`` from ``__bool__`` explicitly raise ``NotImplementedError`` with a suitable message from branch methods that are never expected to be executed (see the comparison chaining use case in the Rationale section below for an example of that). It is proposed that the ``else`` operator use a new precedence level that binds less tightly than the ``or`` operator by adjusting the relevant line in Python's grammar from the current:: test: or_test ['if' or_test 'else' test] | lambdef to instead be:: test: else_test ['if' or_test 'else' test] | lambdef else_test: or_test ['else' test] The definition of ``test_nocond`` would remain unchanged, so circuit breaking expressions would require parentheses when used in the ``if`` clause of comprehensions and generator expressions just as conditional expressions themselves do. This grammar definition means precedence/associativity in the otherwise ambiguous case of ``expr1 if cond else expr2 else epxr3`` resolves as ``(expr1 if cond else expr2) else epxr3``. A guideline will also be added to PEP 8 to say "don't do that", as such a construct will be inherently confusing for readers, regardless of how the interpreter executes it. Overloading logical inversion (``not``) --------------------------------------- Any circuit breaker definition will have a logical inverse that is still a circuit breaker, but inverts the answer as to whether or not to short circuit the expression evaluation. For example, the ``exists`` and ``missing`` circuit breakers proposed in this PEP are each other's logical inverse. A new protocol method, ``__not__(self)``, will be introduced to permit circuit breakers and other types to override ``not`` expressions to return their logical inverse rather than a coerced boolean result. To preserve the semantics of existing language optimisations, ``__not__`` implementations will be obliged to respect the following invariant:: assert not bool(obj) == bool(not obj) Chained comparisons ------------------- A chained comparison like ``0 < x < 10`` written as:: LEFT_BOUND left_op EXPR right_op RIGHT_BOUND is currently roughly semantically equivalent to:: _expr = EXPR _lhs_result = LEFT_BOUND left_op _expr _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) This PEP proposes that this be changed to explicitly check if the left comparison returns a circuit breaker, and if so, use ``else`` rather than ``and`` to implement the comparison chaining:: _expr = EXPR _lhs_result = LEFT_BOUND left_op _expr if hasattr(type(_lhs_result), "__then__"): _expr_result = _lhs_result else (_expr right_op RIGHT_BOUND) else: _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) This allows types like NumPy arrays to control the behaviour of chained comparisons by returning circuit breakers from comparison operations. Existence checking comparisons ------------------------------ Two new builtins implementing the new protocol are proposed to encapsulate the notion of "existence checking": seeing if a value is ``None`` and either falling back to an alternative value (an operation known as "None-coalescing") or passing it through as the result of the overall expression (an operation known as "None-severing" or "None-propagating"). These builtins would be defined as follows:: class CircuitBreaker: """Base circuit breaker type (available as types.CircuitBreaker)""" def __init__(self, value, condition, inverse_type): self.value = value self._condition = condition self._inverse_type = inverse_type def __bool__(self): return self._condition def __not__(self): return self._inverse_type(self.value) def __then__(self): return self.value def __else__(self, other): if other is self: return self.value return other class exists(types.CircuitBreaker): """Circuit breaker for 'EXPR is not None' checks""" def __init__(self, value): super().__init__(value, value is not None, missing) class missing(types.CircuitBreaker): """Circuit breaker for 'EXPR is None' checks""" def __init__(self, value): super().__init__(value, value is None, exists) Aside from changing the definition of ``__bool__`` to be based on ``is not None`` rather than normal truth checking, the key characteristic of ``exists`` is that when it is used as a circuit breaker, it is *ephemeral*: when it is told that short circuiting has taken place, it returns the original value, rather than the existence checking wrapper. ``missing`` is defined as the logically inverted counterpart of ``exists``: ``not exists(obj)`` is semantically equivalent to ``missing(obj)``. The ``__else__`` implementations for both builtin circuit breakers are defined such that the wrapper will always be removed even if you explicitly pass the circuit breaker to both sides of the ``else`` expression:: breaker = exists(foo) assert (breaker else breaker) is foo breaker = missing(foo) assert (breaker else breaker) is foo Other conditional constructs ---------------------------- No changes are proposed to if statements, while statements, conditional expressions, comprehensions, or generator expressions, as the boolean clauses they contain are already used for control flow purposes. However, it's worth noting that while such proposals are outside the scope of this PEP, the circuit breaking protocol defined here would be sufficient to support constructs like:: while exists(dynamic_query()) as result: ... # Code using result and: if exists(re.search(pattern, text)) as match: ... # Code using match Leaving the door open to such a future extension is the main reason for recommending that circuit breaker implementations handle the ``self is other`` case in ``__else__`` implementations the same way as they handle the short-circuiting behaviour in ``__then__``. Style guide recommendations --------------------------- The following additions to PEP 8 are proposed in relation to the new features introduced by this PEP: * In the absence of other considerations, prefer the use of the builtin circuit breakers ``exists`` and ``missing`` over the corresponding conditional expressions * Do not combine conditional expressions (``if-else``) and circuit breaking expressions (the ``else`` operator) in a single expression - use one or the other depending on the situation, but not both. Rationale ========= Adding a new operator --------------------- Similar to PEP 335, early drafts of this PEP focused on making the existing ``and`` and ``or`` operators less rigid in their interpretation, rather than proposing new operators. However, this proved to be problematic for a few reasons: * defining a shared protocol for both ``and`` and ``or`` was confusing, as ``__then__`` was the short-circuiting outcome for ``or``, while ``__else__`` was the short-circuiting outcome for ``and`` * the ``and`` and ``or`` operators have a long established and stable meaning, so readers would inevitably be surprised if their meaning now became dependent on the type of the left operand. Even new users would be confused by this change due to 25+ years of teaching material that assumes the current well-known semantics for these operators * Python interpreter implementations, including CPython, have taken advantage of the existing semantics of ``and`` and ``or`` when defining runtime and compile time optimisations, which would all need to be reviewed and potentially discarded if the semantics of those operations changed Proposing a single new operator instead resolves all of those issues - ``__then__`` always indicates short circuiting, ``__else__`` only indicates "short circuiting" if the circuit breaker itself is also passed in as the right operand, and the semantics of ``and`` and ``or`` remain entirely unchanged. While the semantics of the unary ``not`` operator do change, the invariant required of ``__not__`` implementations means that existing expression optimisations in boolean contexts will remain valid. As a result of that design simplification, the new protocol and operator would even allow us to expose ``operator.true`` and ``operator.false`` as circuit breaker definitions if we chose to do so:: class true(types.CircuitBreaker): """Circuit breaker for 'bool(EXPR)' checks""" def __init__(self, value): super().__init__(value, bool(value), when_false) class false(types.CircuitBreaker): """Circuit breaker for 'not bool(EXPR)' checks""" def __init__(self, value): super().__init__(value, not bool(value), when_true) Given those circuit breakers: * ``LHS or RHS`` would be roughly ``operator.true(LHS) else RHS`` * ``LHS and RHS`` would be roughly ``operator.false(LHS) else RHS`` Naming the operator and protocol -------------------------------- The names "circuit breaking operator", "circuit breaking protocol" and "circuit breaker" are all inspired by the phrase "short circuiting operator": the general language design term for operators that only conditionally evaluate their right operand. The electrical analogy is that circuit breakers in Python detect and handle short circuits in expressions before they trigger any exceptions similar to the way that circuit breakers detect and handle short circuits in electrical systems before they damage any equipment or harm any humans. The Python level analogy is that just as a ``break`` statement lets you terminate a loop before it reaches its natural conclusion, a circuit breaking expression lets you terminate evaluation of the expression and produce a result immediately. Using an existing keyword ------------------------- Using an existing keyword has the benefit of allowing the new expression to be introduced without a ``__future__`` statement. ``else`` is semantically appropriate for the proposed new protocol, and the only syntactic ambiguity introduced arises when the new operator is combined with the explicit ``if-else`` conditional expression syntax. Element-wise chained comparisons -------------------------------- In ultimately rejecting PEP 335, Guido van Rossum noted [1_]: The NumPy folks brought up a somewhat separate issue: for them, the most common use case is chained comparisons (e.g. A < B < C). To understand this observation, we first need to look at how comparisons work with NumPy arrays:: >>> import numpy as np >>> increasing = np.arange(5) >>> increasing array([0, 1, 2, 3, 4]) >>> decreasing = np.arange(4, -1, -1) >>> decreasing array([4, 3, 2, 1, 0]) >>> increasing < decreasing array([ True, True, False, False, False], dtype=bool) Here we see that NumPy array comparisons are element-wise by default, comparing each element in the lefthand array to the corresponding element in the righthand array, and producing a matrix of boolean results. If either side of the comparison is a scalar value, then it is broadcast across the array and compared to each individual element:: >>> 0 < increasing array([False, True, True, True, True], dtype=bool) >>> increasing < 4 array([ True, True, True, True, False], dtype=bool) However, this broadcasting idiom breaks down if we attempt to use chained comparisons:: >>> 0 < increasing < 4 Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() The problem is that internally, Python implicitly expands this chained comparison into the form:: >>> 0 < increasing and increasing < 4 Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() And NumPy only permits implicit coercion to a boolean value for single-element arrays where ``a.any()`` and ``a.all()`` can be assured of having the same result:: >>> np.array([False]) and np.array([False]) array([False], dtype=bool) >>> np.array([False]) and np.array([True]) array([False], dtype=bool) >>> np.array([True]) and np.array([False]) array([False], dtype=bool) >>> np.array([True]) and np.array([True]) array([ True], dtype=bool) The proposal in this PEP would allow this situation to be changed by updating the definition of element-wise comparison operations in NumPy to return a dedicated subclass that implements the new circuit breaking protocol and also changes the result array's interpretation in a boolean context to always return ``False`` and hence never trigger the short-circuiting behaviour:: class ComparisonResultArray(np.ndarray): def __bool__(self): return False def _raise_NotImplementedError(self): msg = ("Comparison array truth values are ambiguous outside " "chained comparisons. Use a.any() or a.all()") raise NotImplementedError(msg) def __not__(self): self._raise_NotImplementedError() def __then__(self): self._raise_NotImplementedError() def __else__(self, other): return np.logical_and(self, other.view(ComparisonResultArray)) With this change, the chained comparison example above would be able to return:: >>> 0 < increasing < 4 ComparisonResultArray([ False, True, True, True, False], dtype=bool) Existence checking expressions ------------------------------ An increasingly common requirement in modern software development is the need to work with "semi-structured data": data where the structure of the data is known in advance, but pieces of it may be missing at runtime, and the software manipulating that data is expected to degrade gracefully (e.g. by omitting results that depend on the missing data) rather than failing outright. Some particularly common cases where this issue arises are: * handling optional application configuration settings and function parameters * handling external service failures in distributed systems * handling data sets that include some partial records At the moment, writing such software in Python can be genuinely awkward, as your code ends up littered with expressions like: * ``value1 = expr1.field.of.interest if expr1 is not None else None`` * ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else None`` * ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not None else expr5`` PEP 531 goes into more detail on some of the challenges of working with this kind of data, particularly in data transformation pipelines where dealing with potentially missing content is the norm rather than the exception. The combined impact of the proposals in this PEP is to allow the above sample expressions to instead be written as: * ``value1 = missing(expr1) else expr1.field.of.interest`` * ``value2 = missing(expr2) else expr2.["field"]["of"]["interest"]`` * ``value3 = exists(expr3) else exists(expr4) else expr5`` In these forms, significantly more of the text presented to the reader is immediately relevant to the question "What does this code do?", while the boilerplate code to handle missing data by passing it through to the output or falling back to an alternative input, has shrunk to two uses of the new ``missing`` builtin, and two uses of the new ``exists`` builtin. In the first two examples, the 31 character boilerplate suffix ``if exprN is not None else None`` (minimally 27 characters for a single letter variable name) has been replaced by a 19 character ``missing(expr1) else`` prefix (minimally 15 characters with a single letter variable name), markedly improving the signal-to-pattern-noise ratio of the lines (especially if it encourages the use of more meaningful variable and field names rather than making them shorter purely for the sake of expression brevity). The additional syntactic sugar proposals in PEP 505 would further reduce this boilerplate to a single ``?`` character that also eliminated the repetition of the expession being checked for existence. In the last example, not only are two instances of the 21 character boilerplate, `` if exprN is not None`` (minimally 17 characters) replaced with the 8 character function call ``exists()``, but that function call is placed directly around the original expression, eliminating the need to duplicate it in the conditional existence check. Risks and concerns ================== This PEP has been designed specifically to address the risks and concerns raised when discussing PEPs 335, 505 and 531. * it defines a new operator and adjusts the definition of chained comparison rather than impacting the existing ``and`` and ``or`` operators * the changes to the ``not`` unary operator are defined in such a way that control flow optimisations based on the existing semantics remain valid * rather than the cryptic ``??``, it uses ``else`` as the operator keyword in exactly the same sense as it is already used in conditional expressions * it defines a general purpose short-circuiting binary operator that can even be used to express the existing semantics of ``and`` and ``or`` rather than focusing solely and inflexibly on existence checking * it names the proposed builtins in such a way that they provide a strong mnemonic hint as to when the expression containing them will short-circuit and skip evaluating the right operand Possible confusion with conditional expressions ----------------------------------------------- The proposal in this PEP is essentially for an "implied ``if``" where if you omit the ``if`` clause from a conditional expression, you invoke the circuit breaking protocol instead. That is:: exists(foo) else calculate_default() invokes the new protocol, but:: foo.field.of.interest if exists(foo) else calculate_default() bypasses it entirely, *including* the non-short-circuiting ``__else__`` method. This mostly wouldn't be a problem for the proposed ``types.CircuitBreaker`` implementation (and hence the ``exists`` and ``missing`` builtins), as the only purpose the extended protocol serves in that case is to remove the wrapper in the short-circuiting case - the ``__else__`` method passes the right operand through unchanged. However, this discrepancy could potentially be eliminated entirely by also updating conditional expressions to use the circuit breaking protocol if the condition defines those methods. In that case, ``__then__`` would need to be updated to accept the left operand as a parameter, with short-circuiting indicated by passing in the circuit breaker itself:: class CircuitBreaker: """Base circuit breaker type (available as types.CircuitBreaker)""" def __init__(self, value, condition, inverse_type): self.value = value self._condition = condition self._inverse_type = inverse_type def __bool__(self): return self._condition def __not__(self): return self._inverse_type(self.value) def __then__(self, other): if other is not self: return other return self.value # Short-circuit, remove the wrapper def __else__(self, other): if other is not self: return other return self.value # Short-circuit, remove the wrapper With this symmetric protocol, the definition of conditional expressions could be updated to also make the ``else`` clause optional:: test: else_test ['if' or_test ['else' test]] | lambdef else_test: or_test ['else' test] (We would avoid the apparent simplification to ``else_test ('if' else_test)*`` in order to make it easier to correctly preserve the semantics of normal conditional expressions) Given that expanded definition, the following statements would be functionally equivalent:: foo = calculate_default() if missing(foo) foo = calculate_default() if foo is None else foo Just as the base proposal already makes the following equivalent:: foo = exists(foo) else calculate_default() foo = foo if foo is not None else calculate_default() The ``if`` based circuit breaker form has the virtue of reading significantly better when used for conditional imperative commands like debug messages:: print(some_expensive_query()) if verbosity > 2 If we went down this path, then ``operator.true`` would need to be declared as the nominal implicit circuit breaker when the condition didn't define the circuit breaker protocol itself (so the above example would produce ``None`` if the debugging message was printed, and ``False`` otherwise) The main objection to this expansion of the proposal is that it makes it a more intrusive change that may potentially affect the behaviour of existing code, while the main point in its favour is that allowing both ``if`` and ``else`` as circuit breaking operators and also supporting the circuit breaking protocol for normal conditional expressions would be significantly more self-consistent than special-casing a bare ``else`` as a stand-alone operator. Design Discussion ================= Arbitrary sentinel objects -------------------------- Unlike PEPs 505 and 531, this proposal readily handles custom sentinel objects:: class defined(types.CircuitBreaker): MISSING = object() def __init__(self, value): super().__init__(self, value is not self.MISSING, undefined) class undefined(types.CircuitBreaker): def __init__(self, value): super().__init__(self, value is defined.MISSING, defined) # Using the sentinel to check whether or not an argument was supplied def my_func(arg=defined.MISSING): arg = defined(arg) else calculate_default() Implementation ============== As with PEP 505, actual implementation has been deferred pending in-principle interest in the idea of making these changes - aside from the possible syntactic ambiguity concerns covered by the grammer proposals above, the implementation isn't really the hard part of these proposals, the hard part is deciding whether or not this is a change where the long term benefits for new and existing Python users outweigh the short term costs involved in the wider ecosystem (including developers of other implementations, language curriculum developers, and authors of other Python related educational material) adjusting to the change. ...TBD... Acknowledgements ================ Thanks go to Mark E. Haase for feedback on and contributions to earlier drafts of this proposal. However, his clear and exhaustive explanation of the original protocol design that modified the semantics of ``if-else`` conditional expressions to use an underlying ``__then__``/``__else__`` protocol helped convince me it was too complicated to keep, so this iteration contains neither that version of the protocol, nor Mark's explanation of it. References ========== .. [1] PEP 335 rejection notification (http://mail.python.org/pipermail/python-dev/2012-March/117510.html) Copyright ========= This document has been placed in the public domain under the terms of the CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From srkunze at mail.de Sat Nov 5 07:59:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 5 Nov 2016 12:59:24 +0100 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: <2eb87a13-a94e-9608-102e-df137fb250d5@mail.de> Thanks, Nick. This PEP is far easier to read than the previous one. On 05.11.2016 10:50, Nick Coghlan wrote: > This PEP has been designed specifically to address the risks and concerns > raised when discussing PEPs 335, 505 and 531. > > * it defines a new operator and adjusts the definition of chained comparison > rather than impacting the existing ``and`` and ``or`` operators > * the changes to the ``not`` unary operator are defined in such a way that > control flow optimisations based on the existing semantics remain valid > * rather than the cryptic ``??``, it uses ``else`` as the operator keyword in > exactly the same sense as it is already used in conditional expressions The "else" keyword I proposed more than a year ago finally lands in a PEP. So, I am +1 on this. ;-) > * it defines a general purpose short-circuiting binary operator that can even > be used to express the existing semantics of ``and`` and ``or`` rather than > focusing solely and inflexibly on existence checking This is why I like this PEP most. > * it names the proposed builtins in such a way that they provide a strong > mnemonic hint as to when the expression containing them will short-circuit > and skip evaluating the right operand If you are referring to "missing" and to "exists", I tend to dislike them as production code use those very, very often. I think that the mnemonic hint comes from the "else" keyword, not from the proposed builtins. Furthermore, I still don't find "missing(expr1) else expr1.field" very intuitive to read. Regarding the ?. and ?[] syntactic sugar definitions, they seem very well explained. Unfortunately, having a look at our production code (configs etc.) specifically, it wouldn't help so much as we usually check for attribute existence not obj existence. So, ?. and ?[] define in the proposed form might help 50% of the use-cases but would disappoint those other 50%. Cheers, Sven From gjcarneiro at gmail.com Sat Nov 5 14:29:20 2016 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Sat, 5 Nov 2016 18:29:20 +0000 Subject: [Python-ideas] Null coalescing operator In-Reply-To: <20161103230638.GT3365@ando.pearwood.info> References: <20161103001954.GM3365@ando.pearwood.info> <20161103230638.GT3365@ando.pearwood.info> Message-ID: On 3 November 2016 at 23:06, Steven D'Aprano wrote: > On Thu, Nov 03, 2016 at 12:35:07PM -0700, Chris Barker wrote: > > On Thu, Nov 3, 2016 at 12:00 PM, MRAB > wrote: > > > > > self.an_arg = the_default if an_arg is None else an_arg > > > > > No, ?? is a bit like 'or', except that only None is falsey, so it > would be: > > > > > > self.an_arg = an_arg ?? the_default > > > > > > thanks! and actually, that reads much better to me. > > That suggests a possible different colour for this operator: `?or`. > > (Apologies if that's already been suggested and rejected earlier.) > > The None-aware "safe navigation" operators ?. and ?[] will be used where > Python already uses punctuation: > > spam.eggs # ordinary attribute lookup > spam?.eggs # None-aware attribute lookup > > spam[eggs] # ordinary item lookup > spam?[eggs] # None-aware item lookup > > > which is simple enough to remember: just prefix your usual operator with > a question mark to make it None-aware. But one of the disadvantages of > ?? as an operator is that it replaces a keyword with completely > unrelated punctuation: > > spam or default # default only if spam is any Falsey value > spam ?? default # default only if spam is None > > Compared to: > > spam ?or default > > gives us the same rule: prefix the `or` operator with ? to make it None- > aware. > Agree 100%. Prefixing an operator with ? meaning that it becomes None-aware is easy to remember. Just think of `?or` the same as `or`, but while `or` is truth-aware, `?or` would be None-aware. Similarly to other operators like . and []. How many times have I wanted to do something like `foo or bar`, but then again this is not always correct in case foo is an object that can be true or false, in which case it has to be expanded to `foo if foo is not None else bar`. With the proposal, we could write `foo ?or bar`, which is intuitively similar to the well known `foo or bar` expression and therefore easy to pick up. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sat Nov 5 20:38:34 2016 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 6 Nov 2016 01:38:34 +0100 Subject: [Python-ideas] Small improvements to the profile/cProfile API In-Reply-To: References: Message-ID: Long ago I posted a patch for this (decorator + context manager) but I bumped into a weird error I wasn't able to fix (see last comment): http://bugs.python.org/issue9285 On Wed, Nov 2, 2016 at 10:45 PM, Tim Mitchell wrote: > Hi Ben, > > Mostly I just print to stdout, I imagine more flexibility would be needed > in general. > > This is for python 2.7 - don't know if it works for 3. > > > def profile(sort='time', restriction=(), callers=None, callees=None, filename=None): > def _profileDecorator(func): > "print profile stats for decorated function" > def wrapper(*args, **kwargs): > print 'Profile for:', func.__name__ > > prof = cProfile.Profile() > result = prof.runcall(func, *args, **kwargs) > _, statsFileName = tempfile.mkstemp() > prof.dump_stats(statsFileName) > if filename is None: > stats = pstats.Stats(statsFileName) > else: > stats = pstats.Stats(statsFileName, stream=open(filename, 'w')) > if isinstance(sort, basestring): > stats.sort_stats(sort) > else: > stats.sort_stats(*sort) > if isinstance(restriction, (tuple, list)): > stats.print_stats(*restriction) > else: > stats.print_stats(restriction) > if callers is not None: > if isinstance(callers, basestring): > stats.print_callers(callers) > else: > stats.print_callers(*callers) > if callees is not None: > if isinstance(callees, basestring): > stats.print_callees(callees) > else: > stats.print_callees(*callees) > return result > return wrapper > return _profileDecorator > > Cheers > > Tim > > > On 3 November 2016 at 09:58, Ben Hoyt wrote: > >> Okay, got it, that sounds fair enough. With your @profile decorator how >> do you tell it when and where to print the output? Can you post the source >> for your decorator? >> >> On Wed, Nov 2, 2016 at 4:52 PM, Tim Mitchell > > wrote: >> >>> I use an @profile() decorator for almost all my profiling. If you want >>> to profile function foo you just decorate it and re-run the program. >>> With a with block you have to find the places where foo is called and >>> put with statements around the calls. >>> I think both approaches are equally valid and useful. >>> >>>> conduct/ >>>> >>> >>> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Sun Nov 6 00:46:40 2016 From: ram at rachum.com (Ram Rachum) Date: Sun, 6 Nov 2016 06:46:40 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ Message-ID: Hi everyone, Here is a simplification of a problem that's been happening in my code: import contextlib @contextlib.contextmanager def f(): print('1') try: yield finally: print('2') g = f() g.__enter__() This code prints 1 and then 2, not just 1 like you might expect. This is because when the generator is garbage-collected, it gets `GeneratorExit` sent to it. This has been a problem in my code since in some instances, I tell a context manager not to do its `__exit__` function. (I do this by using `ExitStack.pop_all()`. However the `__exit__` is still called here. I worked around this problem by adding `except GeneratorExit: raise` in my context manager, but that's an ugly solution. Do you think that something could be done so I won't have to add `except GeneratorExit: raise` to each context manager to get the desired behavior? Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 6 01:51:05 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Nov 2016 15:51:05 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On 6 November 2016 at 14:46, Ram Rachum wrote: > I worked around this problem by adding `except GeneratorExit: raise` in my > context manager, but that's an ugly solution. Adding `except GeneratorExit: raise` to a try statement with a finally clause won't prevent the finally clause from executing. Selective non-idempotent cleanup behaviour really isn't a good idea, so the language is fighting you for a reason here - the meaning of a "finally" clause is that the code it contains *will* get executed, and you have to try incredibly hard to keep that from happening since it's an implicit part of cleaning up unfinished generators, and we don't let you switch the nominal class of generator objects at runtime. Indeed, yield inside try/finally was prohibited for years prior to PEP 342, as the runtime previously couldn't guarantee that the finally clause would actually execute. If you *don't* want the code to execute unconditionally, then you need to be more explicit about when you *do* want it to execute with some combination of "except" and "else" clauses. For example: >>> @contextmanager ... def cm(): ... print("enter") ... try: ... yield ... except Exception: ... print("Normal exception, not GeneratorExit, KeyboardInterrupt or SystemExit") ... else: ... print("No exception") ... >>> cm().__enter__() enter Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ram at rachum.com Sun Nov 6 01:02:34 2016 From: ram at rachum.com (Ram Rachum) Date: Sun, 6 Nov 2016 08:02:34 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: Sorry, I was wrong at quoting the workaround I do, it's actually this: try: yield except GeneratorExit: raise except: cleanup() raise else: cleanup() This works, but it's ugly! And I basically need to do this to every generator-based context manager that I want to be able to run just the `__enter__` of without the `__exit__`. I wish there was a better solution than including this in my code. On Sun, Nov 6, 2016 at 7:51 AM, Nick Coghlan wrote: > On 6 November 2016 at 14:46, Ram Rachum wrote: > > I worked around this problem by adding `except GeneratorExit: raise` in > my > > context manager, but that's an ugly solution. > > Adding `except GeneratorExit: raise` to a try statement with a finally > clause won't prevent the finally clause from executing. > > Selective non-idempotent cleanup behaviour really isn't a good idea, > so the language is fighting you for a reason here - the meaning of a > "finally" clause is that the code it contains *will* get executed, and > you have to try incredibly hard to keep that from happening since it's > an implicit part of cleaning up unfinished generators, and we don't > let you switch the nominal class of generator objects at runtime. > Indeed, yield inside try/finally was prohibited for years prior to PEP > 342, as the runtime previously couldn't guarantee that the finally > clause would actually execute. > > If you *don't* want the code to execute unconditionally, then you need > to be more explicit about when you *do* want it to execute with some > combination of "except" and "else" clauses. For example: > > >>> @contextmanager > ... def cm(): > ... print("enter") > ... try: > ... yield > ... except Exception: > ... print("Normal exception, not GeneratorExit, > KeyboardInterrupt or SystemExit") > ... else: > ... print("No exception") > ... > >>> cm().__enter__() > enter > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Sun Nov 6 01:07:27 2016 From: ram at rachum.com (Ram Rachum) Date: Sun, 6 Nov 2016 08:07:27 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: Heh, I just played with this, and found a workaround. If I do something like this after creating the generator: sys.g = g Then it wouldn't get closed when Python finishes, and the cleanup won't happen, which is what I want. This is still a bit ugly but now it's something I can do automatically for every generator-based context manager that I create, so it's not as bad as the previous workaround in my opinion. (I wouldn't create a new attribute for each context manager, but just have a list with a mangled name on `sys` that'll hold all of them.) On Sun, Nov 6, 2016 at 8:02 AM, Ram Rachum wrote: > Sorry, I was wrong at quoting the workaround I do, it's actually this: > > try: > yield > except GeneratorExit: > raise > except: > cleanup() > raise > else: > cleanup() > > This works, but it's ugly! And I basically need to do this to every > generator-based context manager that I want to be able to run just the > `__enter__` of without the `__exit__`. I wish there was a better solution > than including this in my code. > > On Sun, Nov 6, 2016 at 7:51 AM, Nick Coghlan wrote: > >> On 6 November 2016 at 14:46, Ram Rachum wrote: >> > I worked around this problem by adding `except GeneratorExit: raise` in >> my >> > context manager, but that's an ugly solution. >> >> Adding `except GeneratorExit: raise` to a try statement with a finally >> clause won't prevent the finally clause from executing. >> >> Selective non-idempotent cleanup behaviour really isn't a good idea, >> so the language is fighting you for a reason here - the meaning of a >> "finally" clause is that the code it contains *will* get executed, and >> you have to try incredibly hard to keep that from happening since it's >> an implicit part of cleaning up unfinished generators, and we don't >> let you switch the nominal class of generator objects at runtime. >> Indeed, yield inside try/finally was prohibited for years prior to PEP >> 342, as the runtime previously couldn't guarantee that the finally >> clause would actually execute. >> >> If you *don't* want the code to execute unconditionally, then you need >> to be more explicit about when you *do* want it to execute with some >> combination of "except" and "else" clauses. For example: >> >> >>> @contextmanager >> ... def cm(): >> ... print("enter") >> ... try: >> ... yield >> ... except Exception: >> ... print("Normal exception, not GeneratorExit, >> KeyboardInterrupt or SystemExit") >> ... else: >> ... print("No exception") >> ... >> >>> cm().__enter__() >> enter >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathanfdunn at gmail.com Sun Nov 6 01:28:34 2016 From: nathanfdunn at gmail.com (Nathan Dunn) Date: Sun, 6 Nov 2016 01:28:34 -0500 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) Message-ID: Python has very intuitive and clear syntax, except when it comes to method definitions, particularly dunder methods. class Vec(object): def __init__(self, x, y): self.x, self.y = x, y def __add__(self, other): return Vec(self.x + other.x, self.y + other.y) def __getitem__(self, key): return self.x if key == 'x' else self.y if key == 'y' else None def __contains__(self, item): return self.x == item or self.y == item def __bool__(self): return self.x or self.y def display(self): print('x:', self.x, 'y:', self.y) Having to declare a self parameter is confusing since you don't pass anything in when you call the method on an instance (I am aware of bound vs. unbound methods, etc. but a beginner would not be). The double underscores are also confusing. I propose syntactic sugar to make these method signatures more intuitive and clean. class Vec(object): def class(x, y): self.x, self.y = x, y def self + other: return Vec(self.x + other.x, self.y + other.y) def self[key]: return self.x if key == 'x' else self.y if key == 'y' else None def item in self: return self.x == item or self.y == item def bool(self): return self.x or self.y def self.display(): print('x:', self.x, 'y:', self.y) There are some immediate problems with this, such as `bool(self)` being indistinguishable from a regular method signature and `class(x, y)` not declaring the `self` identifier. These and other problems can be solved to some extent, but I thought I would see if there is any interest around this before going too in depth. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 6 01:53:52 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Nov 2016 16:53:52 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On 6 November 2016 at 16:07, Ram Rachum wrote: > Heh, I just played with this, and found a workaround. If I do something like > this after creating the generator: > > sys.g = g > > Then it wouldn't get closed when Python finishes, and the cleanup won't > happen, which is what I want. The interpreter goes to significant lengths to make sure that finally clauses get executed prior to or during interpreter shutdown, and any means you find by which they don't get executed is considered a bug (not always a fixable bug, but a bug nonetheless). If you rely on those bugs and limitations to get your program to perform the way you want it to you're going to run into problems later when upgrading to new CPython versions, or trying out different interpreter implementations. There's still something seriously odd going in relation to your overall resource management architecture if "cleanup, maybe, unless I decide to tell you not to" is a behaviour you regularly need. Cleanup functions in a garbage collected environment should be idempotent, so it doesn't matter if you redundantly call them again later. However, if you *do* need that pattern regularly, then the pattern itself can be encapsulated in a context manager: class callback_unless_exit: def __init__(self, callback): self.callback = callback def __enter__(self): return self def __exit__(self, exc_type, exc_value, exc_tb): if issubclass(exc_type, GeneratorExit): return self.callback() and then do: with callback_unless_exit(cleanup): yield in the context managers where you want that behaviour. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ram at rachum.com Sun Nov 6 02:18:14 2016 From: ram at rachum.com (Ram Rachum) Date: Sun, 6 Nov 2016 09:18:14 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On Sun, Nov 6, 2016 at 8:53 AM, Nick Coghlan wrote: > On 6 November 2016 at 16:07, Ram Rachum wrote: > > Heh, I just played with this, and found a workaround. If I do something > like > > this after creating the generator: > > > > sys.g = g > > > > Then it wouldn't get closed when Python finishes, and the cleanup won't > > happen, which is what I want. > > The interpreter goes to significant lengths to make sure that finally > clauses get executed prior to or during interpreter shutdown, and any > means you find by which they don't get executed is considered a bug > (not always a fixable bug, but a bug nonetheless). If you rely on > those bugs and limitations to get your program to perform the way you > want it to you're going to run into problems later when upgrading to > new CPython versions, or trying out different interpreter > implementations. > I understand, and I agree with the reasoning. Still, I think I'll take my chances. There's still something seriously odd going in relation to your > overall resource management architecture if "cleanup, maybe, unless I > decide to tell you not to" is a behaviour you regularly need. Cleanup > functions in a garbage collected environment should be idempotent, so > it doesn't matter if you redundantly call them again later. > Well, you think it's weird that I want a `finally` clause to not be called in some circumstances. Do you think it's equally weird to want an `__exit__` method that is not called in some circumstances? > > However, if you *do* need that pattern regularly, then the pattern > itself can be encapsulated in a context manager: > > class callback_unless_exit: > def __init__(self, callback): > self.callback = callback > def __enter__(self): > return self > def __exit__(self, exc_type, exc_value, exc_tb): > if issubclass(exc_type, GeneratorExit): > return > self.callback() > > and then do: > > with callback_unless_exit(cleanup): > yield > > in the context managers where you want that behaviour. > > Thanks for the workaround but I feel it's even less elegant than my original workaround. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 6 02:22:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Nov 2016 17:22:19 +1000 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: References: Message-ID: On 6 November 2016 at 16:28, Nathan Dunn wrote: > There are some immediate problems with this, such as `bool(self)` being > indistinguishable from a regular method signature and `class(x, y)` not > declaring the `self` identifier. These and other problems can be solved to > some extent, but I thought I would see if there is any interest around this > before going too in depth. The syntax is the least confusing part of special method overrides, so if folks are still struggling with that aspect of defining them, there are plenty of other things that are going to trip them up. >From your examples: * __add__ is only part of the addition protocol, there is also __radd__ and __iadd__ * likewise, there is not a one-to-one correspondence between the bool() builtin and the __bool__() special method (there are other ways to support bool(), like defining __len__() on a container) * the mapping protocol covers more than just __getitem__ (and you also need to decide if you're implementing a mapping, sequence, or multi-dimensional array) If the current syntax makes people think "This looks tricky and complicated and harder than defining normal methods", that's a good thing, as magic methods *are* a step up in complexity from normal method definitions, since you need to learn more about how and when they get called and the arguments they receive, while normal methods are accessed via plain function calls. My concern with the last suggestion is different (permitting the first parameter to be specified on the left of the method name), which is that it would break the current symmetry between between name binding in def statements and target binding in assignment statements - currently, all permitted binding targets in def and class statements behave the same way as they do in normal assigment statements, and throw SyntaxError otherwise. With the proposed change, we'd face the problem that the following would both be legal, but meant very different things: cls.mymethod = lambda self: print(self) def cls.mymethod(self): print(self) The former is already legal and assigns the given lambda function as a method on the existing class, `cls` The latter currently throws SyntaxError. With the proposed change, rather than throwing SyntaxError as it does now, the latter would instead be equivalent to: def mymethod(cls, self): print(self) which would be a very surprising difference in behaviour. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brenbarn at brenbarn.net Sun Nov 6 02:35:31 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 06 Nov 2016 00:35:31 -0700 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: <581EDD43.9060606@brenbarn.net> On 2016-11-06 00:18, Ram Rachum wrote: > Well, you think it's weird that I want a `finally` clause to not be > called in some circumstances. Do you think it's equally weird to want an > `__exit__` method that is not called in some circumstances? It's weird to not want the __exit__ to be called if it's defined as a finally block, which is what you're doing with the way you're using contextlib.contextmanager. The __exit__ block there is effectively whatever is after the yield in the generator function you write. If there's code you don't want to always be run, don't put that code inside a finally where the yield is in the try. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From ncoghlan at gmail.com Sun Nov 6 02:38:37 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Nov 2016 17:38:37 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On 6 November 2016 at 17:18, Ram Rachum wrote: > On Sun, Nov 6, 2016 at 8:53 AM, Nick Coghlan wrote: >> There's still something seriously odd going in relation to your >> overall resource management architecture if "cleanup, maybe, unless I >> decide to tell you not to" is a behaviour you regularly need. Cleanup >> functions in a garbage collected environment should be idempotent, so >> it doesn't matter if you redundantly call them again later. > > > Well, you think it's weird that I want a `finally` clause to not be called > in some circumstances. Do you think it's equally weird to want an `__exit__` > method that is not called in some circumstances? Yes, as the whole point of __exit__ is that the interpreter goes to great lengths to make sure it always gets called, no matter what else happens with the currently executing frame (whether it finishes normally, returns early, breaks out of a loop, continues with the next iteration, raises an exception, or gets suspended without ever resuming normal execution). If you don't want that behaviour, then __exit__ likely isn't the right tool (although it may provide the technical basis for a selective cleanup framework). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ram at rachum.com Sun Nov 6 02:44:36 2016 From: ram at rachum.com (Ram Rachum) Date: Sun, 6 Nov 2016 09:44:36 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On Sun, Nov 6, 2016 at 9:38 AM, Nick Coghlan wrote: > On 6 November 2016 at 17:18, Ram Rachum wrote: > > On Sun, Nov 6, 2016 at 8:53 AM, Nick Coghlan wrote: > >> There's still something seriously odd going in relation to your > >> overall resource management architecture if "cleanup, maybe, unless I > >> decide to tell you not to" is a behaviour you regularly need. Cleanup > >> functions in a garbage collected environment should be idempotent, so > >> it doesn't matter if you redundantly call them again later. > > > > > > Well, you think it's weird that I want a `finally` clause to not be > called > > in some circumstances. Do you think it's equally weird to want an > `__exit__` > > method that is not called in some circumstances? > > Yes, as the whole point of __exit__ is that the interpreter goes to > great lengths to make sure it always gets called, no matter what else > happens with the currently executing frame (whether it finishes > normally, returns early, breaks out of a loop, continues with the next > iteration, raises an exception, or gets suspended without ever > resuming normal execution). > > If you don't want that behaviour, then __exit__ likely isn't the right > tool (although it may provide the technical basis for a selective > cleanup framework). > > Cheers, > Nick. > > I understand your point of view. I see that Python does allow you to not call `__exit__` if you don't want to, so I wish it'll have the same approach to not calling `generator.close()` if you don't want to. (This is what it's really about, not `finally`.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Nov 6 03:07:12 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 6 Nov 2016 19:07:12 +1100 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: <20161106080711.GV3365@ando.pearwood.info> On Sun, Nov 06, 2016 at 06:46:40AM +0200, Ram Rachum wrote: > Hi everyone, > > Here is a simplification of a problem that's been happening in my code: > > import contextlib > > @contextlib.contextmanager > def f(): > print('1') > try: > yield > finally: > print('2') > > > g = f() > g.__enter__() > > > This code prints 1 and then 2, not just 1 like you might expect. I expect it to print 2. After all, its in a finally clause. And why are you calling g.__enter__() directly? Calling dunder methods by hand is nearly always the wrong thing to do. > This is > because when the generator is garbage-collected, it gets `GeneratorExit` > sent to it. Right. That's what they're designed to do. Later, in another thread you say: "Well, you think it's weird that I want a `finally` clause to not be called in some circumstances. Do you think it's equally weird to want an `__exit__` method that is not called in some circumstances?" Yes to both. It is seriously weird. You might as well be complaining that Python calls your __iter__ method when you call iter(my_instance). That's the whole point of __iter__, and the whole point of finally clauses and the __exit__ method is that they are unconditionally called, always, when you leave the with block. But judging from your code above, it looks like you're not even using a with block. In that case, instead of abusing the __enter__ and __exit__ methods, why not just create a class with non-dunder enter() and exit() methods and call them by hand? g = f() # implementation of f is left as an exercise g.enter() if condition: g.exit() I'm having a lot of difficulty in understanding your use-case here, and so maybe I've completely misunderstood something. > This has been a problem in my code since in some instances, I tell a > context manager not to do its `__exit__` function. (I do this by using > `ExitStack.pop_all()`. However the `__exit__` is still called here. Have you considered something like: def f(): print('1') try: yield finally: if f.closing: print('2') You can then write a decorator to set f.closing to True or False as needed. But again, I don't understand why you would want this feature, or how you are using it, so I might have this completely wrong. -- Steve From ncoghlan at gmail.com Sun Nov 6 07:33:55 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Nov 2016 22:33:55 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On 6 November 2016 at 17:44, Ram Rachum wrote: > I understand your point of view. I see that Python does allow you to not > call `__exit__` if you don't want to, so I wish it'll have the same approach > to not calling `generator.close()` if you don't want to. (This is what it's > really about, not `finally`.) No, as that's like asking that Python not call close() on files automatically, or not wait for non-daemon threads to terminate when it's shutting down. When Python is discarding a frame that was previously suspended and never finished normally, it throws an exception into it in order to give it a chance to release any resources it might be holding. If you want to deliberately make it leak resources in such cases instead of cleaning them up, you're going to have to leak them deliberately and explicitly, just as you would in normal synchronous code. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eric at trueblade.com Sun Nov 6 08:06:27 2016 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 6 Nov 2016 08:06:27 -0500 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions Message-ID: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Creating a new thread, instead of hijacking the PEP 532 discussion. From PEP 532: > Abstract > ======== > > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP > proposes the addition of a new protocol-driven circuit breaking operator to > Python that allows the left operand to decide whether or not the expression > should short circuit and return a result immediately, or else continue > on with evaluation of the right operand:: > > exists(foo) else bar > missing(foo) else foo.bar() Instead of new syntax that only works in this one specific case, I'd prefer a more general solution. I accept being "more general" probably seals the deal in killing any proposal! I realize the following proposal has at least been hinted at before, but I couldn't find a specific discussion about it. Since it applies to the short-circuiting issues addressed by PEP 532 and its predecessors, I thought I'd bring it up here. It could also be used to solve some of the problems addressed by the rejected PEP 463 (Exception-catching expressions). See also PEP 312 (Simple Implicit Lambda). It might also be usable for some of the use cases presented in PEP 501 (General purpose string interpolation, aka i-strings). I'd rather see the ability to have unevaluated expressions, that can later be evaluated. I'll use backticks here to mean: "parse, but do not execute the enclosed code". This produces an object that can later be evaluated with a new builtin I'll call "evaluate_now". Obviously these are strawmen, and partly chosen to be ugly and unacceptable names and symbols in the form I'll discuss here. Then you could write a function: eval_else(`foo.bar`, `some_func()`) whose value is foo.bar, unless foo.bar cannot be evaluated, in which case the value is some_func(). def eval_else(expr, fallback, exlist=(AttributeError,)): try: return evaluate_now(expr) except exlist: return evaluate_now(fallback) Exactly which exceptions you catch is up to you. Of course there's the chance that someone would pass in something for which the caught exception is too broad, and it's raised deep inside evaluating the first expression, but that's no different than catching exceptions now. Except I grant that hiding the try/except inside a called function increases the risk. Like f-strings, the expressions are entirely created at the site they're specified inside ``. So they'd have access to locals and globals, etc., at the definition site. def x(foo, i): return eval_else(`foo.bar`, `some_func(i, __name__)`) And like the expressions in f-strings, they have to be valid expressions. But unlike f-strings, they aren't evaluated right when they're encountered. The fact that they may never be evaluated is one of their features. For example the if/else expression: if_else(`y`, x is None, `x.a`) could be defined as being exactly like: y if x is None else x.a including only evaluating x.a if x is not None. def if_else(a, test, b): if test: return evaluate_now(a) return evaluate_now(b) You could do fancier things that require more than 2 expressions. Whether `` returns an AST that could later be manipulated, or it's something else that's opaque is another discussion. Let's assume it's opaque for now. You could go further and say that any argument to a function that's specially marked would get an unevaluated expression. Suppose that you can mark arguments as & to mean "takes an unevaluated expression". Then you could write: def if_else(&a, test, &b): if test: return evaluate_now(a) return evaluate_now(b) And call it as: if_else(y, x is None, x.a) But now you've made it non-obvious at the caller site exactly what's happening. There are other downsides, such as only being able to create an unevaluated expression when calling a function. Or maybe that's a good thing! In any event, having unevaluated expressions would open up more possibilities than just the short-circuit evaluation model. And it doesn't involve a new protocol. Eric. From eric at trueblade.com Sun Nov 6 09:31:06 2016 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 6 Nov 2016 09:31:06 -0500 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: [top posting from my phone] Chris Angelico points out the & part of the idea interacts poorly with *args and **kwargs, so I drop that idea. Re-reading PEP 312, this idea is basically identical, with different spellings. The point remains: do we want to be able to create unevaluated expressions that can be evaluated at a different point? -- Eric. > On Nov 6, 2016, at 8:06 AM, Eric V. Smith wrote: > > Creating a new thread, instead of hijacking the PEP 532 discussion. > > From PEP 532: > > > Abstract > > ======== > > > > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP > > proposes the addition of a new protocol-driven circuit breaking operator to > > Python that allows the left operand to decide whether or not the expression > > should short circuit and return a result immediately, or else continue > > on with evaluation of the right operand:: > > > > exists(foo) else bar > > missing(foo) else foo.bar() > > Instead of new syntax that only works in this one specific case, I'd prefer a more general solution. I accept being "more general" probably seals the deal in killing any proposal! > > I realize the following proposal has at least been hinted at before, but I couldn't find a specific discussion about it. Since it applies to the short-circuiting issues addressed by PEP 532 and its predecessors, I thought I'd bring it up here. It could also be used to solve some of the problems addressed by the rejected PEP 463 (Exception-catching expressions). See also PEP 312 (Simple Implicit Lambda). It might also be usable for some of the use cases presented in PEP 501 (General purpose string interpolation, aka i-strings). > > I'd rather see the ability to have unevaluated expressions, that can later be evaluated. I'll use backticks here to mean: "parse, but do not execute the enclosed code". This produces an object that can later be evaluated with a new builtin I'll call "evaluate_now". Obviously these are strawmen, and partly chosen to be ugly and unacceptable names and symbols in the form I'll discuss here. > > Then you could write a function: > > eval_else(`foo.bar`, `some_func()`) > > whose value is foo.bar, unless foo.bar cannot be evaluated, in which case the value is some_func(). > > def eval_else(expr, fallback, exlist=(AttributeError,)): > try: > return evaluate_now(expr) > except exlist: > return evaluate_now(fallback) > > Exactly which exceptions you catch is up to you. Of course there's the chance that someone would pass in something for which the caught exception is too broad, and it's raised deep inside evaluating the first expression, but that's no different than catching exceptions now. Except I grant that hiding the try/except inside a called function increases the risk. > > Like f-strings, the expressions are entirely created at the site they're specified inside ``. So they'd have access to locals and globals, etc., at the definition site. > > def x(foo, i): > return eval_else(`foo.bar`, `some_func(i, __name__)`) > > And like the expressions in f-strings, they have to be valid expressions. But unlike f-strings, they aren't evaluated right when they're encountered. The fact that they may never be evaluated is one of their features. > > For example the if/else expression: > > if_else(`y`, x is None, `x.a`) > > could be defined as being exactly like: > > y if x is None else x.a > > including only evaluating x.a if x is not None. > > def if_else(a, test, b): > if test: > return evaluate_now(a) > return evaluate_now(b) > > You could do fancier things that require more than 2 expressions. > > Whether `` returns an AST that could later be manipulated, or it's something else that's opaque is another discussion. Let's assume it's opaque for now. > > You could go further and say that any argument to a function that's specially marked would get an unevaluated expression. Suppose that you can mark arguments as & to mean "takes an unevaluated expression". Then you could write: > > def if_else(&a, test, &b): > if test: > return evaluate_now(a) > return evaluate_now(b) > > And call it as: > if_else(y, x is None, x.a) > > But now you've made it non-obvious at the caller site exactly what's happening. There are other downsides, such as only being able to create an unevaluated expression when calling a function. Or maybe that's a good thing! > > In any event, having unevaluated expressions would open up more possibilities than just the short-circuit evaluation model. And it doesn't involve a new protocol. > > Eric. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From tjreedy at udel.edu Sun Nov 6 09:40:09 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 6 Nov 2016 09:40:09 -0500 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: On 11/6/2016 2:18 AM, Ram Rachum wrote: > Well, you think it's weird that I want a `finally` clause to not be > called in some circumstances. Do you think it's equally weird to want an > `__exit__` method that is not called in some circumstances? Without a deeper understanding of why you want to do so, I would. The automatic exit cleanup typically the main purpose of a context manager and 'with' block. If, for instance, I want a file only maybe closed, I would just call 'open' instead of 'with open' and then conditionally (with 'if' or 'try') call 'close'. -- Terry Jan Reedy From danilo.bellini at gmail.com Sun Nov 6 13:46:42 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Sun, 6 Nov 2016 16:46:42 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> Message-ID: 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > As for recursion, the syntax you proposed doesn't say "recursion" to > me, it says "let's add more complexity to a syntax that already is > hard to fit on a line so we can turn a sequence into a series." > 1. The "fit on a line" is just another weird taboo: 1.1. An expression isn't a physical line. 1.2. Sub-expressions in an expression might be on other statements (e.g. assignments, other functions). 1.4. Several of my examples in PyScanPrev aren't oneliners, including one you pasted in an e-mail. 1.3. When you call itertools.accumulate, it's still an expression. 1.5. List comprehensions are expressions, including the triple-for-section comprehensions with repeated target variable names. 1.6. Being anti-expression because of an anti-oneliner bias sounds like moving towards an assembly language (using only atomic imperative commands) or something merely bureaucratic. 2. The itertools.accumulate function signature is broken: 2.1. Its arguments are reversed when compared to other higher order functions like map, filter and reduce. 2.2. It lacks a "start" parameter, requiring more complexity to include it (nesting a itertools.chain or a "prepend" function call). 3. It's not about "adding complexity", it's the other way around: 3.1. The proposal is about explicit recursion in a list comprehension (a way to access the previous output value/result, a.k.a. accumulator/state/memory). 3.2. Today you can only do (3.1) using either: 3.2.1. A triple-for-section list comprehension with repeated target names (as described in the PyScanPrev rationale section). 3.2.2. Functions for stack frame manipulation (like the example I gave in this maillist using stackfull). 3.2.3. Bytecode manipulation (the PyScanPrev approach), or something alike (e.g. AST manipulation). 3.2.4. Raw Python code pre-processing (e.g. by customizing some import hooks). Only 3.2.4 allows a new syntax. I'm not sure what you meant with "turn a sequence into a series", that sounds like the itertools.accumulate from Python 3.2, which didn't have a function as a parameter. 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > > Anyway, that makes this talking about computational accuracy sound > > like an argument for my proposal here, not against it. > > Not as I see it. My point is that functions like accumulate already > get me as much of your proposal as I can see being useful in my own > applications, and so I'd rather spend effort on the inherent > complexity of accurate computation than on learning new syntax which > as far as I can see buys me no extra simplicity. > What do you mean by the word "accuracy"? IMHO, now you're not talking about accuracy anymore. It's an argument like "map/filter and generator expressions are the same" again. That's just an argument against generator expressions and list/set/dict comprehensions in general, as they "already get us as much of" map/filter. About the effort, do you really find the examples below with the new proposed syntax difficult to understand? >>> # Running product >>> [prev * k for k in [5, 2, 4, 3] from prev = 1] [1, 5, 10, 40, 120] >>> # Accumulate (prefix sum / cumulative sum) >>> [prev + k for k in [5, 2, 4, 3] from prev = 0] [0, 5, 7, 11, 14] >>> # Pairs to be used in the next example (nothing new here) >>> pairs = [(a, b) for a in [0, 1, 2, 3] for b in [-2, 2] if b != a] >>> pairs [(0, -2), (0,2), (1, -2), (1, 2), (2, -2), (3, -2), (3, 2)] >>> # The recursive list comprehension with the new syntax >>> [b + prev * a for a, b in pairs from prev = 0] [0, -2, 2, 0, 2, 2, 4, 14] >>> # The same with itertools.accumulate (for comparison) >>> from itertools import accumulate, chain >>> list(accumulate(chain([0], pairs), ... lambda prev, pair: pair[1] + prev * pair[0])) [0, -2, 2, 0, 2, 2, 4, 14] >>> # The same in a single expression using the new syntax >>> [b + prev * a for a in [0, 1, 2, 3] ... for b in [-2, 2] ... if b != a ... from prev = 0] [0, -2, 2, 0, 2, 2, 4, 14] Everything needs some effort to be understood, some effort to be memorized, some effort to be maintained, etc.. But Rob Cliffe could understand the running product when he had read this thread and disclaimed that he never heard about accumulate. For someone who doesn't know what accumulate/scan/fold/reduce is, I think the proposed syntax is more descriptive/explicit/instructional/didactic/maintainable. Please read the "Rationale" section and the "Comparison" example in PyScanPrev. Also, please keep in mind that even with bytecode manipulation, PyScanPrev is still limited to valid Python syntax, and the syntax proposed here is different and more general (e.g. the last example in this e-mail can't be done in PyScanPrev as a single list comprehension). 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > Consider the existing comprehension syntax. I use it all the time > because it's very expressive, it "looks like" what I would write in a > declarative mathematical definition, and the constructive alternative > is not a function call, it's a loop, in many cases with a nested if > test as well. However, when you try to express more than one loop > with a comprehension, you end up with a counterintuitive ordering > based on the current "this expands naturally to for loops and if > conditionals according to this simple rule" syntax, along with a > separation of the result expression from the loop where it's actually > produced. It's not obvious to me that use of comprehension syntax > beyond [ f(x) for x in y if g(x) ] is all that useful, but that very > basic form is extremely useful by itself. > And what I'm proposing is a very expressive syntax that "looks like" what I would write in a declarative mathematical definition. The state-space example is really explicit on this. Also, it's something VERY useful, I've written several examples and I know they aren't exhaustive. Counterintuitive ordering in list comprehensions? More than one loop? You don't like the multiple for sections cartesian product in a list comprehension, as it seems. That's not the point in this proposal, anyway. In your example, we can use either the generator expression (f(x) for x in y if g(x)) or the point-free map(f, filter(g, y)). Why do you prefer the first? You said "it's not a function call, it's a loop", but at the end you wrote "f(x)" and "g(x)". Maybe that's why you didn't see the recursion: it's tail-recursive / stackless. Say we have: [f(p, x) for x in y from p = z] The result for y = [y0, y1, y2, ...] would be: [z, f(z, y0), f(f(z, y0), y1), f(f(f(z, y0), y1), y2), ...] 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > If your proposal will help make more complex comprehensions easy to > understand, then you've got something I'd like to have. But I don't > yet see how it fixes that, and if not, I ain't gonna need it and I > don't want to have to explain it to my students when they ain't gonna > need it either. > Nice to know. The proposed: >>> [f(p, x) for x in y from p = z] Should return the same from this, if one insists on using a list comprehension today without the new syntax: >>> (lambda start: [start] + [p for p in [start] ... for x in y ... for p in [f(p, x)]] ... )(z) The proposed comprehension is easier/simpler than that in every criterion I'm aware of. 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > itertools functions are widely used and generally well-thought-of. > Avoiding accumulate is a non-goal AFAIK. AIUI, Guido does think that > avoiding reduce is a goal, but I believe that's because he thinks the > semantics are just plain hard to understand. If your syntax is in > large part a substitute for reduce, I doubt he will like it more than > reduce just because it's syntax. (I do not speak for Guido; I offer > my observation in the hope it will help you prepare to present your > idea to him in the most favorable light.) > I doubt Guido is against solving problems that requires a fold to be solved. AFAIK he's against the "reduce" higher order function itself, not against what it does. Mixing that is like acting against list comprehensions just because Guido doesn't seem to like the map/filter higher order functions. Guido either prefers what exists or an alternative, I'm pretty sure he wouldn't ever say "solving this kind of problem is forbidden in Python". And that's again an argument for my proposal, as getting a fold from a scan is obvious: it's just the last result. Using reduce, let's get the last value after scanning through the "pairs" I defined in a previous example in this e-mail: >>> # The last value with reduce (Python 2 only, for comparison) >>> reduce(lambda prev, (a, b): b + prev * a, pairs, 0) 14 >>> # ... and with functools.reduce (Python 2 and 3) >>> from functools import reduce >>> reduce(lambda prev, pair: pair[1] + prev * pair[0], pairs, 0) 14 Actually, using a [:-1] slice or iterating until we get the last element in the previous [0, -2, 2, 0, 2, 2, 4, 14] result would be way more explicit (and wouldn't require the "reduce" function itself). I've proposed a "last" function to do that, but that's another point, not so required in the sense that one can write it as a function elsewhere. In Python 2.7, it's worth noting that reduce doesn't require the import, and lambdas are more powerful as they allow the "prev, (a, b)" signature. This "unpacking" syntax was lost in Python 3 but not when we're dealing with "for" loops/comprehensions. Something similar could be said about itertools.accumulate. 2016-11-03 8:10 GMT-02:00 Stephen J. Turnbull : > That is, comprehensions are clearly useful, and I use them in almost > every program I write. I still don't see when I'd ever strongly > prefer the syntax you propose to the itertools we already have, so I > don't care if it's a consistent extension. That doesn't mean it's not > useful to you or others, it just means I'm not convinced -- and so I > will not join the proponents. That's all. > That's not really about consistency, it sounds more like a "Don't touch!". There are 2 consistencies we're talking about here: API consistency and argumentation consistency. APIs can have backward compatibility issues and legacy code, which are sources of inconsistency. OTOH, inconsistency in argumentation is either a mistake or a deceitful behavior. For me, the idea I'm proposing is clearly useful. I wanted to use it last friday to create a function that auto-indents some text based on some simple rules, and the resulting indentation is the "accumulation" (scan) of indent/dedent markers. But I couldn't even use itertools.accumulate, because that project requires Python 2.7. I need a really strong reason to migrate to Python 3 a proprietary legacy code that isn't mine, and any "anti-functional" / "anti-expression" bias is a strong reason not to do so. Also, it's worth noting that the "functional" package in PyPI doesn't work on Python 3. Stuff like the f-strings are really nice, I can't count how many times I wrote some_string_literal.format(**locals()), but replacing all lambda signatures by cryptic indexing or naming them elsewhere still doesn't worth it (Would you prefer to replace every single list comprehension body by a named function defined with "def"? If not, why others should?). I'm proposing something that would make Python 3 more inviting, and not only for people coming from a functional programming background. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Nov 6 15:00:14 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 7 Nov 2016 05:00:14 +0900 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> Message-ID: <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> Danilo J. S. Bellini writes: > About the effort, do you really find the examples below with the new > proposed syntax difficult to understand? No. I just don't see how they would become tools I would use. My main interest here was in your claim to have economic applications, but the examples you give don't seem to offer big wins for the kind of calculations I, my students, or my colleagues do. Perhaps you will have better luck interesting/persuading others. From danilo.bellini at gmail.com Sun Nov 6 17:01:50 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Sun, 6 Nov 2016 20:01:50 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> Message-ID: 2016-11-06 18:00 GMT-02:00 Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp>: > Danilo J. S. Bellini writes: > > > About the effort, do you really find the examples below with the new > > proposed syntax difficult to understand? > > No. I just don't see how they would become tools I would use. My main > interest here was in your claim to have economic applications, but the > examples you give don't seem to offer big wins for the kind of > calculations I, my students, or my colleagues do. Perhaps you will > have better luck interesting/persuading others. > If you want something simple, the itertools.accumulate examples from docs.python.org include a simple "loan amortization" example: >>> # Amortize a 5% loan of 1000 with 4 annual payments of 90 >>> cashflows = [1000, -90, -90, -90, -90] >>> list(accumulate(cashflows, lambda bal, pmt: bal*1.05 + pmt)) [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] >>> # With the proposed syntax >>> payments = [90, 90, 90, 90] >>> [bal * 1.05 - pmt for pmt in payments from bal = 1000] [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] >From the Wilson J. Rugh "Linear Systems Theory" book, chapter 20 "Discrete Time State Equations", p. 384-385 (the very first example on the topic): """ A simple, classical model in economics for national income y(k) in year k describes y(k) in terms of consumer expenditure c(k), private investment i(k), and government expenditure g(k) according to: y(k) = c(k) + i(k) + g(k) These quantities are interrelated by the following assumptions. First, consumer expenditure in year k+1 is proportional to the national income in year k, c(k+1) = ??y(k) where the constant ? is called, impressively enough, the marginal propensity to consume. Second, the private investment in year k+1 is proportional to the increase in consumer expenditure from year k to year k+1, i(k+1) = ??[c(k+1) - c(k)] where the constant ? is a growth coefficient. Typically 0 < ? < 1 and ? > 0. >From these assumptions we can write the two scalar difference equations c(k+1) = ??c(k) + ??i(k) + ??g(k) i(k+1) = (???-?)?c(k) + ????i(k) + ????g(k) Defining state variables as x?(k) = c(k) and x?(k) = i(k), the output as y(k), and the input as g(k), we obtain the linear state equation # ? ? ? ? ? ? ? # x(k+1) = ? ??x(k) + ? ??g(k) # ???(?-1) ???? ????? # # y(k) = [1 1]?x(k) + g(k) Numbering the years by k = 0, 1, ..., the initial state is provided by c(0) and i(0). """ You can use my "ltiss" or "ltvss" (if alpha/beta are time varying) functions from the PyScanPrev state-space example to simulate that, or some dedicated function. The linear time varying version with the proposed syntax would be (assuming alpha, beta and g are sequences like lists/tuples): >>> from numpy import mat >>> def first_digital_linear_system_example_in_book(alpha, beta, c0, i0, g): ... A = (mat([[a, a ], ... [b*(a-1), b*a]]) for a, b in zip(alpha, beta)) ... B = (mat([[a ], ... [b*a]]) for a, b in zip(alpha, beta)) ... x0 = mat([[c0], ... [i0]]) ... x = (Ak*xk + Bk*gk for Ak, Bk, gk in zip(A, B, g) from xk = x0) ... return [xk.sum() + gk for xk, gk in zip(x, g)] If A and B were constants, it's simpler, as the scan line would be: x = (A*xk + B*gk for gk in g from xk = x0) -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From askvictor at gmail.com Sun Nov 6 17:25:08 2016 From: askvictor at gmail.com (victor rajewski) Date: Sun, 06 Nov 2016 22:25:08 +0000 Subject: [Python-ideas] PythonOS Message-ID: Firstly, apologies if this is the wrong forum for this idea; it's not so much about the language itself, but what surrounds it. If you have any ideas for better forums, please let me know. Also, if there is any work started on anything like this, let me know Now to the idea: PythonOS - a (likely linux-based) OS built specifically for python. I've been working in the education space as well as with embedded systems for quite a few years, and am really excited about tiny affordable standalone computing devices as a computing and science educational tool. At the moment however, there are two options for python in this space: - micropython, which, despite its awesomeness, lacks many library possibilities, doesn't have a debugger, and is restricted to a handful of devices. The learning curve for getting this up and running can be quite steep - A full linux setup on an SBC - let's take the raspberry pi. This has access to (just about) everything, but beyond just programming the python code, you need to have some (basic) system administration training to get it running. For someone new to programming, this can be quite intimidating (recent developments with drag-and-drop Pi configuration notwithstanding). In the latter category, there seems to be a lot of newer, cheaper hardware appearing - for example the Pi Zero, C.H.I.P and VoCore2, and these are already at very affordable price point; I think that before long the rationale for micropython will be lost, and $2 hardware will be running a fully capable linux setup. However, micropython has the advantage of simplicity. Upload main.py, and press reset. That's it. My proposal is a Linux distribution that has the simplicity of micropython, but the power of full python. Drop a file (let's call it main.py) on an SD card from your laptop, plug it into the device, and it boots and runs that file straight away. Or drop an entire project. If the device has USB device capabilities, then just drag the file across, or even edit it live. Easy SSH setup could allow more advanced users remote development and debugging options. Maybe jupyter could be leveraged to provide other network-based dev options (pynq already has a working linux distro like this). Connecting to a console would give you a python prompt. Ideally, key OS functions should be able to be controlled from within python, so that the user never has to leave the python prompt. For a start, things like network config, and possibly cron. What do you think? I'm not in a state to do all of this myself, but could give it a start if there is interest in the concept. -- Victor Rajewski Sent from my mobile device. please. Please excuse brevity and any errors. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Sun Nov 6 17:36:42 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Sun, 6 Nov 2016 20:36:42 -0200 Subject: [Python-ideas] PythonOS In-Reply-To: References: Message-ID: On 11/06/2016 08:25 PM, victor rajewski wrote: > What do you think? I'm not in a state to do all of this myself, but > could give it a start if there is interest in the concept. Even though there is nothing wrong with the idea, I think there is not enough motivation for it. Most of the people learning Python for tinkering around with robots or doing some statistics will both be able to and eventually need to learn some basic system administration. Also, in comparison with some other languages, at least under Linux and BSD systems, setting up a Python environment is very straightforward. It is packaged for most distributions out there and executing a script does not require any cryptic initiation rituals. All in all, I don't know if this solves a problem or not. > > Sent from my mobile device. please. Please excuse brevity and any errors. > Did you type all that in a touchscreen? -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From greg.ewing at canterbury.ac.nz Sun Nov 6 16:28:12 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 07 Nov 2016 10:28:12 +1300 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: <581FA06C.4040606@canterbury.ac.nz> Eric V. Smith wrote: > I'd rather see the ability to have unevaluated expressions, that can > later be evaluated. I'll use backticks here to mean: "parse, but do not > execute the enclosed code". This produces an object that can later be > evaluated with a new builtin I'll call "evaluate_now". So far you've essentially got a compact notation for a lambda with no arguments. Suggestions along these lines have been made before, but didn't go anywhere. > You could go further and say that any argument to a function that's > specially marked would get an unevaluated expression. Suppose that you > can mark arguments as & to mean "takes an unevaluated expression". Now *that* would be truly interesting, but it would complicate some fundamental parts of the implementation tremendously. Before calling any function, it would be necessary to introspect it to find out which parameters should be evaluated. Alternatively, every parameter would have to be passed as a lambda, with the function deciding which ones to evaluate. I fear this would be far too big a change to swallow. -- Greg From wes.turner at gmail.com Sun Nov 6 20:27:35 2016 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 6 Nov 2016 19:27:35 -0600 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> Message-ID: - So, IIUC, for recursive list comprehensions - "prev" = x_(n-1) - there is a need to define an initial value - chain([1000], [...]) - sometimes, we actually need window function - __[0] = x_(n-1) - __[1] = x_(n-2) # this - __[-1] = x_(n-2) # or this - this can be accomplished with dequeue - __= dequeue([1000], maxlen) - for recursive list comprehensions, we'd want to bind e.g. __ to a dequeue [f(__[0], x) for x in y with __ = dequeue((1000,), 1)] But the proposed syntax differs from this interpretation: - "from bal = 1000" # ~= with prev = dequeue((1000,), 1)[-1] (Recursive) fibonacci would then require a dequeue (..., 2) Other than brevity, is there any advantage to list comprehensions over a for loop? - IIRC, reduce() and fold() can avoid unnecessary variable binding, but require lower-level debugging. A recursive list comprehension syntax would be cool. Is there a better variable name than '__'? On Sunday, November 6, 2016, Danilo J. S. Bellini wrote: > 2016-11-06 18:00 GMT-02:00 Stephen J. Turnbull tsukuba.ac.jp > >: > >> Danilo J. S. Bellini writes: >> >> > About the effort, do you really find the examples below with the new >> > proposed syntax difficult to understand? >> >> No. I just don't see how they would become tools I would use. My main >> interest here was in your claim to have economic applications, but the >> examples you give don't seem to offer big wins for the kind of >> calculations I, my students, or my colleagues do. Perhaps you will >> have better luck interesting/persuading others. >> > > If you want something simple, the itertools.accumulate examples from > docs.python.org include a simple "loan amortization" example: > > >>> # Amortize a 5% loan of 1000 with 4 annual payments of 90 > >>> cashflows = [1000, -90, -90, -90, -90] > >>> list(accumulate(cashflows, lambda bal, pmt: bal*1.05 + pmt)) > [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] > > >>> # With the proposed syntax > >>> payments = [90, 90, 90, 90] > >>> [bal * 1.05 - pmt for pmt in payments from bal = 1000] > [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] > > > From the Wilson J. Rugh "Linear Systems Theory" book, chapter 20 "Discrete > Time State Equations", p. 384-385 (the very first example on the topic): > > """ > A simple, classical model in economics for national income y(k) in year k > describes y(k) in terms of consumer expenditure c(k), private investment > i(k), and government expenditure g(k) according to: > > y(k) = c(k) + i(k) + g(k) > > These quantities are interrelated by the following assumptions. First, > consumer expenditure in year k+1 is proportional to the national income in > year k, > > c(k+1) = ??y(k) > > where the constant ? is called, impressively enough, the marginal > propensity to consume. Second, the private investment in year k+1 is > proportional to the increase in consumer expenditure from year k to year > k+1, > > i(k+1) = ??[c(k+1) - c(k)] > > where the constant ? is a growth coefficient. Typically 0 < ? < 1 and ? > > 0. > > From these assumptions we can write the two scalar difference equations > > c(k+1) = ??c(k) + ??i(k) + ??g(k) > i(k+1) = (???-?)?c(k) + ????i(k) + ????g(k) > > Defining state variables as x?(k) = c(k) and x?(k) = i(k), the output as > y(k), and the input as g(k), we obtain the linear state equation > > # ? ? ? ? ? ? ? > # x(k+1) = ? ??x(k) + ? ??g(k) > # ???(?-1) ???? ????? > # > # y(k) = [1 1]?x(k) + g(k) > > Numbering the years by k = 0, 1, ..., the initial state is provided by > c(0) and i(0). > """ > > You can use my "ltiss" or "ltvss" (if alpha/beta are time varying) > functions from the PyScanPrev state-space example to simulate that, or some > dedicated function. The linear time varying version with the proposed > syntax would be (assuming alpha, beta and g are sequences like > lists/tuples): > > >>> from numpy import mat > >>> def first_digital_linear_system_example_in_book(alpha, beta, c0, i0, > g): > ... A = (mat([[a, a ], > ... [b*(a-1), b*a]]) for a, b in zip(alpha, beta)) > ... B = (mat([[a ], > ... [b*a]]) for a, b in zip(alpha, beta)) > ... x0 = mat([[c0], > ... [i0]]) > ... x = (Ak*xk + Bk*gk for Ak, Bk, gk in zip(A, B, g) from xk = x0) > ... return [xk.sum() + gk for xk, gk in zip(x, g)] > > If A and B were constants, it's simpler, as the scan line would be: > > x = (A*xk + B*gk for gk in g from xk = x0) > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Nov 6 20:32:29 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 Nov 2016 17:32:29 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On Sun, Nov 6, 2016 at 5:06 AM, Eric V. Smith wrote: > Creating a new thread, instead of hijacking the PEP 532 discussion. > > From PEP 532: > >> Abstract >> ======== >> >> Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this >> PEP >> proposes the addition of a new protocol-driven circuit breaking operator >> to >> Python that allows the left operand to decide whether or not the >> expression >> should short circuit and return a result immediately, or else continue >> on with evaluation of the right operand:: >> >> exists(foo) else bar >> missing(foo) else foo.bar() > > Instead of new syntax that only works in this one specific case, I'd prefer > a more general solution. I accept being "more general" probably seals the > deal in killing any proposal! > > I realize the following proposal has at least been hinted at before, but I > couldn't find a specific discussion about it. Since it applies to the > short-circuiting issues addressed by PEP 532 and its predecessors, I thought > I'd bring it up here. It could also be used to solve some of the problems > addressed by the rejected PEP 463 (Exception-catching expressions). See also > PEP 312 (Simple Implicit Lambda). It might also be usable for some of the > use cases presented in PEP 501 (General purpose string interpolation, aka > i-strings). > > I'd rather see the ability to have unevaluated expressions, that can later > be evaluated. I'll use backticks here to mean: "parse, but do not execute > the enclosed code". This produces an object that can later be evaluated with > a new builtin I'll call "evaluate_now". Obviously these are strawmen, and > partly chosen to be ugly and unacceptable names and symbols in the form I'll > discuss here. If we're considering options along these lines, then I think the local optimum is actually a "quoted-call" operator, rather than a quote operator. So something like (borrowing Rust's "!"): eval_else!(foo.bar, some_func()) being sugar for eval_else.__macrocall__(, ) You can trivially use this to recover a classic quote operator if you really want one: def quote!(arg): return arg but IMO this way is more ergonomic for most use cases (similar to your '&' suggestion), while retaining the call-site marking that "something magical is happening here" (which is also important, both for readability + implementation simplicity -- it lets the compiler know statically when it needs to retain the AST, solving the issue that Greg pointed out). Some other use cases: Log some complicated object, but only pay the cost of stringifying the object if debugging is enabled: log.debug!(f"Message: {message_object!r}") Generate a plot where the axes are automatically labeled "x" and "np.sin(x)" (this is one place where R's plotting APIs are more convenient than Python's): import numpy as np import matplotlib.pyplot as plt x = np.linspace(0, 10) plt.plot!(x, np.sin(x)) What PonyORM does, but without the thing where currently their implementation involves decompiling bytecode...: db.select!(c for c in Customer if sum(c.orders.price) > 1000) Filtering out a subset of rows from a data frame in pandas; 'height' and 'age' refer to columns in the data frame (equivalent to data_frame[data_frame["height"] > 100 and data_frame["age"] < 5], but more ergonomic and faster (!)): data_frame.subset!(height > 100 and age < 5) (IIRC pandas has at least experimented with various weird lambda hacks for this kind of thing; not sure what the current status is.) Every six months or so I run into someone who's really excited about the idea of adding macros to python, and I suggest this approach. So far none of them have been excited enough to actually write a PEP, but if I were going to write a PEP then this is the direction that I'd take :-). -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Sun Nov 6 20:27:25 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 Nov 2016 12:27:25 +1100 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: References: Message-ID: <20161107012724.GW3365@ando.pearwood.info> On Sun, Nov 06, 2016 at 01:28:34AM -0500, Nathan Dunn wrote: > Python has very intuitive and clear syntax, except when it comes to method > definitions, particularly dunder methods. I disagree with your premise here. Python's method definitions are just as intuitive and clear as the rest of Python's syntax: methods are just functions, indented in the body of the class where they belong, with an explicit "self" parameter. And dunder methods are just a naming convention. They're not the most visually attractive methods, due to the underscores, but its just a naming convention. Otherwise they are declared in exactly the same way as any other method: using normal function syntax, indented inside the body of the class, with an explicit "self" the same as other methods. So there's no magic to learn. Once you know how to declare a function, it is a tiny step to learn to declare a method: put it inside a class, indent it, and add "self", and now you have a method. And once you know how to declare a method, there's nothing more to learn to handle dunder methods. All you need know is the name of the method or methods you need, including the underscores. [...] > Having to declare a self parameter is confusing since you don't pass > anything in when you call the method on an instance (I am aware of bound > vs. unbound methods, etc. but a beginner would not be). You are mistaking "mysterious" for "confusing". "Why do I have to explicitly declare a self parameter?" is a mystery, and the answer can be given as: - you just do - because internally methods are just functions - because it is actually useful (e.g. for unbound methods) depending on the experience of the person asking. But its not *confusing*. "Sometimes I have to implicitly declare self, and sometimes I don't, and there doesn't seem to be any pattern to which it is" would be confusing. "Always explicitly declare self" is not. > The double underscores are also confusing. I've certainly a few cases of people who misread __init__ as _init_ and was surprised by their code not working. In over a decade of dealing with beginners' questions on comp.lang.python and the tutor mailing list. So it is an easy mistake to make, but apparently a *rare* mistake to make, and very easy to correct. So I disagree that double underscores are "confusing". What is confusing about the instructions "press underscore twice at the beginning and end of the method name"? > I propose syntactic sugar to make these method signatures more intuitive > and clean. > > class Vec(object): > def class(x, y): > self.x, self.y = x, y I don't think that there is anything intuitive about changing the name of the method from __init__ to "class". What makes you think that people will intuit the word "class" to create instance? That seems like a dubious idea to me. And it certainly isn't *clean*. At the moment, Python's rules are nicely clean: keywords can never be used as identifiers. You would either break that rule, or have some sort of magic where *some* keywords can *sometimes* be used as identifiers, but not always. That's the very opposite of clean -- it is a nasty, yucky design, and it doesn't scale to other protocols: def with: # is this __enter__ or __exit__? It doesn't even work for instance construction! Is class(...) the __new__ or __init__ method? Not all beginners to Python are beginners to programming at all. Other languages typically use one of three naming conventions for the constructor: - a method with the same name as the class itself e.g. Java, C#, PHP 4, C++, ActionScript. - special predefined method names e.g. "New" in VisualBasic, "alloc" and "init" in Objective C, "initialize" in Ruby, "__construct" in PHP 5. - a keyword used before an otherwise normal method definition e.g. "constructor" in Object Pascal, "initializer" in Ocaml, "create" in Eiffel, "new" in F#. So there's lots of variation in how constructors are written, and what seems "intuitive" will probably depend on the reader's background. Total beginners to OOP don't have any pre-conceived expectations, because the very concept of initialising an instance is new to them. Whether it is spelled "New" or "__init__" or "mzygplwts" is just a matter of how hard it is to spell correctly and memorise. > def self + other: > return Vec(self.x + other.x, self.y + other.y) My guess is that this is impossible in a LL(1) parser, but even if possible, how do you write the reversed __radd__ method? My guess is that you would need: def other + self: but for that to work, "self" now needs to be a keyword rather than just a regular identifier which is special only because it is the first in the parameter list. And that's a problem because there are cases (rare, but they do happen) where we don't want to use "self" for the instance parameter. A very common pattern in writing classes is: def __add__(self, other): # implementation goes here __radd__ = __add__ since addition is usually commutative. How would your new syntax handle that? -- Steve From steve at pearwood.info Sun Nov 6 20:55:21 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 Nov 2016 12:55:21 +1100 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> Message-ID: <20161107015520.GX3365@ando.pearwood.info> On Sun, Nov 06, 2016 at 04:46:42PM -0200, Danilo J. S. Bellini wrote: > 1.2. Sub-expressions in an expression might be on other statements (e.g. > assignments, other functions). Not in Python it can't be. Assignment is not an expression, you cannot say (x = 2) + 1. > 2. The itertools.accumulate function signature is broken: > 2.1. Its arguments are reversed when compared to other higher order > functions like map, filter and reduce. That's only "broken" if you think that we should be able to swap accumulate for map, filter and reduce. But that's not the case: accumulate is NOT broken because the API is not intended to be the same as map, filter and reduce. The API for accumulate is that the iterable is mandatory, but the function argument is *not*. > 2.2. It lacks a "start" parameter, requiring more complexity to include it > (nesting a itertools.chain or a "prepend" function call). Then just create your own wrapper: def accum(func, iterable, start=None): if start is not None: iterable = itertools.chain([start], iterable) return itertools.accumulate(iterable, func) > 3. It's not about "adding complexity", it's the other way around: No, I'm sorry, it does add complexity. > 3.1. The proposal is about explicit recursion in a list comprehension (a > way to access the previous output value/result, a.k.a. > accumulator/state/memory). List comprehensions are not intended for these sorts of complex calculations. Regular for-loops are easy to use and can be as general as you like. > I'm not sure what you meant with "turn a sequence into a series", I think Stephen is referring to the mathematical concept of sequences and series. A sequence is an ordered set of numbers obeying some rule, e.g.: [1, 2, 3, 4, 5] while a series is the partial sums found by adding each term to the previous sum: [1, 3, 6, 10, 15] -- Steve From ethan at stoneleaf.us Sun Nov 6 21:27:24 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 06 Nov 2016 18:27:24 -0800 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: <581FE68C.9050401@stoneleaf.us> On 11/06/2016 12:18 AM, Ram Rachum wrote: > Well, you think it's weird that I want a `finally` clause to not be called > in some circumstances. Yes I (we) do. > Do you think it's equally weird to want an `__exit__` method that is not > called in some circumstances? Yes I (we) do. -- ~Ethan~ From ethan at stoneleaf.us Sun Nov 6 21:25:56 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 06 Nov 2016 18:25:56 -0800 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: Message-ID: <581FE634.2010601@stoneleaf.us> On 11/06/2016 12:44 AM, Ram Rachum wrote: > I see that Python does allow you to not call `__exit__` if you don't want > to [...] Um, how? I was unaware of this (mis-)feature. -- ~Ethan~ From ncoghlan at gmail.com Sun Nov 6 22:51:20 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Nov 2016 13:51:20 +1000 Subject: [Python-ideas] PythonOS In-Reply-To: References: Message-ID: On 7 November 2016 at 08:25, victor rajewski wrote: > My proposal is a Linux distribution that has the simplicity of micropython, > but the power of full python. Drop a file (let's call it main.py) on an SD > card from your laptop, plug it into the device, and it boots and runs that > file straight away. Or drop an entire project. If the device has USB device > capabilities, then just drag the file across, or even edit it live. Easy SSH > setup could allow more advanced users remote development and debugging > options. Maybe jupyter could be leveraged to provide other network-based dev > options (pynq already has a working linux distro like this). Connecting to a > console would give you a python prompt. > > Ideally, key OS functions should be able to be controlled from within > python, so that the user never has to leave the python prompt. For a start, > things like network config, and possibly cron. > > What do you think? I'm not in a state to do all of this myself, but could > give it a start if there is interest in the concept. A potentially simpler option to explore would be a derivative of an existing beginner-friendly Linux distro like Raspbian that configures xon.sh ( http://xon.sh/ ) as the default system shell and IPython as the default Python REPL. That still keeps a distinction between the system shell, the interactive Python shell, and normal Python application programming, but I think those are actually good distinctions to preserve, as "query and control the currently running machine", "perform ad hoc interactive IO and data manipulation" and "perform IO manipulation and user interaction repeatably for the benefit of other users" are genuinely different tasks, even though they have common needs when it comes to basic control flow constructs. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mikhailwas at gmail.com Sun Nov 6 22:58:58 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 7 Nov 2016 04:58:58 +0100 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On 7 November 2016 at 02:32, Nathaniel Smith wrote: > On Sun, Nov 6, 2016 at 5:06 AM, Eric V. Smith wrote: >> Creating a new thread, instead of hijacking the PEP 532 discussion. >> >> From PEP 532: >> >>> Abstract >>> ======== >>> >>> Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this >>> PEP >>> proposes the addition of a new protocol-driven circuit breaking operator >>> to >>> Python that allows the left operand to decide whether or not the >>> expression >>> should short circuit and return a result immediately, or else continue >>> on with evaluation of the right operand:: >>> >>> exists(foo) else bar >>> missing(foo) else foo.bar() >> >> Instead of new syntax that only works in this one specific case, I'd prefer >> a more general solution. I accept being "more general" probably seals the >> deal in killing any proposal! >> >> I realize the following proposal has at least been hinted at before, but I >> couldn't find a specific discussion about it. Since it applies to the >> short-circuiting issues addressed by PEP 532 and its predecessors, I thought >> I'd bring it up here. It could also be used to solve some of the problems >> addressed by the rejected PEP 463 (Exception-catching expressions). See also >> PEP 312 (Simple Implicit Lambda). It might also be usable for some of the >> use cases presented in PEP 501 (General purpose string interpolation, aka >> i-strings). >> >> I'd rather see the ability to have unevaluated expressions, that can later >> be evaluated. I'll use backticks here to mean: "parse, but do not execute >> the enclosed code". This produces an object that can later be evaluated with >> a new builtin I'll call "evaluate_now". Obviously these are strawmen, and >> partly chosen to be ugly and unacceptable names and symbols in the form I'll >> discuss here. > > If we're considering options along these lines, then I think the local > optimum is actually a "quoted-call" operator, rather than a quote > operator. So something like (borrowing Rust's "!"): > > eval_else!(foo.bar, some_func()) > > being sugar for > > eval_else.__macrocall__(, > ) > > You can trivially use this to recover a classic quote operator if you > really want one: > > def quote!(arg): > return arg > > but IMO this way is more ergonomic for most use cases (similar to your > '&' suggestion), while retaining the call-site marking that "something > magical is happening here" (which is also important, both for > readability + implementation simplicity -- it lets the compiler know > statically when it needs to retain the AST, solving the issue that > Greg pointed out). > > Some other use cases: > > Log some complicated object, but only pay the cost of stringifying the > object if debugging is enabled: > > log.debug!(f"Message: {message_object!r}") > > Generate a plot where the axes are automatically labeled "x" and > "np.sin(x)" (this is one place where R's plotting APIs are more > convenient than Python's): > > import numpy as np > import matplotlib.pyplot as plt > x = np.linspace(0, 10) > plt.plot!(x, np.sin(x)) > > What PonyORM does, but without the thing where currently their > implementation involves decompiling bytecode...: > > db.select!(c for c in Customer if sum(c.orders.price) > 1000) > > Filtering out a subset of rows from a data frame in pandas; 'height' > and 'age' refer to columns in the data frame (equivalent to > data_frame[data_frame["height"] > 100 and data_frame["age"] < 5], but > more ergonomic and faster (!)): > > data_frame.subset!(height > 100 and age < 5) > > (IIRC pandas has at least experimented with various weird lambda hacks > for this kind of thing; not sure what the current status is.) > > Every six months or so I run into someone who's really excited about > the idea of adding macros to python, and I suggest this approach. So > far none of them have been excited enough to actually write a PEP, but > if I were going to write a PEP then this is the direction that I'd > take :-). > Oh great! Good to know I am not alone thinking in this direction. I have however one minor problem here: the problem is that "!" sign is almost invisible in the code, unless there is syntax highlighting which paints it in some very bright color. On the other hand I am not sure if it *must* be very visible... So to make it more distinctive in code I would propose something like: macros<>( x, y ) macros>( x, y ) macros::( x, y ) And those are already used operators, sad :( It would look so neat... But probably still possible to do? Mikhail From ncoghlan at gmail.com Sun Nov 6 23:11:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Nov 2016 14:11:22 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: <581FE634.2010601@stoneleaf.us> References: <581FE634.2010601@stoneleaf.us> Message-ID: On 7 November 2016 at 12:25, Ethan Furman wrote: > On 11/06/2016 12:44 AM, Ram Rachum wrote: > >> I see that Python does allow you to not call `__exit__` if you don't want >> to [...] > > Um, how? I was unaware of this (mis-)feature. It involves wrapping the context manager in another context manager that deliberately doesn't delegate the call to __exit__ in some cases (cf contextlib.ExitStack.pop_all()). By contrast, if __del__ is defined (as it is on generators), if you don't keep the context manager itself alive, you can only prevent the cleanup happening if you can define a subclass to use instead, and that's not always possible (deliberately so, in the case of generator cleanup). So the odd part of Ram's request isn't wanting to have conditional resource cleanup - the recipes in the contextlib docs gives some examples of where conditional local resource management is useful and how to achieve it using ExitStack. The odd part is wanting to make the resource cleanup implicitly unreliable, rather than having it be reliable by default and folks having to explicitly opt in to disabling it, since the easiest way to obtain non-deterministic resource management is to just avoid using the context management features in the first place. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun Nov 6 23:19:19 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 06 Nov 2016 20:19:19 -0800 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: <581FE634.2010601@stoneleaf.us> Message-ID: <582000C7.5060403@stoneleaf.us> On 11/06/2016 08:11 PM, Nick Coghlan wrote: > On 7 November 2016 at 12:25, Ethan Furman wrote: >> On 11/06/2016 12:44 AM, Ram Rachum wrote: >> >>> I see that Python does allow you to not call `__exit__` if you don't want >>> to [...] >> >> Um, how? I was unaware of this (mis-)feature. > > It involves wrapping the context manager in another context manager > that deliberately doesn't delegate the call to __exit__ in some cases > (cf contextlib.ExitStack.pop_all()). Ah, okay. Perhaps a custom iterator class would suit Ram's needs better than using a generator shortcut. -- ~Ethan~ From danilo.bellini at gmail.com Sun Nov 6 23:21:04 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Mon, 7 Nov 2016 02:21:04 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: <20161107015520.GX3365@ando.pearwood.info> References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <20161107015520.GX3365@ando.pearwood.info> Message-ID: 2016-11-06 23:55 GMT-02:00 Steven D'Aprano : > On Sun, Nov 06, 2016 at 04:46:42PM -0200, Danilo J. S. Bellini wrote: > > > 1.2. Sub-expressions in an expression might be on other statements (e.g. > > assignments, other functions). > > Not in Python it can't be. Assignment is not an expression, you cannot > say (x = 2) + 1. > O.o ... how is that (x = 2) + 1 related to what I wrote? Say you have "d = a + b + c", you can write it as "h = a + b" then "d = h + c"... The expression "a + b + c" and the new "h + c" are equivalent because the sub-expression "a + b" was assigned to "h" elsewhere (another statement). 2016-11-06 23:55 GMT-02:00 Steven D'Aprano : > > 2. The itertools.accumulate function signature is broken: > > 2.1. Its arguments are reversed when compared to other higher order > > functions like map, filter and reduce. > > That's only "broken" if you think that we should be able to swap > accumulate for map, filter and reduce. But that's not the case: > accumulate is NOT broken because the API is not intended to be the same > as map, filter and reduce. The API for accumulate is that the iterable > is mandatory, but the function argument is *not*. > I'd say not even on map and filter requires a function, as "None" means "lambda x: x" on them, strangely enough. All of them are higher order functions in the standard Python, and you can include other functions like itertools.takewhile and itertools.dropwhile to this comparison. The accumulate signature is simply broken/different/surprising in that context. 2016-11-06 23:55 GMT-02:00 Steven D'Aprano : > > 3. It's not about "adding complexity", it's the other way around: > > No, I'm sorry, it does add complexity. > Compared to the alternatives, it's the other way around: my proposal removes complexity from the resulting code that requires a scan. Unless you're saying so because it's a proposal to change CPython, and as CPython itself would get a new feature, it would "be more complex"... I agree: any change would either add complexity somewhere or be backwards incompatible. Which one is better? Why this mail list exists? 2016-11-06 23:55 GMT-02:00 Steven D'Aprano : > List comprehensions are not intended for these sorts of complex > calculations. > I know list comprehensions aren't yet intended for scan, otherwise why would I propose this? Triple-for-sections list comprehensions are complicated, and it's actually surprising that Python allows using the same target variable name twice on them. But the calculation itself and its declarative description aren't complex. Even you wrote such a declarative description in your e-mail, when explaining what "series" are. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony at xtfx.me Mon Nov 7 00:08:22 2016 From: anthony at xtfx.me (C Anthony Risinger) Date: Sun, 6 Nov 2016 23:08:22 -0600 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On Nov 6, 2016 7:32 PM, "Nathaniel Smith" wrote: > > [...] > > Some other use cases: > > Log some complicated object, but only pay the cost of stringifying the > object if debugging is enabled: > > log.debug!(f"Message: {message_object!r}") Would the log.debug implementation need to fetch the context to evaluate the delayed expression (say by using sys._getframe) or would that be bound? Is a frame necessary or just a (bound?) symbol table? Could a substitute be provided by on evaluation? Curious how this looks to the callee and what is possible. Also what is the meaning (if desirable) of something like: def debug!(...): pass Persistent delayed calls? Delayed default arguments? Something else? Not valid? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Mon Nov 7 00:12:52 2016 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Sun, 6 Nov 2016 21:12:52 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On Sun, Nov 6, 2016 at 5:32 PM, Nathaniel Smith wrote: > > If we're considering options along these lines, then I think the local > optimum is actually a "quoted-call" operator, rather than a quote > operator. So something like (borrowing Rust's "!"): > > eval_else!(foo.bar, some_func()) > > being sugar for > > eval_else.__macrocall__(, > ) > > You can trivially use this to recover a classic quote operator if you > really want one: > > def quote!(arg): > return arg > Xonsh does it: http://xon.sh/tutorial_macros.html At least for "function" call, and make use of Python type annotations to decide whether to expand the expression or not, or passing, string, Ast, to the defined macro. I haven't tried it in a while but there were some ideas floating around for context-manager as well, to get the block they wrap. -- M From greg.ewing at canterbury.ac.nz Mon Nov 7 00:13:29 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 07 Nov 2016 18:13:29 +1300 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: <58200D79.9030400@canterbury.ac.nz> C Anthony Risinger wrote: > On Nov 6, 2016 7:32 PM, "Nathaniel Smith" > wrote: > > > > log.debug!(f"Message: {message_object!r}") > > Would the log.debug implementation need to fetch the context to evaluate > the delayed expression Not if it expands to log.debug(lambda: f"Message: {message_object!r}") > Also what is the meaning (if desirable) of something like: > > def debug!(...): pass Nothing like that would be needed. The implementation of debug() would just be an ordinary function receiving callable objects as parameters. -- Greg From steve at pearwood.info Mon Nov 7 00:31:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 Nov 2016 16:31:29 +1100 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <20161107015520.GX3365@ando.pearwood.info> Message-ID: <20161107053128.GA19000@ando.pearwood.info> On Mon, Nov 07, 2016 at 02:21:04AM -0200, Danilo J. S. Bellini wrote: > 2016-11-06 23:55 GMT-02:00 Steven D'Aprano : > > > On Sun, Nov 06, 2016 at 04:46:42PM -0200, Danilo J. S. Bellini wrote: > > > > > 1.2. Sub-expressions in an expression might be on other statements (e.g. > > > assignments, other functions). > > > > Not in Python it can't be. Assignment is not an expression, you cannot > > say (x = 2) + 1. > > > > O.o ... how is that (x = 2) + 1 related to what I wrote? I misread your comment as "Sub-expressions in an expression might be other statements". Sorry for the misunderstanding. -- Steve From steve at pearwood.info Mon Nov 7 00:46:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 Nov 2016 16:46:22 +1100 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: <20161107054621.GB19000@ando.pearwood.info> On Sun, Nov 06, 2016 at 09:31:06AM -0500, Eric V. Smith wrote: > The point remains: do we want to be able to create unevaluated > expressions that can be evaluated at a different point? I sometimes think that such unevaluated expressions (which I usually call "thunks") would be cool to have. But in more realistic moments I think that they're a solution looking for a problem to solve. If your PEP suggests a problem that they will solve, I'm interested. But note that we already have two ways of generating thunk-like objects: functions and compiled byte-code. thunk = lambda: a + b - c thunk() thunk = compile('a + b - c', '', 'single') eval(thunk) Both are very heavyweight: the overhead of function call syntax is significant, and the keyword "lambda" is a lot of typing just to delay evaluation of an expression. compile(...) is even worse. -- Steve From brenbarn at brenbarn.net Mon Nov 7 01:13:18 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 06 Nov 2016 22:13:18 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: <20161107054621.GB19000@ando.pearwood.info> References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> <20161107054621.GB19000@ando.pearwood.info> Message-ID: <58201B7E.8050100@brenbarn.net> On 2016-11-06 21:46, Steven D'Aprano wrote: > I sometimes think that such unevaluated expressions (which I usually > call "thunks") would be cool to have. But in more realistic moments I > think that they're a solution looking for a problem to solve. > > If your PEP suggests a problem that they will solve, I'm interested. > > But note that we already have two ways of generating thunk-like objects: > functions and compiled byte-code. > > thunk = lambda: a + b - c > thunk() > > thunk = compile('a + b - c', '', 'single') > eval(thunk) > > Both are very heavyweight: the overhead of function call syntax is > significant, and the keyword "lambda" is a lot of typing just to delay > evaluation of an expression. compile(...) is even worse. I sometimes want these too. But note that both the solutions you propose are quite a ways from a true "unevaluated expression". The big problem with an ordinary lambda (or def) is that you cannot explicitly control where it will decide to look for its free variables. If it uses a local variable from an enclosing namespace, it will always look for it in that namespace, so you can't "patch in" a value by setting a global variable. If it uses a variable that isn't local to any enclosing namespace, it will always look for it in the global namespace, so you can't patch in a value by setting a local variable in the context where you're calling the function. You can get around this in a def by, for instance, using global to mark all variables global, and then using eval to pass in a custom global namespace. But that is a lot of boilerplate. What I want (when I want this) is a way to create a function that will allow the injection of values for *any* variables, regardless of whether the function originally thought they were local, nonlocal (i.e., local to some enclosing scope) or global. The way it is now, the status of a function's variables is inextricably linked to the syntactic context where it was defined. This is a good thing most of the time, but it's not what you want if you want to define an expression that should later be evaluated in some other context. I consider the compile-based solution a nonstarter, because it puts the code in a string. With the code in a string, you are blocked from using syntax highlighting or any other handy editor features. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From njs at pobox.com Mon Nov 7 01:19:19 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 Nov 2016 22:19:19 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On Sun, Nov 6, 2016 at 9:08 PM, C Anthony Risinger wrote: > On Nov 6, 2016 7:32 PM, "Nathaniel Smith" wrote: >> >> [...] >> >> Some other use cases: >> >> Log some complicated object, but only pay the cost of stringifying the >> object if debugging is enabled: >> >> log.debug!(f"Message: {message_object!r}") > > Would the log.debug implementation need to fetch the context to evaluate the > delayed expression (say by using sys._getframe) or would that be bound? Is a > frame necessary or just a (bound?) symbol table? Could a substitute be > provided by on evaluation? There are a lot of ways one could go about it -- I'll leave the details to whoever decides to actually write the PEP :-) -- but one sufficient solution would be to just pass AST objects. Those are convenient (the compiler has just parsed the code anyway), they allow the code to be read or modified before use (in case you want to inject variables, or convert to SQL as in the PonyORM case, etc.), and if you want to evaluate the thunks then you can look up the appropriate environment using sys._getframe and friends. Maybe one can do even better, but simple ASTs are a reasonable starting point. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Mon Nov 7 03:23:31 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Nov 2016 18:23:31 +1000 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On 6 November 2016 at 23:06, Eric V. Smith wrote: > Creating a new thread, instead of hijacking the PEP 532 discussion. > > From PEP 532: > >> Abstract >> ======== >> >> Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this >> PEP >> proposes the addition of a new protocol-driven circuit breaking operator >> to >> Python that allows the left operand to decide whether or not the >> expression >> should short circuit and return a result immediately, or else continue >> on with evaluation of the right operand:: >> >> exists(foo) else bar >> missing(foo) else foo.bar() > > Instead of new syntax that only works in this one specific case, I'd prefer > a more general solution. I accept being "more general" probably seals the > deal in killing any proposal! Being more general isn't the goal here: just as with decorators, context managers, and coroutines, the goal is pattern extraction based on idioms that people *already use* (both in Python and in other languages) to make them shorter (and hence easier to recognise), as well as to give them names (and hence make them easier to learn, teach and find on the internet). Consider the conditional expression usage examples in the PEP: value1 = expr1.field.of.interest if expr1 is not None else None value2 = expr2["field"]["of"]["interest"] if expr2 is not None else None value3 = expr3 if expr3 is not None else expr4 if expr4 is not None else expr5 Which would become the following in the basic form of the PEP: value1 = missing(expr1) else expr1.field.of.interest value2 = missing(expr2) else expr2.["field"]["of"]["interest"] value3 = exists(expr3) else exists(expr4) else expr5 or else the following in the more comprehensive variant that also allows short-circuiting "if" expressions (and would hence mainly include "missing()" as the answer to "What does 'not exists()' return?"): value1 = expr1.field.of.interest if exists(expr1) value2 = expr2.["field"]["of"]["interest"] if exists(expr2) value3 = exists(expr3) else exists(expr4) else expr5 Now, how would those look with a lambda based solution instead? value1 = eval_if_exists(expr, lambda: expr1.field.of.interest) value2 = eval_if_exists(expr, lambda: expr2["field"]["of"]["interest"]) value3 = first_existing(expr3, lambda: expr4, lambda: expr5) This is not an idiom people typically use, and that's not solely due to them finding the "lambda:" keyword unattractive or function calls being much slower than inline expressions. We can swap in a Java inspired "()->" operator to help illustrate that (where "()" indicates an empty parameter list, and the "->" indicates the LHS is a parameter list and the RHS is a lambda expression): value1 = eval_if_exists(expr, ()->expr1.field.of.interest) value2 = eval_if_exists(expr, ()->expr2["field"]["of"]["interest"]) value3 = first_existing(expr3, ()->expr4, ()->expr5) Without even getting into the subtle differences between inline evaluation and nested scopes in Python, the fundamental challenge with this approach is that it takes something that was previously about imperative conditional control flow and turns it into a use case for callback based programming, which requires an understanding of custom first class functions at the point of *use*, which is conceptually a much more difficult concept than conditional evaluation. By contrast, while as with any other protocol you would need to understand object-oriented programming in order to define your own circuit breakers, *using* circuit breakers would only require knowledge of imperative condition control flow, as well as how the specific circuit breaker being used affects the "if" and "else" branches. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 7 03:26:49 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Nov 2016 18:26:49 +1000 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On 7 November 2016 at 16:19, Nathaniel Smith wrote: > On Sun, Nov 6, 2016 at 9:08 PM, C Anthony Risinger wrote: >> On Nov 6, 2016 7:32 PM, "Nathaniel Smith" wrote: >>> >>> [...] >>> >>> Some other use cases: >>> >>> Log some complicated object, but only pay the cost of stringifying the >>> object if debugging is enabled: >>> >>> log.debug!(f"Message: {message_object!r}") >> >> Would the log.debug implementation need to fetch the context to evaluate the >> delayed expression (say by using sys._getframe) or would that be bound? Is a >> frame necessary or just a (bound?) symbol table? Could a substitute be >> provided by on evaluation? > > There are a lot of ways one could go about it -- I'll leave the > details to whoever decides to actually write the PEP :-) -- but one > sufficient solution would be to just pass AST objects. Those are > convenient (the compiler has just parsed the code anyway), they allow > the code to be read or modified before use (in case you want to inject > variables, or convert to SQL as in the PonyORM case, etc.), and if you > want to evaluate the thunks then you can look up the appropriate > environment using sys._getframe and friends. Maybe one can do even > better, but simple ASTs are a reasonable starting point. PEP 501 suggested taking the machinery that was added to support f-string interpolation and making it possible to separate the expression evaluation and string rendering steps: https://www.python.org/dev/peps/pep-0501/ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rikudou__sennin at live.com Sun Nov 6 12:49:26 2016 From: rikudou__sennin at live.com (adil gourinda) Date: Sun, 6 Nov 2016 17:49:26 +0000 Subject: [Python-ideas] Python Documentation Message-ID: Hi I am an amateur of computer programming and my ambition push me to choose a good programming language to begin with and I chose "Python" because it is an easy language and the documentation is free and complete. For my learning I was based on your "Python Documentation" and one "Python Shell" to exercise what i learned. I want to send you my notes ( and the way by which are noted ) and I wish you to use them to enhance the documentation. Because I am an amateur I don't have a strong base in programming so what is noted I found it easy to learn (I think everyone can do the same ) and what is not noted I had difficulty in learning it ( with ideas from other peoples we can make it easy to learn ) Thank you for accepting my offer and I wish I was helpful -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Python Tutorial.okular Type: application/vnd.kde.okular-archive Size: 601651 bytes Desc: Python Tutorial.okular URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: The Python Language Reference.okular Type: application/vnd.kde.okular-archive Size: 645281 bytes Desc: The Python Language Reference.okular URL: From steve at pearwood.info Mon Nov 7 08:55:10 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Nov 2016 00:55:10 +1100 Subject: [Python-ideas] Python Documentation In-Reply-To: References: Message-ID: <20161107135509.GZ3365@ando.pearwood.info> Hello Adil, and welcome! But... you've converted the ENTIRE Python tutorial AND documentation into PDF files, stuck them into zip files with a non-standard file extension (.okular) that most people won't be able to deal with, and expect us to read the whole thing -- over 200 pages -- looking for your notes? I applaud your enthusiasm, that is wonderful, but this is not a very practical way to communicate. Can't you just tell us your ideas, in an email, rather than expect us to unzip two mystery .okular files and read through 200+ pages of PDF files? Thank you for wanting to help. -- Steve From toddrjen at gmail.com Mon Nov 7 09:56:22 2016 From: toddrjen at gmail.com (Todd) Date: Mon, 7 Nov 2016 09:56:22 -0500 Subject: [Python-ideas] Python Documentation In-Reply-To: References: Message-ID: On Nov 7, 2016 7:33 AM, "adil gourinda" wrote: > Hi > > > I am an amateur of computer programming and my ambition push me to choose > a good programming language to begin with and I chose "Python" because it > is an easy language and the documentation is free and complete. > > > For my learning I was based on your "Python Documentation" and one "Python > Shell" to exercise what i learned. > > > I want to send you my notes ( and the way by which are noted ) and I wish > you to use them to enhance the documentation. Because I am an amateur I > don't have a strong base in programming so what is noted I found it easy to > learn (I think everyone can do the same ) and what is not noted I had > difficulty in learning it ( with ideas from other peoples we can make it > easy to learn ) > > > Thank you for accepting my offer and I wish I was helpful > > Okular is a great document viewer, but also not a very common one. Most people don't have access to it. So if you want people to be able to read your notes you are going to need a more universal approach. Everyone reading this, by definition, has email, so that is probably the best way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Nov 7 12:34:08 2016 From: brett at python.org (Brett Cannon) Date: Mon, 07 Nov 2016 17:34:08 +0000 Subject: [Python-ideas] Civility on this mailing list In-Reply-To: References: Message-ID: I wanted to give people an update on civility on this mailing list. First, one person has received a Code of Conduct violation warning and another has received a general warning that their conduct has worn out any leniency about future conduct that we generally try to give people (think of it as a pre-warning warning). On this list you get one warning for your first CoC violation, and on the second you receive a one year banishment. The admins do take into consideration people's behaviour after a warning and how long it has been since the warning so a single warning won't hover over you until the end of time (i.e. there's a decay factor on warnings, but it's at the discretion of the admins). IOW I want everyone to realize we do take how people act on this list seriously (and for people who don't know or remember, we do already have one permanent banishment on this list). Two, taking something off-list does not get you out from underneath the CoC. If you decide to only reply to someone who posts to this list while leaving the list out of it you are still expected to act as if you posted publicly. We don't care if the discussion was private because you chose to stem your bad behaviour based on something you read on this list. If you can't abide by the CoC when dealing with content on this list then you will simply lose your privileges to be on this list and access its content directly. Three, we have seen at least one instance where someone claimed that what they said was not their fault because they were quoting someone. That's patently false and anything you say is your responsibility. If you don't quote someone then the list would never had known, and thus you reiterating what someone said for this list makes you responsible for it. In philosophy it's called *intentionality:* since you intended for that information to reach the list then you are held responsible for it no matter who originally said it (we are not journalists so there's no real need to quote someone just to report it). If you really need to report what someone said that is going to be hurtful you can summarize it and leave the hurtful part out. Otherwise I just wanted to thank the people who came forward after this initial email saying that they have found the mailing list in general pleasant to be on (notwithstanding the incidents leading to the warnings). I also want to thank the people who did step forward to file their complaints as I know that's a very difficult thing to do. On Mon, 17 Oct 2016 at 11:29 Brett Cannon wrote: > > Based on some emails I read in the " unpacking generalisations for list > comprehension", I feel like I need to address this entire list about its > general behaviour. > > If you don't follow me on Twitter you may not be aware that I am taking > the entire month of October off from volunteering any personal time on > Python for my personal well-being (this reply is being done on work time > for instance). This stems from my wife pointing out that I had been rather > stressed in July and August outside of work in relation to my Python > volunteering (having your weekends ruined is never fun). That stress > stemmed primarily from two rather bad interactions I had to contend with on > the issue track in July and August ... and this mailing list. > > When I have talked to people about this mailing list it's often referred > to by others as the "wild west" of Python development discussions (if > you're not familiar with US culture, that turn of phrase basically means > "anything goes"). To me that is not a compliment. When I created this list > with Titus the goal was to provide a safe place where people could bring up > ideas for Python where people could quickly provide basic feedback so > people could know whether there was any chance that python-dev would > consider the proposal. This was meant to be a win for proposers by not > feeling like they were wasting python-dev's time and a win for python-dev > by keeping that list focused on the development of Python and not fielding > every idea that people want to propose. > > And while this list has definitely helped with the cognitive load on > python-dev, it has not always provided a safe place for people to express > ideas. I have seen people completely dismiss people's expertise and > opinion. There has been name calling and yelling at people (which is always > unnecessary). There have been threads that have completely derailed itself > and gone entirely off-topic. IOW I would not hold this mailing list up as > an example of the general discourse that I experience elsewhere within the > community. > > Now I realize that we are all human beings coming from different cultural > backgrounds and lives. We all have bad days and may not take the time to > stop and think about what we are typing before sending it, leading to > emails that are worded in a way that can be hurtful to others. It's also > easy to forget that various cultures views things differently and so that > can lead to people "reading between the lines" a lot and picking up things > that were never intended. There are 1,031 people on this mailing list from > around the world and it's easy to forget that e.g. Canadian humour may not > translate well to Ukrainian culture (or something). What this means is it's > okay to *nicely* say that something bothered you, but also try to give > people the benefit of the doubt as you don't know what their day had been > like before they wrote that email (I personally don't like the "just mute > the thread" approach to dealing with bad actors when the muting is silent > as that doesn't help new people who join this mailing list and the first > email they see is someone being rude that everyone else didn't see because > they muted the thread days ago). > > As for the off-topic threads, please remember there are 1,031 people on > this mailing list (this doesn't count people reading through gmane or > Google Groups). Being extremely generous and assuming every person on this > list only spends 10 seconds deciding if they care about your email, that's > still nearly 3 hours of cumulative time spent on your email. So please be > cognisant when you reply, and if you want to have an off-topic > conversation, please take it off-list. > > And finally, as one of the list administrators I am in a position of power > when it comes to the rules of this list and the CoC. While I'm one of the > judges on when someone has violated the CoC, I purposefully try not to play > the role of police to avoid bias and abuse of power. What that means is > that I never personally lodge a CoC complaint against anyone. That means > that if you feel someone is being abusive here you cannot rely on list > admins noticing and doing something about it. If you feel someone has > continuously been abusive on this list and violating the CoC then you must > email the list admins about it if you wish to see action taken (all > communications are kept private among the admins). Now I'm not asking > people to email us on every small infraction (as I said above, try to give > everyone a break knowing we all have bad days), but if you notice a pattern > then you need to speak up if you would like to see something change. > > When I started my month off I thought that maybe if I only read this > mailing list once a week that the frequency would be low enough that I > could handle the stress of being both a participant and admin who is > ultimately responsible for the behaviour here, but I'm afraid that isn't > going to cut it. What I don't think people realize is that I don't take my > responsibility as admin lightly; any time anyone acts rudely I take it > personally like I somehow failed by letting the atmosphere and discourse on > this list become what it is. Because of this I'm afraid I need to mute this > mailing list for the rest of my vacation from volunteering in the Python > community after I send this email. I personally hope people do take the > time to read this email and reflect upon how they conduct themselves on > this mailing list -- and maybe on other lists as well -- so that when I > attempt to come back in November I don't have to permanent stop being a > participant on this list and simply become an admin for this list to > prevent complete burn-out for me in the Python community (and I know this > last sentence sounds dramatic, but I'm being serious; the irony of > receiving the Frank Willison award the same year I'm having to contemplate > fundamentally shifting how I engage with the community to not burn out is > not lost on me). > > -Brett > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathanfdunn at gmail.com Mon Nov 7 22:22:14 2016 From: nathanfdunn at gmail.com (Nathan Dunn) Date: Mon, 7 Nov 2016 22:22:14 -0500 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) Message-ID: > * __add__ is only part of the addition protocol, there is also > __radd__ and __iadd__ __iadd__ could be represented as def self += value:. Reflected binary operators do pose a problem. A possible solution would be to give self special meaning in this context, so def self + other: would correspond to __add__ and def other + self: to __radd__. But of course, this would be a more fundamental change to the language than just syntactic sugar. > * likewise, there is not a one-to-one correspondence between the > bool() builtin and the __bool__() special method (there are other ways > to support bool(), like defining __len__() on a container) That is a fair point, but I think this would be only a slight point of confusion. I believe for the most part other builtins such as __abs__, __len__, etc. do have a one-to-one correspondence so this syntax would make sense in almost all situations. But while were on the topic, I will take this opportunity to note that all the methods to support builtins suffer from the same ambiguity I alluded to earlier with bool(self). One solution could be to use a @builtin decorator or something to that effect, but that would more or less defeat the purpose of cleaner, more intuitive syntax. It may be that there is no viable alternative for the builtin function dunder methods. > * the mapping protocol covers more than just __getitem__ __setitem__(self, key, value) could be def self[key] = value likewise __delitem__(self, key) def del self[key]: __iter__(self) iter(self) __len__(self) len(self) > (and you also > need to decide if you're implementing a mapping, sequence, or > multi-dimensional array) Could you speak more about this? It doesn't seem to me that this would affect the method signatures since it is up to the implementer to interpret the semantic meaning of the keys. > it would break the current symmetry between between name binding > in def statements and target binding in assignment statements > With the proposed change, we'd face the > problem that the following would both be legal, but meant very > different things: > > cls.mymethod = lambda self: print(self) > def cls.mymethod(self): print(self) > > The former is already legal and assigns the given lambda function as a > method on the existing class, `cls` > > The latter currently throws SyntaxError. > > With the proposed change, rather than throwing SyntaxError as it does > now, the latter would instead be equivalent to: > > def mymethod(cls, self): print(self) > > which would be a very surprising difference in behaviour. I will admit that there is potential for confusion here as this class Formatter(object): def Formatter.displayDefault(obj): print('[{}]'.format(obj)) is rather suggestive of a static method, but I don't think the confusion arises because the syntax breaks the symmetry you reference. The self-dot syntax *is* meant to elicit target binding in assignment statements. Just as self.mymethod = lambda p1, p2: p1+p2 indicates that an instance of the class has a callable with two parameters bound to its mymethod attribute, so too does the statement def self.mymethod(p1, p2):. So I don't think this inherently breaks any symmetry. It is specifically in the case that the identifier is the same as the class name that this leads to surprising behavior, but there is already a similar risk of confusion any time the name of a class is used as an identifier within the class body. Perhaps my proposed syntax does increase this risk since the identifier doesn't look like a parameter or a variable, but hopefully the self form of this syntax would be so commonplace that seeing anything else before the dot would throw up a red flag. Thanks, -Nate -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Nov 7 23:49:09 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 8 Nov 2016 13:49:09 +0900 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: References: Message-ID: <22561.22853.957495.508409@turnbull.sk.tsukuba.ac.jp> Nathan Dunn writes: > > * the mapping protocol covers more than just __getitem__ > > __setitem__(self, key, value) could be def self[key] = value > likewise > __delitem__(self, key) def del self[key]: > __iter__(self) iter(self) > __len__(self) len(self) The template used by last two (BTW, shouldn't they be "def iter(self)" and "def len(self)"?) seems like a really bad idea to me. This would imply that any class that defines such methods in some arbitrary fashion will appear to participate in the corresponding protocol although it doesn't. That seems likely to result in bugs, that might be hard to diagnose (haven't thought about how hard that might be, though). It would require the programmer to know about all such protocols when designing a public API, while with the dunder convention, you only need to find out whether the dunders you want to use for your new protocol are already in use somewhere, and programmers who aren't designing protocols don't need to know about them at all. It would also require a modification to the compiler for every new protocol. I personally don't see that the other syntaxes are helpful, either, but that's a matter of opinion and I'd be interested to hear about experience with teaching languages that implement such "method definitions look like invocation syntax". Regards, From steve at pearwood.info Tue Nov 8 11:15:11 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 Nov 2016 03:15:11 +1100 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: <20161108161509.GC3365@ando.pearwood.info> On Sat, Nov 05, 2016 at 07:50:44PM +1000, Nick Coghlan wrote: > Hi folks, > > As promised, here's a follow-up to the withdrawn PEP 531 that focuses > on coming up with a common protocol driven solution to conditional > evaluation of subexpressions that also addresses the element-wise > comparison chaining problem that Guido noted when rejecting PEP 335. This feels much more promising to me, but I'm still not quite convinced, possibly because it's a very complicated proposal. I've had to read it multiple times to really understand it. I wonder whether part of the difficulty is the size of the proposal. Perhaps this should be split into two PEPs: one to describe the protocol alone, and a second to propose built-ins and new syntax that takes advantage of the protocol. That might help keep this proposal in digestible pieces. (By the way, I really like the "circuit breaker" name.) > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP > proposes the addition of a new protocol-driven circuit breaking operator to > Python that allows the left operand to decide whether or not the expression > should short circuit and return a result immediately, or else continue > on with evaluation of the right operand:: > > exists(foo) else bar > missing(foo) else foo.bar() > > These two expressions can be read as: > > * "the expression result is 'foo' if it exists, otherwise it is 'bar'" > * "the expression result is 'foo' if it is missing, otherwise it is 'foo.bar()'" These built-in names are quite problematic. I've seen people on Reddit and elsewhere who understood this as a form of "is this variable defined?" which is badly wrong. That is, they understood: exists(foo) else bar as being equivalent to: try: return foo except NameError: # variable foo doesn't exist/isn't defined return bar So I think that is going to be an area of confusion. I'm not really keen on these names for the proposed built-ins. The names are much too generic for what they actually do, which is test for None, not just some generic idea of "existence". I don't really have a better solution if your aim is to dispense with the ?? operator. Some (bad) ideas: isnone, is_none, nullity. None of these are as compact as the ?? operator. On the other hand, if we keep the ?? operator, and implicitly define it as: lhs_expression ?? rhs_expression => exists(lhs_expression) else rhs_expression then there's no need for the user to explicitly write out exists() in their code, and the name can be as precise and as long as we need: # ?? operator types.NoneCoalesingType(lhs_expression) else rhs_expression > Execution of these expressions relies on a new circuit breaking protocol that > implicitly avoids repeated evaluation of the left operand while letting > that operand fully control the result of the expression, regardless of whether > it skips evaluation of the right operand or not:: > > _lhs = LHS > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) My first reaction on reading that was to wonder if you had written the first two terms backwards. My reading was that *any* truthy value would trigger the __then__ method call, and any falsey value the __else__ call, which would make this (almost) just another spelling of the existing `or` operator. My thought was that people could write: None else 2 which would return NoneType.__else__(None, 2), which I presumed would be 2. But I now think that's wrong. I thought that the intention was to give `object` default __then__ and __else__ methods, and over-ride them as needed, but I now think that's wrong. I think that should be clarified early in the PEP, as I wasted a lot of time thinking about this wrongly. So I now think that most objects will not define __then__ and __else__, and consequently code like `None else 2` will raise an error. Is that right? I now think that only the builtins `exists` and `missing`, and the CircuitBreaker actually need __then__ and __else__ methods. (Plus of course any user-defined types.) [...] > In addition to being usable as simple boolean operators (e.g. as in > ``assert all(exists, items)`` or ``if any(missing, items):``), these circuit > breakers will allow existence checking fallback operations (aka None-coalescing > operations) to be written as:: > > value = exists(expr1) else exists(expr2) else expr3 I take it that your intention is to avoid the ?? None-coalescing operator. Your exists() version risks being misunderstood as testing for the existence of the expression/name (e.g. catching NameError, LookupError), and it's still quite verbose compared to the ?? operator but without being any more explicit in what it is doing. # your exists() proposal value = exists(expr1) else exists(expr2) else expr3 # ?? operator value = expr1 ?? expr2 ?? expr3 [...] > Overloading logical inversion (``not``) > --------------------------------------- > > Any circuit breaker definition will have a logical inverse that is still a > circuit breaker, but inverts the answer as to whether or not to short circuit > the expression evaluation. For example, the ``exists`` and ``missing`` circuit > breakers proposed in this PEP are each other's logical inverse. > > A new protocol method, ``__not__(self)``, will be introduced to permit circuit > breakers and other types to override ``not`` expressions to return their > logical inverse rather than a coerced boolean result. So how will this effect the existing semantics of `not`? Currently, I think `not obj` is equivalent to `not bool(obj)`, where bool calls the __bool__ method, or __len__ if that's not defined, and falls back to True. I think your proposal means that this will change to something close to this: __not__ = getattr(type(obj), '__not__', None) if __not__ is not None: x = __not__(obj) if x is not NotImplemented: return x flag = bool(obj) return False if flag else True That means that now any object, not just Circuit Breakers, can customise how they respond to `not`. [...] > This PEP has been designed specifically to address the risks and concerns > raised when discussing PEPs 335, 505 and 531. > > * it defines a new operator and adjusts the definition of chained comparison > rather than impacting the existing ``and`` and ``or`` operators That's a point in its favour. > * the changes to the ``not`` unary operator are defined in such a way that > control flow optimisations based on the existing semantics remain valid I think the changes to `not` are neutral. > * rather than the cryptic ``??``, it uses ``else`` as the operator keyword in > exactly the same sense as it is already used in conditional expressions I don't think that ?? is really cryptic. Like any symbol, it has to be learned, but ?? is used in a number of other major languages. Its also obviously related to the ?. and ?[] sugar, so once people learn that they are related to None testing they will have a strong clue that ?? is too. I completely accept that it is not self-explanatory. That's the cost of symbols: they have to be learned, because they aren't self-explanatory in the way that well-named functions may be. But many things need to be learned, including such fundamental necessary (pseudo-)operators as . for attribute lookup and [] for item lookup, as well as maths operators like ** etc. To fairly label a symbol as "cryptic", I would expect that their effects are mysterious, hard to understand or complicated. "await" and "async" are cryptic because to understand them, you have to grok asyncronous programming, and that's hard. Nevertheless, they're important enough that they deserve to be syntax, hard to understand or not. But ?? is not hard to understand. It has a very simple explanation: expr1 ?? expr2 is just equivalent to _tmp = expr1 _tmp if _tmp is not None else expr2 except it doesn't create a temporary variable. > * it defines a general purpose short-circuiting binary operator that can even > be used to express the existing semantics of ``and`` and ``or`` rather than > focusing solely and inflexibly on existence checking Another point in its favour. > * it names the proposed builtins in such a way that they provide a strong > mnemonic hint as to when the expression containing them will short-circuit > and skip evaluating the right operand This is my biggest problem with your proposal. I just don't think this part is correct. I think that the exists() and missing() builtins are the weakest part of the proposal: - exists() can be easily misunderstood as testing for NameError or LookupError, rather than whether the value is None; - likewise for missing(), in reverse. The problem here is that exists() *seems* to be so self-evident and obvious that there's no need to read the documentation in any detail. It's actually misleading -- while it is true that `is None` is a special case of existence checking, for applications that treat None as a missing value, that's not what most people expect "existence" to mean, and I'm not convinced that those coming from languages with a ?? operator think of it as existence checking either. That's why they call it Null Coalesing rather than Existence Checking. Whereas ?? is just unfamiliar enough to keep the user on their toes and discourage them from making assumptions. If they know the ?? operator from another language, there won't be any big surprises for them. If they don't, they should find it documented as both the actual implementation in terms of the `else` circuit breaker (for experts!) and as a conceptually simpler form in terms of if...else. * * * Whew! Nick, this is a big, complex PEP, and thank you for taking the time to write it. But its also hard to understand -- there's a lot of detail, and there are places where it is easy for the reader to get mislead by their assumptions and only get corrected deep, deep into the PEP. At least that's my experience. I think I'd find this PEP easier to understand if it were split into two, like Guido's type-hints PEPs: one to explain the protocol, and one to detail the new built-ins and syntactic sugar which rely on the protocol. Or maybe that's just me. I really like this PEP as a way of introducing configurable short- circuiting behaviour, without messing with `and` and `or`. That's really, really nice. I like your decision to keep the ?. and ?[] sugar, as short-cuts for code based on this Circuit Breaker protocol. But I cannot support the exists() and missing() builtins as they stand. I think the circuit breakers themselves work well as a concept, but: - I think that the names will be more harmful than helpful; - I don't think that having to explicitly call a circuit breaker is a good substitute for the ?? operator. If I absolutely had to choose between this and nothing, I'd say +1 for this. But if I had to choose between ?? as a operator, and a generic circuit breaking protocol with no operator but exists() builtin instead, well, that would be a really hard decision. -- Steve From rainventions at gmail.com Tue Nov 8 11:55:28 2016 From: rainventions at gmail.com (Ryan Birmingham) Date: Tue, 8 Nov 2016 11:55:28 -0500 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: <22561.22853.957495.508409@turnbull.sk.tsukuba.ac.jp> References: <22561.22853.957495.508409@turnbull.sk.tsukuba.ac.jp> Message-ID: I think that the most important simple thing with dunder/magic methods is name mangling, and that an abstraction like your proposed one makes it less clear what will happen to a name. Also, as Nick Coghlan pointed out, there are a lot of dunder/magic methods . Either each of those methods would need an alternate representation, or some would be left out - which would lead to inconsistent style. Maybe I'm resisting change too much, but I think that the solution to this issue is to make the magic method documentation easier to teach, as opposed to changing them. -Ryan Birmingham On 7 November 2016 at 23:49, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Nathan Dunn writes: > > > > * the mapping protocol covers more than just __getitem__ > > > > __setitem__(self, key, value) could be def self[key] = value > > likewise > > __delitem__(self, key) def del self[key]: > > __iter__(self) iter(self) > > __len__(self) len(self) > > The template used by last two (BTW, shouldn't they be "def iter(self)" > and "def len(self)"?) seems like a really bad idea to me. This would > imply that any class that defines such methods in some arbitrary > fashion will appear to participate in the corresponding protocol > although it doesn't. That seems likely to result in bugs, that might > be hard to diagnose (haven't thought about how hard that might be, > though). It would require the programmer to know about all such > protocols when designing a public API, while with the dunder > convention, you only need to find out whether the dunders you want to > use for your new protocol are already in use somewhere, and > programmers who aren't designing protocols don't need to know about > them at all. It would also require a modification to the compiler for > every new protocol. > > I personally don't see that the other syntaxes are helpful, either, > but that's a matter of opinion and I'd be interested to hear about > experience with teaching languages that implement such "method > definitions look like invocation syntax". > > Regards, > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Nov 8 16:56:24 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Nov 2016 10:56:24 +1300 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: References: <22561.22853.957495.508409@turnbull.sk.tsukuba.ac.jp> Message-ID: <58224A08.608@canterbury.ac.nz> Another advantage of dunder method names is that you can google them. Someone coming across a method called "__foo__" can easily find documentation about it, but it's not so easy to do that for special syntax. -- Greg From prometheus235 at gmail.com Tue Nov 8 17:05:48 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 8 Nov 2016 16:05:48 -0600 Subject: [Python-ideas] Method signature syntactic sugar (especially for dunder methods) In-Reply-To: <58224A08.608@canterbury.ac.nz> References: <22561.22853.957495.508409@turnbull.sk.tsukuba.ac.jp> <58224A08.608@canterbury.ac.nz> Message-ID: Also you can support future changes to the syntax (e.g. __matmul__ and friends from 3.5, __aiter__ from 3.5.2) with a single codebase rather than having to push that grammar back to previous versions (impossible?) or have the grammar for magic methods be extraordinarily general (messy?) On Tue, Nov 8, 2016 at 3:56 PM, Greg Ewing wrote: > Another advantage of dunder method names is that you > can google them. Someone coming across a method called > "__foo__" can easily find documentation about it, but > it's not so easy to do that for special syntax. > > -- > Greg > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 10 11:13:20 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Nov 2016 02:13:20 +1000 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: <20161108161509.GC3365@ando.pearwood.info> References: <20161108161509.GC3365@ando.pearwood.info> Message-ID: On 9 November 2016 at 02:15, Steven D'Aprano wrote: > Whew! Nick, this is a big, complex PEP, and thank you for taking the > time to write it. Thanks :) > But its also hard to understand -- there's a lot of > detail, and there are places where it is easy for the reader to get > mislead by their assumptions and only get corrected deep, deep into the > PEP. At least that's my experience. This is another reason why I think the symmetric proposal I mentioned in the Risk and Concerns section may actually be a better idea, since it aligns better with the full "LHS if COND else RHS" spelling. It does make the short-circuiting more complex to explain, but if I went back to that I'd also reinstate Mark's explanation of how the short-circuiting would work and the associated diagram. > I think I'd find this PEP easier to understand if it were split into > two, like Guido's type-hints PEPs: one to explain the protocol, and one > to detail the new built-ins and syntactic sugar which rely on the > protocol. Or maybe that's just me. It's hard to explain the protocol without concrete examples to draw on, and `operator.true` and `operator.false` aren't really sufficient for that purpose (since they're more likely to elicit a reaction of "But that's the way conditional expressions work anyway..."). > I really like this PEP as a way of introducing configurable short- > circuiting behaviour, without messing with `and` and `or`. That's > really, really nice. I like your decision to keep the ?. and ?[] sugar, > as short-cuts for code based on this Circuit Breaker protocol. > > But I cannot support the exists() and missing() builtins as they stand. > I think the circuit breakers themselves work well as a concept, but: > > - I think that the names will be more harmful than helpful; I did consider the possible misinterpretation in terms of NameError, but probably didn't give it sufficient credence (since it isn't going to be immediately obvious to everyone that there's no way to readily delegate that check to a callable of any kind in Python, especially when other languages do offer that kind of "undefined" check) > - I don't think that having to explicitly call a circuit breaker is a > good substitute for the ?? operator. I think the "line signal to noise ratio" measures in the examples do a pretty decent job of highlighting that - the patterns involved are still quite verbose when using a named circuit breaker. In the next revision, I'll update the PEP to say it doesn't compete with 505 at all, and merely offers a proposal for making that PEP a semantically protocol based one. > If I absolutely had to choose between this and nothing, I'd say +1 for > this. But if I had to choose between ?? as a operator, and a generic > circuit breaking protocol with no operator but exists() builtin instead, > well, that would be a really hard decision. One of the reasons I was keen to get this written relatively quickly (aside from wanting to see for myself whether or not it actually hung together as a coherent design concept) was so we'd have plenty of time to consider the risks, opportunities and alternatives between now and 3.7. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From leewangzhong+python at gmail.com Thu Nov 10 16:07:25 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Thu, 10 Nov 2016 16:07:25 -0500 Subject: [Python-ideas] "Proposing a change" document? (was: Civility on this mailing list) Message-ID: (Is that the right way to change subject? Anyway, context here: https://mail.python.org/pipermail/python-ideas/2016-October/043208.html) On Fri, Oct 21, 2016 at 11:07 AM, Paul Moore wrote: > > I wonder. Would there be value in adding a sign-up email to the list > (supported by a posting of that email to the list, to catch existing > contributors) that set out some of the basic principles of how changes > are judged for inclusion in Python? Turn it around. Have a document outlining how one should _argue_ for their idea. Instead of, "We get a lot of ideas. Are you SURE your idea is good?" say, "We WANT to hear your idea, and here is the best way to get us to listen." A list of reasons why ideas get rejected will make them defensive, while a list of points they need to prove will put them on the offensive, and also get them thinking deeply about the potential costs/issues of their idea. Encourage arguments with sample code (before and after their change). Encourage having real-world code (or a stripped-down version of actual code) and showing how their change makes the code better. Emphasize _how_ this helps convince others. I think this attitude will reduce, not increase, the number of posts (from the people who read the document). It's easier to think, "My idea is amazing," than to think, "I've demonstrated all that I needed to." You might also get better-quality posts: instead of _mentally_ going down a checklist, they'd be writing out how their idea fulfills that checklist. I'm worried about how the community would self-police with such a document. Maybe they'll bite harder when someone doesn't follow it. > * The fact that the default answer is typically "no", along with an > overview of the reasons *why* the status quo wins by default. Instead: "Any new feature adds maintenance burden in the code, in the docs, and in the minds of Python developers. You should show how the benefits of the feature will outweigh these costs." (P.S.: I don't believe a lack of implementor is a point against an idea (except when there's a competing idea). It doesn't further the discussion for _whether_ the idea is good. It's not unheard of for ideas to reach PEP status, then stop there because no one will do the work. Someone can always find the discussion later, see that it's popular, and decide to work on it. Anyone wanna convince me otherwise?) > * Some of the simple "rules of thumb" like "not every 2-line function > should be a builtin. That, in particular, is a bad response type. It implies the poster's supporting arguments are strongly based on having _universal_ consistency, which is rarely the case. In my experience, "not every" (anti-universal?) counterarguments are usually used against non-universal arguments, and then you have "I didn't say they did" responses. Instead, contrast a few levels of inclusion, like built-ins, batteries, itertools recipes, PyPI, personal library. Talk about different scales to measure features, like "broadly useful" and "easy to specify/understand, but hard to implement". Have them pick a function or two at the same level of inclusion as their function, and show that their function measures favorably against the included function on the given scales. Have them think about (putting an implementation on) PyPI as a way to prove that their idea will be useful to people. It's not a backseat to stdlib, it's the way toward stdlib. > * Basic reminders that Python is used by a very diverse set of users, > and proposals that are only beneficial for a limited group need to be > weighed against the disruption to the majority who get no benefit. Instead, have them discuss who will benefit from their idea, and what it will cost everyone else who doesn't need it. In particular, how would a Python user, especially a novice, learn to read code written that uses their idea? Do other implementations need to implement this feature to be compatible with CPython? > * The above comment, that we welcome ideas because it's important that > we don't stagnate and having assumptions challenged is valuable, even > if the bar for getting such ideas accepted is (necessarily) high. I don't like that last part. It's sort of prideful, and also unnecessary. From ethan at stoneleaf.us Thu Nov 10 16:18:57 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 10 Nov 2016 13:18:57 -0800 Subject: [Python-ideas] "Proposing a change" document? (was: Civility on this mailing list) In-Reply-To: References: Message-ID: <5824E441.5020106@stoneleaf.us> On 11/10/2016 01:07 PM, Franklin? Lee wrote: > On Fri, Oct 21, 2016 at 11:07 AM, Paul Moore wrote: >> >> I wonder. Would there be value in adding a sign-up email to the list >> (supported by a posting of that email to the list, to catch existing >> contributors) that set out some of the basic principles of how changes >> are judged for inclusion in Python? > > Turn it around. Have a document outlining how one should _argue_ for > their idea. Instead of, "We get a lot of ideas. Are you SURE your idea > is good?" say, "We WANT to hear your idea, and here is the best way to > get us to listen." A list of reasons why ideas get rejected will make > them defensive, while a list of points they need to prove will put > them on the offensive, and also get them thinking deeply about the > potential costs/issues of their idea. +1. These seem like *really* good ideas to me. -- ~Ethan~ From p.f.moore at gmail.com Thu Nov 10 17:01:54 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Nov 2016 22:01:54 +0000 Subject: [Python-ideas] "Proposing a change" document? (was: Civility on this mailing list) In-Reply-To: <5824E441.5020106@stoneleaf.us> References: <5824E441.5020106@stoneleaf.us> Message-ID: On 10 November 2016 at 21:18, Ethan Furman wrote: > On 11/10/2016 01:07 PM, Franklin? Lee wrote: >> >> On Fri, Oct 21, 2016 at 11:07 AM, Paul Moore wrote: >>> >>> >>> I wonder. Would there be value in adding a sign-up email to the list >>> (supported by a posting of that email to the list, to catch existing >>> contributors) that set out some of the basic principles of how changes >>> are judged for inclusion in Python? >> >> >> Turn it around. Have a document outlining how one should _argue_ for >> their idea. Instead of, "We get a lot of ideas. Are you SURE your idea >> is good?" say, "We WANT to hear your idea, and here is the best way to >> get us to listen." A list of reasons why ideas get rejected will make >> them defensive, while a list of points they need to prove will put >> them on the offensive, and also get them thinking deeply about the >> potential costs/issues of their idea. > > > +1. These seem like *really* good ideas to me. Agreed - I like all of the points made in the post. Paul From shoyer at gmail.com Thu Nov 10 21:08:36 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 10 Nov 2016 18:08:36 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation of expressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: On Sun, Nov 6, 2016 at 5:32 PM, Nathaniel Smith wrote: > Filtering out a subset of rows from a data frame in pandas; 'height' > and 'age' refer to columns in the data frame (equivalent to > data_frame[data_frame["height"] > 100 and data_frame["age"] < 5], but > more ergonomic and faster (!)): > > data_frame.subset!(height > 100 and age < 5) > > (IIRC pandas has at least experimented with various weird lambda hacks > for this kind of thing; not sure what the current status is.) > We abandoned the experiment because we couldn't make it work properly. There's no way to inject local variables in the appropriate scope around the lambda function: https://github.com/pandas-dev/pandas/issues/13040 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Nov 10 21:47:44 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 10 Nov 2016 18:47:44 -0800 Subject: [Python-ideas] "Proposing a change" document? (was: Civility on this mailing list) In-Reply-To: References: <5824E441.5020106@stoneleaf.us> Message-ID: > +1. These seem like *really* good ideas to me. > > Agreed - I like all of the points made in the post. Me too. And while we're at it, it would be nice to document the process beyond the early stages -- once there seems to be some support for your idea -- what then? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 10 22:31:23 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Nov 2016 13:31:23 +1000 Subject: [Python-ideas] "Proposing a change" document? (was: Civility on this mailing list) In-Reply-To: References: <5824E441.5020106@stoneleaf.us> Message-ID: On 11 November 2016 at 08:01, Paul Moore wrote: > On 10 November 2016 at 21:18, Ethan Furman wrote: >> On 11/10/2016 01:07 PM, Franklin? Lee wrote: >>> >>> On Fri, Oct 21, 2016 at 11:07 AM, Paul Moore wrote: >>>> >>>> >>>> I wonder. Would there be value in adding a sign-up email to the list >>>> (supported by a posting of that email to the list, to catch existing >>>> contributors) that set out some of the basic principles of how changes >>>> are judged for inclusion in Python? >>> >>> >>> Turn it around. Have a document outlining how one should _argue_ for >>> their idea. Instead of, "We get a lot of ideas. Are you SURE your idea >>> is good?" say, "We WANT to hear your idea, and here is the best way to >>> get us to listen." A list of reasons why ideas get rejected will make >>> them defensive, while a list of points they need to prove will put >>> them on the offensive, and also get them thinking deeply about the >>> potential costs/issues of their idea. >> >> >> +1. These seem like *really* good ideas to me. > > Agreed - I like all of the points made in the post. Providing some notes on what we currently have along these lines. In the top level of the devguide, we have: * https://docs.python.org/devguide/#proposing-changes-to-python-itself Which links to: * https://docs.python.org/devguide/stdlibchanges.html * https://docs.python.org/devguide/langchanges.html * https://docs.python.org/devguide/faq.html#suggesting-changes And the FAQ further links to: * http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html * http://www.curiousefficiency.org/posts/2011/02/justifying-python-language-changes.html For PEP level changes, PEP 1 gives some limited guidance when describing the recommended "Motivation" and "Rationale" sections: https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep We don't currently have anything specific linked from the python-ideas list description, and the above can be a bit hard to follow since it's scattered across several pages. We also don't have anything like the document Franklin suggested explaining Python's feature distribution hierarchy (language features, standard library features, documented recipes/patterns, bundled modules, general purpose PyPI modules, project specific utility modules, private utility modules) and the trade-offs between them (basically, as you get lower down the stack, your potential distribution reach goes up, but your rate of change goes down and your adoption cycles start taking longer). Along the same lines, we don't currently have anything specifically stating that dependency management has a non-zero cost and that copying and pasting utility code into a project specific utility module is sometimes the right thing to do. Not specifically related to python-ideas and somewhat dated in the specifics, but related to the topic of conflict management on the core design lists, there is also: * http://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html And more recently Brett published a good piece on time and energy management as a volunteer contributor to open source project maintenance: * http://www.snarky.ca/why-i-took-october-off-from-oss-volunteering So there are probably two different documents that could be helpful if someone had the time and inclination to draft them based on what we have already: - "Creating Effective Change Proposals for Python" - "Expectations for Contributors to the Python Design Process" The first would be about constructing good design arguments, and pointing out the implications of core Python development placing a significant emphasis on educational use cases (i.e. a way for people to acquire computational literacy in the first place, not just a way for us to tell computers what we want them to do) and "incidental programming" use cases (i.e. a way for folks whose primary expertise lies in areas other than programming or computer science to nevertheless be able to automate the routine aspects of their work). The second would be about setting appropriate boundaries around what's reasonable for participants to expect from fellow participants (including explicitly pointing that if folks are looking for a customer experience rather than a contributor experience, there are multiple commercial Python redistributors out there willing to sell them one). Neither document would need to be perfect - as with many things, we could start with something relatively simple and iterate from there. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From drekin at gmail.com Fri Nov 11 05:48:31 2016 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Fri, 11 Nov 2016 11:48:31 +0100 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol Message-ID: Just a small suggestion: wouldn't it be better to use adjectives instead of verbs for proposed builtin names and their variants? existing(foo) else bar non_None(foo) else bar instead of exists(foo) else bar is_not_None(foo) else bar Regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From aresowj at gmail.com Fri Nov 11 16:39:29 2016 From: aresowj at gmail.com (Ares Ou) Date: Fri, 11 Nov 2016 13:39:29 -0800 Subject: [Python-ideas] Add a method to get the subset of a dictionnary. In-Reply-To: References: <0945e829-6936-9bb9-5d4f-5c85ef01fd69@gmail.com> Message-ID: > > On 10/12/2016 5:52 PM, Terry Reedy wrote: > > Test code before posting. The above is a set comprehension creating a > set of tupes. > I should have followed my own advice. The above is a SyntaxError until > 'k,v' is wrapped in parens, '(k,v)'. On 10/12/2016 12:06 PM, Enguerrand Pelletier wrote: > b = {k, v for k,v in a.items() if k in interesting_keys} >> > Just my own opinion, I don't pretty much get you on adding a method generating Sets to the dictionary object. Best regards, Ares Ou *Software Engineer / Full-Stack Python Developer* *Blog:* http://aresou.net | *Github:* https://github.com/aresowj *Stack Overflow:* http://stackoverflow.com/users/5183727/ares-ou Ares Ou On Wed, Oct 12, 2016 at 3:03 PM, Terry Reedy wrote: > On 10/12/2016 5:52 PM, Terry Reedy wrote: > >> On 10/12/2016 12:06 PM, Enguerrand Pelletier wrote: >> > > b = {k, v for k,v in a.items() if k in interesting_keys} >>> >> >> Test code before posting. The above is a set comprehension creating a >> set of tupes. >> > > I should have followed my own advice. The above is a SyntaxError until > 'k,v' is wrapped in parens, '(k,v)'. > > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Nov 12 03:32:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 Nov 2016 19:32:33 +1100 Subject: [Python-ideas] Add a method to get the subset of a dictionnary. In-Reply-To: <0945e829-6936-9bb9-5d4f-5c85ef01fd69@gmail.com> References: <0945e829-6936-9bb9-5d4f-5c85ef01fd69@gmail.com> Message-ID: <20161112083233.GL3365@ando.pearwood.info> On Wed, Oct 12, 2016 at 06:06:51PM +0200, Enguerrand Pelletier wrote: > Hi all, > > It always bothered me to write something like this when i want to strip > keys from a dictionnary in Python: > > a = {"foo": 1, "bar": 2, "baz": 3, "foobar": 42} > interesting_keys = ["foo", "bar", "baz"] > b = {k, v for k,v in a.items() if k in interesting_keys} (You have a small typo: should be "k: v" not "k, v".) Why does it bother you? It is simple, easy to understand, and explict. > Wouldn't it be nice to have a syntactic sugar such as: Syntactic sugar is not really the right term, "syntactic sugar" means a special form of syntax as a short-cut for something longer. This is not special syntax, it is just a method. > b = a.subset(interesting_keys) Personally, I have never missed this method, but if I did, it would be easy to make a helper function: def subset(adict, keys): """Return a new dict from existing keys.""" return {k: v for k,v in a.items() if k in keys} Not every simple helper function needs to be built-in to the class. That is the beauty of Python, you can make your own helper functions, once you give up the idea that everything needs to be a method. There are some problems with making this a method. To start with, it means that every dict and mapping would have to support it. Perhaps that is acceptible, but it does mean that the question is bigger than just dict. It also involves: ChainMap Counter Mapping MutableMapping OrderedDict UserDict defaultdict at the very least. (Perhaps this is easy to implement, by just adding this to the Mapping ABC and letting everything else inherit from that. But even so, it increases the complexity of the entire Mapping ABC and all its classes.) But a bigger problem with making this a built-in dict method is deciding exactly what it should do. Here are some options: - should the method return a new dict, or modify the existing dict? - should it keep the "interesting keys" or remove them? - is it an error if one of the interesting keys is missing? - or should it be silently skipped? - or automatically added? using what value? Whatever options we pick here, you can be sure that some people will want a different set of options. Unless we are sure that one combination is much more common than the other combinations, we're better off letting people write their own helper functions that behave exactly as they want: def subset(d, keys): # Version which raises if any of the keys are missing return {key: d[key] for key in keys} -- Steve From steve at pearwood.info Sat Nov 12 03:53:53 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 Nov 2016 19:53:53 +1100 Subject: [Python-ideas] Add a method to get the subset of a dictionnary. In-Reply-To: <0945e829-6936-9bb9-5d4f-5c85ef01fd69@gmail.com> References: <0945e829-6936-9bb9-5d4f-5c85ef01fd69@gmail.com> Message-ID: <20161112085353.GM3365@ando.pearwood.info> Oh, I forgot to mention... On Wed, Oct 12, 2016 at 06:06:51PM +0200, Enguerrand Pelletier wrote: > Hi all, > > It always bothered me to write something like this when i want to strip > keys from a dictionnary in Python: [snip] You've started a completely new discussion on an unrelated topic, but you appear to have done so by replying to an existing email conversation and editing the subject line. Specifically the thread with subject: "Improve error message when missing 'self' ..." Editing the subject line to indicate a change in topic is good! Thank you for doing that! But you should be aware that many email programs will thread this new, unrelated conversation under the "Improve error message" conversation. That means some people may see a threaded view something like this: Subject: [Python-ideas] Improve error message when missing 'self' ... ??> Re: [Python-ideas] Improve error message ... ? ??> ? ????> ? ? ??> ? ??> ??>[Python-ideas] Add a method to get the subset ... ??> Re: [Python-ideas] Add a method to get ... which may not matter if they are actually related threads, but anyone who has blocked or muted the original "Improve error message..." will not see your new conversation either. This may be considered part of the older conversation, and muted as well. When you plan to start a brand new conversation, unrelated to an existing thread, it is best to start a fresh email, don't reply to an existing one. Changing the subject line is not sufficient to break the link between your email and the previous conversation. You should use your email program's New Email command, type in the new subject line, and then set the To address to python-ideas at python.org. That's the only way to avoid the risk that some people won't see your email because they've muted the parent thread. Hope this helps, Steve From steve at pearwood.info Sat Nov 12 04:26:26 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 Nov 2016 20:26:26 +1100 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> Message-ID: <20161112092624.GN3365@ando.pearwood.info> On Thu, Oct 06, 2016 at 04:19:17PM -0700, Neil Girdhar wrote: > Currently str(slice(10)) returns "slice(None, 10, None)" > > If the start and step are None, consider not emitting them. Similarly > slice(None) is rendered slice(None, None, None). > > When you're printing a lot of slices, it's a lot of extra noise. I have an alternative suggestion. Wouldn't it be nice if slice objects looked something like the usual slice syntax? If you think the answer is No, then you'll hate my suggestion :-) Let's keep the current repr() of slice objects as they are, using the full function-call syntax complete with all three arguments show explicitly: repr(slice(None, None, None)) => "slice(None, None, None)" But let's make str() of a slice more suggestive of actual slicing, and as a bonus, make slices easier to create too. str(slice(None, None, None)) => "slice[:]" Let the slice type itself be sliceable, as an alternate constuctor: slice[:] => returns slice(None) slice[start:] => returns slice(start, None) slice[:end] => returns slice(None, end) slice[start::step] => returns slice(start, None, step) and so forth. (This probably would require changing the type of slice to a new metaclass.) And then have str() return the compact slice syntax. At worst, the compact slice syntax is one character longer than the optimal function syntax: # proposed slice str() slice[:7] # 9 characters # proposed compact str() slice(7) # 8 characters # current str() slice(None, 7, None) # 20 characters but it will be more compact more often: slice[1:] # 9 characters versus: slice(1, None) # 14 characters slice(None, 1, None) # 20 characters -- Steve From levkivskyi at gmail.com Sat Nov 12 04:37:26 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 12 Nov 2016 10:37:26 +0100 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: <20161112092624.GN3365@ando.pearwood.info> References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On 12 November 2016 at 10:26, Steven D'Aprano wrote: > But let's make str() of a slice more suggestive of actual slicing, and > as a bonus, make slices easier to create too. > > str(slice(None, None, None)) => "slice[:]" > > Let the slice type itself be sliceable, as an alternate constuctor: > > slice[:] => returns slice(None) > slice[start:] => returns slice(start, None) > slice[:end] => returns slice(None, end) > slice[start::step] => returns slice(start, None, step) > > and so forth. (This probably would require changing the type of slice to > a new metaclass.) > > And then have str() return the compact slice syntax. > +1, I like this idea, this is very close to what NumPy does. I would also mention http://bugs.python.org/issue24379 -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Nov 12 09:06:15 2016 From: steve.dower at python.org (Steve Dower) Date: Sat, 12 Nov 2016 06:06:15 -0800 Subject: [Python-ideas] Alternative to PEP 532: delayed evaluation ofexpressions In-Reply-To: References: <798e40e2-54db-1037-ae7b-637e2116190e@trueblade.com> Message-ID: Would it be helped by an explicit "free variable" marker? (I'm sure I've seen someone demo a prototype of this): >>> data_frame.subset($height > 100 and $age < 5) Which essentially translates into: >>> data_frame.subset(lambda **a: a["height"] > 100 and a["age"] < 5) Maybe the generated thunk can keep the AST around too in case there are better transformations available (e.g. convert into a SQL/Blaze query), but simply calling it with named arguments or a mapping (I am deliberately not requiring the eventual caller to know the exact signature) would get you the result with a mix of closed and free variables. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Stephan Hoyer" Sent: ?11/?10/?2016 18:09 To: "Nathaniel Smith" Cc: "Eric V. Smith" ; "Python-Ideas" Subject: Re: [Python-ideas] Alternative to PEP 532: delayed evaluation ofexpressions On Sun, Nov 6, 2016 at 5:32 PM, Nathaniel Smith wrote: Filtering out a subset of rows from a data frame in pandas; 'height' and 'age' refer to columns in the data frame (equivalent to data_frame[data_frame["height"] > 100 and data_frame["age"] < 5], but more ergonomic and faster (!)): data_frame.subset!(height > 100 and age < 5) (IIRC pandas has at least experimented with various weird lambda hacks for this kind of thing; not sure what the current status is.) We abandoned the experiment because we couldn't make it work properly. There's no way to inject local variables in the appropriate scope around the lambda function: https://github.com/pandas-dev/pandas/issues/13040 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rikudou__sennin at live.com Sat Nov 12 11:44:28 2016 From: rikudou__sennin at live.com (adil gourinda) Date: Sat, 12 Nov 2016 16:44:28 +0000 Subject: [Python-ideas] Python Documentation Message-ID: I) Thank you for responding to my message because it makes me happy to know that my message was read by your team. II) The ?.okular? file is an archive made by ?Okular software? and contains the ?PDF? file with its annotations. ?https://okular.kde.org/? this is the link to the software, I work with ?openSUSE? ?https://www.opensuse.org/?. III) For someone who wants to learn programming language and chose ?Python? to start with it, Having a good tutorial is a crucial moment. Me as an example, I don?t follow any study in informatics field so I build my knowledge thanks to my efforts, In consequence I don?t have a strong base in programming language. Even that I found some parts in ?Python Language Reference? and ?Python library reference? easy to learn for someone like me. I suggest to aggregate those informations and add them to the ?Python tutorial? to build a strong introduction to your great language ?Python?. From mehaase at gmail.com Sat Nov 12 11:47:00 2016 From: mehaase at gmail.com (Mark E. Haase) Date: Sat, 12 Nov 2016 11:47:00 -0500 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: I like PEP-532. Given the opposition to non-Pythonic syntax like ?? (i.e. PEP-505), Nick's proposal offers a Pythonic alternative that is protocol based, more generalized, and uses built-ins and keywords to avoid punctuation. I agree with other posters that the terms "exists" and "missing" could lead developers to think that it tests for NameError. Maybe "value(foo) else bar"? I can't think of a better spelling for the inverse. Maybe the "if" syntax described in the PEP is better: "foo.bar if value(foo)". In that case, we wouldn't need an inverse to exists()/value()/whatever. I also wanted to mention a couple of unmentioned benefits of this PEP: 1. It is easier to Google a name. E.g., Google "c# ??" and you'll get nothing related to null coalescing in c#". ("C# question marks" does find the right content, however.) 2. Dismay over the meaning of foo?.bar.baz is much clearer when expressed as missing(foo) else foo.bar.baz -- it's very close to the ternary logic you'd write if you didn't have a circuit breaking operator: None if foo is None else foo.bar.baz. On Sat, Nov 5, 2016 at 5:50 AM, Nick Coghlan wrote: > Hi folks, > > As promised, here's a follow-up to the withdrawn PEP 531 that focuses > on coming up with a common protocol driven solution to conditional > evaluation of subexpressions that also addresses the element-wise > comparison chaining problem that Guido noted when rejecting PEP 335. > > I quite like how it came out, but see the "Risks & Concerns" section > for a discussion of an internal inconsistency it would introduce into > the language if accepted as currently written, and the potentially > far-reaching consequences actually resolving that inconsistency might > have on the way people write their Python code (if the PEP was > subsequently approved). > > Regards, > Nick. > > ================================= > PEP: 532 > Title: A circuit breaking operator and protocol > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 30-Oct-2016 > Python-Version: 3.7 > Post-History: 5-Nov-2016 > > Abstract > ======== > > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this > PEP > proposes the addition of a new protocol-driven circuit breaking operator to > Python that allows the left operand to decide whether or not the expression > should short circuit and return a result immediately, or else continue > on with evaluation of the right operand:: > > exists(foo) else bar > missing(foo) else foo.bar() > > These two expressions can be read as: > > * "the expression result is 'foo' if it exists, otherwise it is 'bar'" > * "the expression result is 'foo' if it is missing, otherwise it is > 'foo.bar()'" > > Execution of these expressions relies on a new circuit breaking protocol > that > implicitly avoids repeated evaluation of the left operand while letting > that operand fully control the result of the expression, regardless of > whether > it skips evaluation of the right operand or not:: > > _lhs = LHS > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) > > To properly support logical negation of circuit breakers, a new ``__not__`` > protocol method would also be introduced allowing objects to control > the result of ``not obj`` expressions. > > As shown in the basic example above, the PEP further proposes the addition > of > builtin ``exists`` and ``missing`` circuit breakers that provide > conditional > branching based on whether or not an object is ``None``, but return the > original object rather than the existence checking wrapper when the > expression > evaluation short circuits. > > In addition to being usable as simple boolean operators (e.g. as in > ``assert all(exists, items)`` or ``if any(missing, items):``), these > circuit > breakers will allow existence checking fallback operations (aka > None-coalescing > operations) to be written as:: > > value = exists(expr1) else exists(expr2) else expr3 > > and existence checking precondition operations (aka None-propagating > or None-severing operations) to be written as:: > > value = missing(obj) else obj.field.of.interest > value = missing(obj) else obj["field"]["of"]["interest"] > > A change to the definition of chained comparisons is also proposed, where > the comparison chaining will be updated to use the circuit breaking > operator > rather than the logical disjunction (``and``) operator if the left hand > comparison returns a circuit breaker as its result. > > While there are some practical complexities arising from the current > handling > of single-valued arrays in NumPy, this change should be sufficient to allow > elementwise chained comparison operations for matrices, where the result > is a matrix of boolean values, rather than tautologically returning > ``True`` > or raising ``ValueError``. > > > Relationship with other PEPs > ============================ > > This PEP is a direct successor to PEP 531, replacing the existence checking > protocol and the new ``?then`` and ``?else`` syntactic operators defined > there > with a single protocol driven ``else`` operator and adjustments to the > ``not`` > operator. The existence checking use cases are taken from that PEP. > > It is also a direct successor to PEP 335, which proposed the ability to > overload the ``and`` and ``or`` operators directly, with the ability to > overload the semantics of comparison chaining being one of the consequences > of that change. The proposal in this PEP to instead handle the element-wise > comparison use case by changing the semantic definition of comparison > chaining > is drawn from Guido's rejection of PEP 335. > > This PEP competes with the dedicated null-coalescing operator in PEP 505, > proposing that improved support for null-coalescing operations be offered > through a more general protocol-driven short circuiting operator and > related > builtins, rather than through a dedicated single-purpose operator. > > It doesn't compete with PEP 505's proposed shorthands for existence > checking > attribute access and subscripting, but instead offers an alternative > underlying > semantic framework for defining them: > > * ``EXPR?.attr`` would be syntactic sugar for ``missing(EXPR) else > EXPR.attr`` > * ``EXPR?[key]`` would be syntactic sugar for ``missing(EXPR) else > EXPR[key]`` > > In both cases, the dedicated syntactic form could be optimised to avoid > actually creating the circuit breaker instance. > > > Specification > ============= > > The circuit breaking operator (``else``) > ---------------------------------------- > > Circuit breaking expressions would be written using ``else`` as a new > binary > operator, akin to the existing ``and`` and ``or`` logical operators:: > > LHS else RHS > > Ignoring the hidden variable assignment, this is semantically equivalent > to:: > > _lhs = LHS > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) > > The key difference relative to the existing ``or`` operator is that the > value > determining which branch of the conditional expression gets executed *also* > gets a chance to postprocess the results of the expressions on each of the > branches. > > As part of the short-circuiting behaviour, interpreter implementations > are expected to access only the protocol method needed for the branch > that is actually executed, but it is still recommended that circuit > breaker authors that always return ``True`` or always return ``False`` from > ``__bool__`` explicitly raise ``NotImplementedError`` with a suitable > message from branch methods that are never expected to be executed (see the > comparison chaining use case in the Rationale section below for an example > of that). > > It is proposed that the ``else`` operator use a new precedence level that > binds > less tightly than the ``or`` operator by adjusting the relevant line in > Python's grammar from the current:: > > test: or_test ['if' or_test 'else' test] | lambdef > > to instead be:: > > test: else_test ['if' or_test 'else' test] | lambdef > else_test: or_test ['else' test] > > The definition of ``test_nocond`` would remain unchanged, so circuit > breaking expressions would require parentheses when used in the ``if`` > clause of comprehensions and generator expressions just as conditional > expressions themselves do. > > This grammar definition means precedence/associativity in the otherwise > ambiguous case of ``expr1 if cond else expr2 else epxr3`` resolves as > ``(expr1 if cond else expr2) else epxr3``. > > A guideline will also be added to PEP 8 to say "don't do that", as such a > construct will be inherently confusing for readers, regardless of how the > interpreter executes it. > > > Overloading logical inversion (``not``) > --------------------------------------- > > Any circuit breaker definition will have a logical inverse that is still a > circuit breaker, but inverts the answer as to whether or not to short > circuit > the expression evaluation. For example, the ``exists`` and ``missing`` > circuit > breakers proposed in this PEP are each other's logical inverse. > > A new protocol method, ``__not__(self)``, will be introduced to permit > circuit > breakers and other types to override ``not`` expressions to return their > logical inverse rather than a coerced boolean result. > > To preserve the semantics of existing language optimisations, ``__not__`` > implementations will be obliged to respect the following invariant:: > > assert not bool(obj) == bool(not obj) > > > Chained comparisons > ------------------- > > A chained comparison like ``0 < x < 10`` written as:: > > LEFT_BOUND left_op EXPR right_op RIGHT_BOUND > > is currently roughly semantically equivalent to:: > > _expr = EXPR > _lhs_result = LEFT_BOUND left_op _expr > _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) > > This PEP proposes that this be changed to explicitly check if the left > comparison returns a circuit breaker, and if so, use ``else`` rather than > ``and`` to implement the comparison chaining:: > > _expr = EXPR > _lhs_result = LEFT_BOUND left_op _expr > if hasattr(type(_lhs_result), "__then__"): > _expr_result = _lhs_result else (_expr right_op RIGHT_BOUND) > else: > _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) > > This allows types like NumPy arrays to control the behaviour of chained > comparisons by returning circuit breakers from comparison operations. > > > Existence checking comparisons > ------------------------------ > > Two new builtins implementing the new protocol are proposed to encapsulate > the > notion of "existence checking": seeing if a value is ``None`` and either > falling back to an alternative value (an operation known as > "None-coalescing") > or passing it through as the result of the overall expression (an operation > known as "None-severing" or "None-propagating"). > > These builtins would be defined as follows:: > > class CircuitBreaker: > """Base circuit breaker type (available as types.CircuitBreaker)""" > def __init__(self, value, condition, inverse_type): > self.value = value > self._condition = condition > self._inverse_type = inverse_type > def __bool__(self): > return self._condition > def __not__(self): > return self._inverse_type(self.value) > def __then__(self): > return self.value > def __else__(self, other): > if other is self: > return self.value > return other > > class exists(types.CircuitBreaker): > """Circuit breaker for 'EXPR is not None' checks""" > def __init__(self, value): > super().__init__(value, value is not None, missing) > > class missing(types.CircuitBreaker): > """Circuit breaker for 'EXPR is None' checks""" > def __init__(self, value): > super().__init__(value, value is None, exists) > > Aside from changing the definition of ``__bool__`` to be based on > ``is not None`` rather than normal truth checking, the key characteristic > of > ``exists`` is that when it is used as a circuit breaker, it is *ephemeral*: > when it is told that short circuiting has taken place, it returns the > original > value, rather than the existence checking wrapper. > > ``missing`` is defined as the logically inverted counterpart of ``exists``: > ``not exists(obj)`` is semantically equivalent to ``missing(obj)``. > > The ``__else__`` implementations for both builtin circuit breakers are > defined > such that the wrapper will always be removed even if you explicitly pass > the > circuit breaker to both sides of the ``else`` expression:: > > breaker = exists(foo) > assert (breaker else breaker) is foo > breaker = missing(foo) > assert (breaker else breaker) is foo > > > Other conditional constructs > ---------------------------- > > No changes are proposed to if statements, while statements, conditional > expressions, comprehensions, or generator expressions, as the boolean > clauses > they contain are already used for control flow purposes. > > However, it's worth noting that while such proposals are outside the scope > of > this PEP, the circuit breaking protocol defined here would be sufficient to > support constructs like:: > > while exists(dynamic_query()) as result: > ... # Code using result > > and: > > if exists(re.search(pattern, text)) as match: > ... # Code using match > > Leaving the door open to such a future extension is the main reason for > recommending that circuit breaker implementations handle the ``self is > other`` > case in ``__else__`` implementations the same way as they handle the > short-circuiting behaviour in ``__then__``. > > > Style guide recommendations > --------------------------- > > The following additions to PEP 8 are proposed in relation to the new > features > introduced by this PEP: > > * In the absence of other considerations, prefer the use of the builtin > circuit breakers ``exists`` and ``missing`` over the corresponding > conditional expressions > > * Do not combine conditional expressions (``if-else``) and circuit breaking > expressions (the ``else`` operator) in a single expression - use one or > the > other depending on the situation, but not both. > > > Rationale > ========= > > Adding a new operator > --------------------- > > Similar to PEP 335, early drafts of this PEP focused on making the existing > ``and`` and ``or`` operators less rigid in their interpretation, rather > than > proposing new operators. However, this proved to be problematic for a few > reasons: > > * defining a shared protocol for both ``and`` and ``or`` was confusing, as > ``__then__`` was the short-circuiting outcome for ``or``, while > ``__else__`` > was the short-circuiting outcome for ``and`` > * the ``and`` and ``or`` operators have a long established and stable > meaning, > so readers would inevitably be surprised if their meaning now became > dependent on the type of the left operand. Even new users would be > confused > by this change due to 25+ years of teaching material that assumes the > current well-known semantics for these operators > * Python interpreter implementations, including CPython, have taken > advantage > of the existing semantics of ``and`` and ``or`` when defining runtime and > compile time optimisations, which would all need to be reviewed and > potentially discarded if the semantics of those operations changed > > Proposing a single new operator instead resolves all of those issues - > ``__then__`` always indicates short circuiting, ``__else__`` only indicates > "short circuiting" if the circuit breaker itself is also passed in as the > right operand, and the semantics of ``and`` and ``or`` remain entirely > unchanged. While the semantics of the unary ``not`` operator do change, the > invariant required of ``__not__`` implementations means that existing > expression optimisations in boolean contexts will remain valid. > > As a result of that design simplification, the new protocol and operator > would > even allow us to expose ``operator.true`` and ``operator.false`` > as circuit breaker definitions if we chose to do so:: > > class true(types.CircuitBreaker): > """Circuit breaker for 'bool(EXPR)' checks""" > def __init__(self, value): > super().__init__(value, bool(value), when_false) > > class false(types.CircuitBreaker): > """Circuit breaker for 'not bool(EXPR)' checks""" > def __init__(self, value): > super().__init__(value, not bool(value), when_true) > > Given those circuit breakers: > > * ``LHS or RHS`` would be roughly ``operator.true(LHS) else RHS`` > * ``LHS and RHS`` would be roughly ``operator.false(LHS) else RHS`` > > > Naming the operator and protocol > -------------------------------- > > The names "circuit breaking operator", "circuit breaking protocol" and > "circuit breaker" are all inspired by the phrase "short circuiting > operator": > the general language design term for operators that only conditionally > evaluate their right operand. > > The electrical analogy is that circuit breakers in Python detect and handle > short circuits in expressions before they trigger any exceptions similar > to the > way that circuit breakers detect and handle short circuits in electrical > systems before they damage any equipment or harm any humans. > > The Python level analogy is that just as a ``break`` statement lets you > terminate a loop before it reaches its natural conclusion, a circuit > breaking > expression lets you terminate evaluation of the expression and produce a > result > immediately. > > > Using an existing keyword > ------------------------- > > Using an existing keyword has the benefit of allowing the new expression to > be introduced without a ``__future__`` statement. > > ``else`` is semantically appropriate for the proposed new protocol, and the > only syntactic ambiguity introduced arises when the new operator is > combined > with the explicit ``if-else`` conditional expression syntax. > > > Element-wise chained comparisons > -------------------------------- > > In ultimately rejecting PEP 335, Guido van Rossum noted [1_]: > > The NumPy folks brought up a somewhat separate issue: for them, > the most common use case is chained comparisons (e.g. A < B < C). > > To understand this observation, we first need to look at how comparisons > work > with NumPy arrays:: > > >>> import numpy as np > >>> increasing = np.arange(5) > >>> increasing > array([0, 1, 2, 3, 4]) > >>> decreasing = np.arange(4, -1, -1) > >>> decreasing > array([4, 3, 2, 1, 0]) > >>> increasing < decreasing > array([ True, True, False, False, False], dtype=bool) > > Here we see that NumPy array comparisons are element-wise by default, > comparing > each element in the lefthand array to the corresponding element in the > righthand > array, and producing a matrix of boolean results. > > If either side of the comparison is a scalar value, then it is broadcast > across > the array and compared to each individual element:: > > >>> 0 < increasing > array([False, True, True, True, True], dtype=bool) > >>> increasing < 4 > array([ True, True, True, True, False], dtype=bool) > > However, this broadcasting idiom breaks down if we attempt to use chained > comparisons:: > > >>> 0 < increasing < 4 > Traceback (most recent call last): > File "", line 1, in > ValueError: The truth value of an array with more than one element > is ambiguous. Use a.any() or a.all() > > The problem is that internally, Python implicitly expands this chained > comparison into the form:: > > >>> 0 < increasing and increasing < 4 > Traceback (most recent call last): > File "", line 1, in > ValueError: The truth value of an array with more than one element > is ambiguous. Use a.any() or a.all() > > And NumPy only permits implicit coercion to a boolean value for > single-element > arrays where ``a.any()`` and ``a.all()`` can be assured of having the same > result:: > > >>> np.array([False]) and np.array([False]) > array([False], dtype=bool) > >>> np.array([False]) and np.array([True]) > array([False], dtype=bool) > >>> np.array([True]) and np.array([False]) > array([False], dtype=bool) > >>> np.array([True]) and np.array([True]) > array([ True], dtype=bool) > > The proposal in this PEP would allow this situation to be changed by > updating > the definition of element-wise comparison operations in NumPy to return a > dedicated subclass that implements the new circuit breaking protocol and > also > changes the result array's interpretation in a boolean context to always > return ``False`` and hence never trigger the short-circuiting behaviour:: > > class ComparisonResultArray(np.ndarray): > def __bool__(self): > return False > def _raise_NotImplementedError(self): > msg = ("Comparison array truth values are ambiguous outside " > "chained comparisons. Use a.any() or a.all()") > raise NotImplementedError(msg) > def __not__(self): > self._raise_NotImplementedError() > def __then__(self): > self._raise_NotImplementedError() > def __else__(self, other): > return np.logical_and(self, other.view(ComparisonResultArray)) > > With this change, the chained comparison example above would be able to > return:: > > >>> 0 < increasing < 4 > ComparisonResultArray([ False, True, True, True, False], dtype=bool) > > > Existence checking expressions > ------------------------------ > > An increasingly common requirement in modern software development is the > need > to work with "semi-structured data": data where the structure of the data > is > known in advance, but pieces of it may be missing at runtime, and the > software > manipulating that data is expected to degrade gracefully (e.g. by omitting > results that depend on the missing data) rather than failing outright. > > Some particularly common cases where this issue arises are: > > * handling optional application configuration settings and function > parameters > * handling external service failures in distributed systems > * handling data sets that include some partial records > > At the moment, writing such software in Python can be genuinely awkward, as > your code ends up littered with expressions like: > > * ``value1 = expr1.field.of.interest if expr1 is not None else None`` > * ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else > None`` > * ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not > None else expr5`` > > PEP 531 goes into more detail on some of the challenges of working with > this > kind of data, particularly in data transformation pipelines where dealing > with > potentially missing content is the norm rather than the exception. > > The combined impact of the proposals in this PEP is to allow the above > sample > expressions to instead be written as: > > * ``value1 = missing(expr1) else expr1.field.of.interest`` > * ``value2 = missing(expr2) else expr2.["field"]["of"]["interest"]`` > * ``value3 = exists(expr3) else exists(expr4) else expr5`` > > In these forms, significantly more of the text presented to the reader is > immediately relevant to the question "What does this code do?", while the > boilerplate code to handle missing data by passing it through to the output > or falling back to an alternative input, has shrunk to two uses of the new > ``missing`` builtin, and two uses of the new ``exists`` builtin. > > In the first two examples, the 31 character boilerplate suffix > ``if exprN is not None else None`` (minimally 27 characters for a single > letter > variable name) has been replaced by a 19 character ``missing(expr1) else`` > prefix (minimally 15 characters with a single letter variable name), > markedly > improving the signal-to-pattern-noise ratio of the lines (especially if it > encourages the use of more meaningful variable and field names rather than > making them shorter purely for the sake of expression brevity). The > additional > syntactic sugar proposals in PEP 505 would further reduce this boilerplate > to > a single ``?`` character that also eliminated the repetition of the > expession > being checked for existence. > > In the last example, not only are two instances of the 21 character > boilerplate, > `` if exprN is not None`` (minimally 17 characters) replaced with the > 8 character function call ``exists()``, but that function call is placed > directly around the original expression, eliminating the need to duplicate > it > in the conditional existence check. > > > Risks and concerns > ================== > > This PEP has been designed specifically to address the risks and concerns > raised when discussing PEPs 335, 505 and 531. > > * it defines a new operator and adjusts the definition of chained > comparison > rather than impacting the existing ``and`` and ``or`` operators > * the changes to the ``not`` unary operator are defined in such a way that > control flow optimisations based on the existing semantics remain valid > * rather than the cryptic ``??``, it uses ``else`` as the operator keyword > in > exactly the same sense as it is already used in conditional expressions > * it defines a general purpose short-circuiting binary operator that can > even > be used to express the existing semantics of ``and`` and ``or`` rather > than > focusing solely and inflexibly on existence checking > * it names the proposed builtins in such a way that they provide a strong > mnemonic hint as to when the expression containing them will > short-circuit > and skip evaluating the right operand > > > Possible confusion with conditional expressions > ----------------------------------------------- > > The proposal in this PEP is essentially for an "implied ``if``" where if > you > omit the ``if`` clause from a conditional expression, you invoke the > circuit > breaking protocol instead. That is:: > > exists(foo) else calculate_default() > > invokes the new protocol, but:: > > foo.field.of.interest if exists(foo) else calculate_default() > > bypasses it entirely, *including* the non-short-circuiting ``__else__`` > method. > > This mostly wouldn't be a problem for the proposed ``types.CircuitBreaker`` > implementation (and hence the ``exists`` and ``missing`` builtins), as the > only purpose the extended protocol serves in that case is to remove the > wrapper in the short-circuiting case - the ``__else__`` method passes the > right operand through unchanged. > > However, this discrepancy could potentially be eliminated entirely by also > updating conditional expressions to use the circuit breaking protocol if > the condition defines those methods. In that case, ``__then__`` would need > to be updated to accept the left operand as a parameter, with > short-circuiting > indicated by passing in the circuit breaker itself:: > > class CircuitBreaker: > """Base circuit breaker type (available as types.CircuitBreaker)""" > def __init__(self, value, condition, inverse_type): > self.value = value > self._condition = condition > self._inverse_type = inverse_type > def __bool__(self): > return self._condition > def __not__(self): > return self._inverse_type(self.value) > def __then__(self, other): > if other is not self: > return other > return self.value # Short-circuit, remove the wrapper > def __else__(self, other): > if other is not self: > return other > return self.value # Short-circuit, remove the wrapper > > With this symmetric protocol, the definition of conditional expressions > could be updated to also make the ``else`` clause optional:: > > test: else_test ['if' or_test ['else' test]] | lambdef > else_test: or_test ['else' test] > > (We would avoid the apparent simplification to ``else_test ('if' > else_test)*`` > in order to make it easier to correctly preserve the semantics of normal > conditional expressions) > > Given that expanded definition, the following statements would be > functionally equivalent:: > > foo = calculate_default() if missing(foo) > foo = calculate_default() if foo is None else foo > > Just as the base proposal already makes the following equivalent:: > > foo = exists(foo) else calculate_default() > foo = foo if foo is not None else calculate_default() > > The ``if`` based circuit breaker form has the virtue of reading > significantly > better when used for conditional imperative commands like debug messages:: > > print(some_expensive_query()) if verbosity > 2 > > If we went down this path, then ``operator.true`` would need to be declared > as the nominal implicit circuit breaker when the condition didn't define > the > circuit breaker protocol itself (so the above example would produce > ``None`` > if the debugging message was printed, and ``False`` otherwise) > > The main objection to this expansion of the proposal is that it makes it a > more intrusive change that may potentially affect the behaviour of existing > code, while the main point in its favour is that allowing both ``if`` and > ``else`` as circuit breaking operators and also supporting the circuit > breaking > protocol for normal conditional expressions would be significantly more > self-consistent than special-casing a bare ``else`` as a stand-alone > operator. > > > Design Discussion > ================= > > Arbitrary sentinel objects > -------------------------- > > Unlike PEPs 505 and 531, this proposal readily handles custom sentinel > objects:: > > class defined(types.CircuitBreaker): > MISSING = object() > def __init__(self, value): > super().__init__(self, value is not self.MISSING, undefined) > > class undefined(types.CircuitBreaker): > def __init__(self, value): > super().__init__(self, value is defined.MISSING, defined) > > # Using the sentinel to check whether or not an argument was supplied > def my_func(arg=defined.MISSING): > arg = defined(arg) else calculate_default() > > > Implementation > ============== > > As with PEP 505, actual implementation has been deferred pending > in-principle > interest in the idea of making these changes - aside from the possible > syntactic ambiguity concerns covered by the grammer proposals above, the > implementation isn't really the hard part of these proposals, the hard part > is deciding whether or not this is a change where the long term benefits > for > new and existing Python users outweigh the short term costs involved in the > wider ecosystem (including developers of other implementations, language > curriculum developers, and authors of other Python related educational > material) adjusting to the change. > > ...TBD... > > > Acknowledgements > ================ > > Thanks go to Mark E. Haase for feedback on and contributions to earlier > drafts > of this proposal. However, his clear and exhaustive explanation of the > original > protocol design that modified the semantics of ``if-else`` conditional > expressions to use an underlying ``__then__``/``__else__`` protocol helped > convince me it was too complicated to keep, so this iteration contains > neither > that version of the protocol, nor Mark's explanation of it. > > > References > ========== > > .. [1] PEP 335 rejection notification > (http://mail.python.org/pipermail/python-dev/2012-March/117510.html) > > Copyright > ========= > > This document has been placed in the public domain under the terms of the > CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/ > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.pythonideas at wamp.us Sat Nov 12 12:01:00 2016 From: g.pythonideas at wamp.us (Gary Godfrey) Date: Sat, 12 Nov 2016 17:01:00 +0000 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' Message-ID: Hi, I looked around for a while but didn't see this proposed anywhere. I apologize if I missed an existing discussion. I do a fair amount of work with pandas and data munging. This means that I'm often doing things like: mydf = df[ ['field1', 'field2', 'field3' ] ] This is a little ugly, so if the list is long enough, I do: mydf=df[ 'field1 field2 field3'.split() ] This is a little more readable, but still a bit ugly. What I'm proposing here is: mydf = df[ w'field1 field2 field3' ] This would be identical in all ways (compile-time) to: mydf = df[ ('field1', 'field2', 'field3') ] This should work with all the python quote variations (w''', w""", etc). The only internal escapes are \\ indicating a \ and indicating a non-splitting space: songs = w'My\ Bloody\ Valentine Blue\ Suede\ Shoes' One question is whether to have w'' be a list or a tuple. I leaned slightly towards tuple because it's faster on internal loops: In [1]: %timeit a=('this','is','a','test') 100000000 loops, best of 3: 11.3 ns per loop In [2]: %timeit a=['this','is','a','test'] 10000000 loops, best of 3: 74.3 ns per loop However, I mostly see lists used in the data science community, so it's a little less convenient: other_fields = df.columns[-3:] new_columns = w'field1 field2' + other_fields # ERROR - can't concatenate list to tuple new_columns = list(w'field1 field2') + other_fields I honestly could go either way with lists or tuples. Other Languages: perl has the qw operator: @a = qw(field1 field2 field3); ruby has %w a=%w{field1 field2} Thanks for reading this far :-) Regards, Gary Godfrey Austin, TX -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 11:57:28 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 08:57:28 -0800 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: If recommend 'valued(foo)'. Without the final 'd' I think of "the value of foo" rather than "does foo have a value?" Obviously, the value of foo is just spelled 'foo' in Python, but it seems confusing. 'exists(foo)' is even more confusing since almost everyone will read it as "is foo defined?" I know you can't do that with a call in Python, but you can in lots of other languages. On Nov 12, 2016 8:50 AM, "Mark E. Haase" wrote: I like PEP-532. Given the opposition to non-Pythonic syntax like ?? (i.e. PEP-505), Nick's proposal offers a Pythonic alternative that is protocol based, more generalized, and uses built-ins and keywords to avoid punctuation. I agree with other posters that the terms "exists" and "missing" could lead developers to think that it tests for NameError. Maybe "value(foo) else bar"? I can't think of a better spelling for the inverse. Maybe the "if" syntax described in the PEP is better: "foo.bar if value(foo)". In that case, we wouldn't need an inverse to exists()/value()/whatever. I also wanted to mention a couple of unmentioned benefits of this PEP: 1. It is easier to Google a name. E.g., Google "c# ??" and you'll get nothing related to null coalescing in c#". ("C# question marks" does find the right content, however.) 2. Dismay over the meaning of foo?.bar.baz is much clearer when expressed as missing(foo) else foo.bar.baz -- it's very close to the ternary logic you'd write if you didn't have a circuit breaking operator: None if foo is None else foo.bar.baz. On Sat, Nov 5, 2016 at 5:50 AM, Nick Coghlan wrote: > Hi folks, > > As promised, here's a follow-up to the withdrawn PEP 531 that focuses > on coming up with a common protocol driven solution to conditional > evaluation of subexpressions that also addresses the element-wise > comparison chaining problem that Guido noted when rejecting PEP 335. > > I quite like how it came out, but see the "Risks & Concerns" section > for a discussion of an internal inconsistency it would introduce into > the language if accepted as currently written, and the potentially > far-reaching consequences actually resolving that inconsistency might > have on the way people write their Python code (if the PEP was > subsequently approved). > > Regards, > Nick. > > ================================= > PEP: 532 > Title: A circuit breaking operator and protocol > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 30-Oct-2016 > Python-Version: 3.7 > Post-History: 5-Nov-2016 > > Abstract > ======== > > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this > PEP > proposes the addition of a new protocol-driven circuit breaking operator to > Python that allows the left operand to decide whether or not the expression > should short circuit and return a result immediately, or else continue > on with evaluation of the right operand:: > > exists(foo) else bar > missing(foo) else foo.bar() > > These two expressions can be read as: > > * "the expression result is 'foo' if it exists, otherwise it is 'bar'" > * "the expression result is 'foo' if it is missing, otherwise it is > 'foo.bar()'" > > Execution of these expressions relies on a new circuit breaking protocol > that > implicitly avoids repeated evaluation of the left operand while letting > that operand fully control the result of the expression, regardless of > whether > it skips evaluation of the right operand or not:: > > _lhs = LHS > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) > > To properly support logical negation of circuit breakers, a new ``__not__`` > protocol method would also be introduced allowing objects to control > the result of ``not obj`` expressions. > > As shown in the basic example above, the PEP further proposes the addition > of > builtin ``exists`` and ``missing`` circuit breakers that provide > conditional > branching based on whether or not an object is ``None``, but return the > original object rather than the existence checking wrapper when the > expression > evaluation short circuits. > > In addition to being usable as simple boolean operators (e.g. as in > ``assert all(exists, items)`` or ``if any(missing, items):``), these > circuit > breakers will allow existence checking fallback operations (aka > None-coalescing > operations) to be written as:: > > value = exists(expr1) else exists(expr2) else expr3 > > and existence checking precondition operations (aka None-propagating > or None-severing operations) to be written as:: > > value = missing(obj) else obj.field.of.interest > value = missing(obj) else obj["field"]["of"]["interest"] > > A change to the definition of chained comparisons is also proposed, where > the comparison chaining will be updated to use the circuit breaking > operator > rather than the logical disjunction (``and``) operator if the left hand > comparison returns a circuit breaker as its result. > > While there are some practical complexities arising from the current > handling > of single-valued arrays in NumPy, this change should be sufficient to allow > elementwise chained comparison operations for matrices, where the result > is a matrix of boolean values, rather than tautologically returning > ``True`` > or raising ``ValueError``. > > > Relationship with other PEPs > ============================ > > This PEP is a direct successor to PEP 531, replacing the existence checking > protocol and the new ``?then`` and ``?else`` syntactic operators defined > there > with a single protocol driven ``else`` operator and adjustments to the > ``not`` > operator. The existence checking use cases are taken from that PEP. > > It is also a direct successor to PEP 335, which proposed the ability to > overload the ``and`` and ``or`` operators directly, with the ability to > overload the semantics of comparison chaining being one of the consequences > of that change. The proposal in this PEP to instead handle the element-wise > comparison use case by changing the semantic definition of comparison > chaining > is drawn from Guido's rejection of PEP 335. > > This PEP competes with the dedicated null-coalescing operator in PEP 505, > proposing that improved support for null-coalescing operations be offered > through a more general protocol-driven short circuiting operator and > related > builtins, rather than through a dedicated single-purpose operator. > > It doesn't compete with PEP 505's proposed shorthands for existence > checking > attribute access and subscripting, but instead offers an alternative > underlying > semantic framework for defining them: > > * ``EXPR?.attr`` would be syntactic sugar for ``missing(EXPR) else > EXPR.attr`` > * ``EXPR?[key]`` would be syntactic sugar for ``missing(EXPR) else > EXPR[key]`` > > In both cases, the dedicated syntactic form could be optimised to avoid > actually creating the circuit breaker instance. > > > Specification > ============= > > The circuit breaking operator (``else``) > ---------------------------------------- > > Circuit breaking expressions would be written using ``else`` as a new > binary > operator, akin to the existing ``and`` and ``or`` logical operators:: > > LHS else RHS > > Ignoring the hidden variable assignment, this is semantically equivalent > to:: > > _lhs = LHS > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) > > The key difference relative to the existing ``or`` operator is that the > value > determining which branch of the conditional expression gets executed *also* > gets a chance to postprocess the results of the expressions on each of the > branches. > > As part of the short-circuiting behaviour, interpreter implementations > are expected to access only the protocol method needed for the branch > that is actually executed, but it is still recommended that circuit > breaker authors that always return ``True`` or always return ``False`` from > ``__bool__`` explicitly raise ``NotImplementedError`` with a suitable > message from branch methods that are never expected to be executed (see the > comparison chaining use case in the Rationale section below for an example > of that). > > It is proposed that the ``else`` operator use a new precedence level that > binds > less tightly than the ``or`` operator by adjusting the relevant line in > Python's grammar from the current:: > > test: or_test ['if' or_test 'else' test] | lambdef > > to instead be:: > > test: else_test ['if' or_test 'else' test] | lambdef > else_test: or_test ['else' test] > > The definition of ``test_nocond`` would remain unchanged, so circuit > breaking expressions would require parentheses when used in the ``if`` > clause of comprehensions and generator expressions just as conditional > expressions themselves do. > > This grammar definition means precedence/associativity in the otherwise > ambiguous case of ``expr1 if cond else expr2 else epxr3`` resolves as > ``(expr1 if cond else expr2) else epxr3``. > > A guideline will also be added to PEP 8 to say "don't do that", as such a > construct will be inherently confusing for readers, regardless of how the > interpreter executes it. > > > Overloading logical inversion (``not``) > --------------------------------------- > > Any circuit breaker definition will have a logical inverse that is still a > circuit breaker, but inverts the answer as to whether or not to short > circuit > the expression evaluation. For example, the ``exists`` and ``missing`` > circuit > breakers proposed in this PEP are each other's logical inverse. > > A new protocol method, ``__not__(self)``, will be introduced to permit > circuit > breakers and other types to override ``not`` expressions to return their > logical inverse rather than a coerced boolean result. > > To preserve the semantics of existing language optimisations, ``__not__`` > implementations will be obliged to respect the following invariant:: > > assert not bool(obj) == bool(not obj) > > > Chained comparisons > ------------------- > > A chained comparison like ``0 < x < 10`` written as:: > > LEFT_BOUND left_op EXPR right_op RIGHT_BOUND > > is currently roughly semantically equivalent to:: > > _expr = EXPR > _lhs_result = LEFT_BOUND left_op _expr > _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) > > This PEP proposes that this be changed to explicitly check if the left > comparison returns a circuit breaker, and if so, use ``else`` rather than > ``and`` to implement the comparison chaining:: > > _expr = EXPR > _lhs_result = LEFT_BOUND left_op _expr > if hasattr(type(_lhs_result), "__then__"): > _expr_result = _lhs_result else (_expr right_op RIGHT_BOUND) > else: > _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND) > > This allows types like NumPy arrays to control the behaviour of chained > comparisons by returning circuit breakers from comparison operations. > > > Existence checking comparisons > ------------------------------ > > Two new builtins implementing the new protocol are proposed to encapsulate > the > notion of "existence checking": seeing if a value is ``None`` and either > falling back to an alternative value (an operation known as > "None-coalescing") > or passing it through as the result of the overall expression (an operation > known as "None-severing" or "None-propagating"). > > These builtins would be defined as follows:: > > class CircuitBreaker: > """Base circuit breaker type (available as types.CircuitBreaker)""" > def __init__(self, value, condition, inverse_type): > self.value = value > self._condition = condition > self._inverse_type = inverse_type > def __bool__(self): > return self._condition > def __not__(self): > return self._inverse_type(self.value) > def __then__(self): > return self.value > def __else__(self, other): > if other is self: > return self.value > return other > > class exists(types.CircuitBreaker): > """Circuit breaker for 'EXPR is not None' checks""" > def __init__(self, value): > super().__init__(value, value is not None, missing) > > class missing(types.CircuitBreaker): > """Circuit breaker for 'EXPR is None' checks""" > def __init__(self, value): > super().__init__(value, value is None, exists) > > Aside from changing the definition of ``__bool__`` to be based on > ``is not None`` rather than normal truth checking, the key characteristic > of > ``exists`` is that when it is used as a circuit breaker, it is *ephemeral*: > when it is told that short circuiting has taken place, it returns the > original > value, rather than the existence checking wrapper. > > ``missing`` is defined as the logically inverted counterpart of ``exists``: > ``not exists(obj)`` is semantically equivalent to ``missing(obj)``. > > The ``__else__`` implementations for both builtin circuit breakers are > defined > such that the wrapper will always be removed even if you explicitly pass > the > circuit breaker to both sides of the ``else`` expression:: > > breaker = exists(foo) > assert (breaker else breaker) is foo > breaker = missing(foo) > assert (breaker else breaker) is foo > > > Other conditional constructs > ---------------------------- > > No changes are proposed to if statements, while statements, conditional > expressions, comprehensions, or generator expressions, as the boolean > clauses > they contain are already used for control flow purposes. > > However, it's worth noting that while such proposals are outside the scope > of > this PEP, the circuit breaking protocol defined here would be sufficient to > support constructs like:: > > while exists(dynamic_query()) as result: > ... # Code using result > > and: > > if exists(re.search(pattern, text)) as match: > ... # Code using match > > Leaving the door open to such a future extension is the main reason for > recommending that circuit breaker implementations handle the ``self is > other`` > case in ``__else__`` implementations the same way as they handle the > short-circuiting behaviour in ``__then__``. > > > Style guide recommendations > --------------------------- > > The following additions to PEP 8 are proposed in relation to the new > features > introduced by this PEP: > > * In the absence of other considerations, prefer the use of the builtin > circuit breakers ``exists`` and ``missing`` over the corresponding > conditional expressions > > * Do not combine conditional expressions (``if-else``) and circuit breaking > expressions (the ``else`` operator) in a single expression - use one or > the > other depending on the situation, but not both. > > > Rationale > ========= > > Adding a new operator > --------------------- > > Similar to PEP 335, early drafts of this PEP focused on making the existing > ``and`` and ``or`` operators less rigid in their interpretation, rather > than > proposing new operators. However, this proved to be problematic for a few > reasons: > > * defining a shared protocol for both ``and`` and ``or`` was confusing, as > ``__then__`` was the short-circuiting outcome for ``or``, while > ``__else__`` > was the short-circuiting outcome for ``and`` > * the ``and`` and ``or`` operators have a long established and stable > meaning, > so readers would inevitably be surprised if their meaning now became > dependent on the type of the left operand. Even new users would be > confused > by this change due to 25+ years of teaching material that assumes the > current well-known semantics for these operators > * Python interpreter implementations, including CPython, have taken > advantage > of the existing semantics of ``and`` and ``or`` when defining runtime and > compile time optimisations, which would all need to be reviewed and > potentially discarded if the semantics of those operations changed > > Proposing a single new operator instead resolves all of those issues - > ``__then__`` always indicates short circuiting, ``__else__`` only indicates > "short circuiting" if the circuit breaker itself is also passed in as the > right operand, and the semantics of ``and`` and ``or`` remain entirely > unchanged. While the semantics of the unary ``not`` operator do change, the > invariant required of ``__not__`` implementations means that existing > expression optimisations in boolean contexts will remain valid. > > As a result of that design simplification, the new protocol and operator > would > even allow us to expose ``operator.true`` and ``operator.false`` > as circuit breaker definitions if we chose to do so:: > > class true(types.CircuitBreaker): > """Circuit breaker for 'bool(EXPR)' checks""" > def __init__(self, value): > super().__init__(value, bool(value), when_false) > > class false(types.CircuitBreaker): > """Circuit breaker for 'not bool(EXPR)' checks""" > def __init__(self, value): > super().__init__(value, not bool(value), when_true) > > Given those circuit breakers: > > * ``LHS or RHS`` would be roughly ``operator.true(LHS) else RHS`` > * ``LHS and RHS`` would be roughly ``operator.false(LHS) else RHS`` > > > Naming the operator and protocol > -------------------------------- > > The names "circuit breaking operator", "circuit breaking protocol" and > "circuit breaker" are all inspired by the phrase "short circuiting > operator": > the general language design term for operators that only conditionally > evaluate their right operand. > > The electrical analogy is that circuit breakers in Python detect and handle > short circuits in expressions before they trigger any exceptions similar > to the > way that circuit breakers detect and handle short circuits in electrical > systems before they damage any equipment or harm any humans. > > The Python level analogy is that just as a ``break`` statement lets you > terminate a loop before it reaches its natural conclusion, a circuit > breaking > expression lets you terminate evaluation of the expression and produce a > result > immediately. > > > Using an existing keyword > ------------------------- > > Using an existing keyword has the benefit of allowing the new expression to > be introduced without a ``__future__`` statement. > > ``else`` is semantically appropriate for the proposed new protocol, and the > only syntactic ambiguity introduced arises when the new operator is > combined > with the explicit ``if-else`` conditional expression syntax. > > > Element-wise chained comparisons > -------------------------------- > > In ultimately rejecting PEP 335, Guido van Rossum noted [1_]: > > The NumPy folks brought up a somewhat separate issue: for them, > the most common use case is chained comparisons (e.g. A < B < C). > > To understand this observation, we first need to look at how comparisons > work > with NumPy arrays:: > > >>> import numpy as np > >>> increasing = np.arange(5) > >>> increasing > array([0, 1, 2, 3, 4]) > >>> decreasing = np.arange(4, -1, -1) > >>> decreasing > array([4, 3, 2, 1, 0]) > >>> increasing < decreasing > array([ True, True, False, False, False], dtype=bool) > > Here we see that NumPy array comparisons are element-wise by default, > comparing > each element in the lefthand array to the corresponding element in the > righthand > array, and producing a matrix of boolean results. > > If either side of the comparison is a scalar value, then it is broadcast > across > the array and compared to each individual element:: > > >>> 0 < increasing > array([False, True, True, True, True], dtype=bool) > >>> increasing < 4 > array([ True, True, True, True, False], dtype=bool) > > However, this broadcasting idiom breaks down if we attempt to use chained > comparisons:: > > >>> 0 < increasing < 4 > Traceback (most recent call last): > File "", line 1, in > ValueError: The truth value of an array with more than one element > is ambiguous. Use a.any() or a.all() > > The problem is that internally, Python implicitly expands this chained > comparison into the form:: > > >>> 0 < increasing and increasing < 4 > Traceback (most recent call last): > File "", line 1, in > ValueError: The truth value of an array with more than one element > is ambiguous. Use a.any() or a.all() > > And NumPy only permits implicit coercion to a boolean value for > single-element > arrays where ``a.any()`` and ``a.all()`` can be assured of having the same > result:: > > >>> np.array([False]) and np.array([False]) > array([False], dtype=bool) > >>> np.array([False]) and np.array([True]) > array([False], dtype=bool) > >>> np.array([True]) and np.array([False]) > array([False], dtype=bool) > >>> np.array([True]) and np.array([True]) > array([ True], dtype=bool) > > The proposal in this PEP would allow this situation to be changed by > updating > the definition of element-wise comparison operations in NumPy to return a > dedicated subclass that implements the new circuit breaking protocol and > also > changes the result array's interpretation in a boolean context to always > return ``False`` and hence never trigger the short-circuiting behaviour:: > > class ComparisonResultArray(np.ndarray): > def __bool__(self): > return False > def _raise_NotImplementedError(self): > msg = ("Comparison array truth values are ambiguous outside " > "chained comparisons. Use a.any() or a.all()") > raise NotImplementedError(msg) > def __not__(self): > self._raise_NotImplementedError() > def __then__(self): > self._raise_NotImplementedError() > def __else__(self, other): > return np.logical_and(self, other.view(ComparisonResultArray)) > > With this change, the chained comparison example above would be able to > return:: > > >>> 0 < increasing < 4 > ComparisonResultArray([ False, True, True, True, False], dtype=bool) > > > Existence checking expressions > ------------------------------ > > An increasingly common requirement in modern software development is the > need > to work with "semi-structured data": data where the structure of the data > is > known in advance, but pieces of it may be missing at runtime, and the > software > manipulating that data is expected to degrade gracefully (e.g. by omitting > results that depend on the missing data) rather than failing outright. > > Some particularly common cases where this issue arises are: > > * handling optional application configuration settings and function > parameters > * handling external service failures in distributed systems > * handling data sets that include some partial records > > At the moment, writing such software in Python can be genuinely awkward, as > your code ends up littered with expressions like: > > * ``value1 = expr1.field.of.interest if expr1 is not None else None`` > * ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else > None`` > * ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not > None else expr5`` > > PEP 531 goes into more detail on some of the challenges of working with > this > kind of data, particularly in data transformation pipelines where dealing > with > potentially missing content is the norm rather than the exception. > > The combined impact of the proposals in this PEP is to allow the above > sample > expressions to instead be written as: > > * ``value1 = missing(expr1) else expr1.field.of.interest`` > * ``value2 = missing(expr2) else expr2.["field"]["of"]["interest"]`` > * ``value3 = exists(expr3) else exists(expr4) else expr5`` > > In these forms, significantly more of the text presented to the reader is > immediately relevant to the question "What does this code do?", while the > boilerplate code to handle missing data by passing it through to the output > or falling back to an alternative input, has shrunk to two uses of the new > ``missing`` builtin, and two uses of the new ``exists`` builtin. > > In the first two examples, the 31 character boilerplate suffix > ``if exprN is not None else None`` (minimally 27 characters for a single > letter > variable name) has been replaced by a 19 character ``missing(expr1) else`` > prefix (minimally 15 characters with a single letter variable name), > markedly > improving the signal-to-pattern-noise ratio of the lines (especially if it > encourages the use of more meaningful variable and field names rather than > making them shorter purely for the sake of expression brevity). The > additional > syntactic sugar proposals in PEP 505 would further reduce this boilerplate > to > a single ``?`` character that also eliminated the repetition of the > expession > being checked for existence. > > In the last example, not only are two instances of the 21 character > boilerplate, > `` if exprN is not None`` (minimally 17 characters) replaced with the > 8 character function call ``exists()``, but that function call is placed > directly around the original expression, eliminating the need to duplicate > it > in the conditional existence check. > > > Risks and concerns > ================== > > This PEP has been designed specifically to address the risks and concerns > raised when discussing PEPs 335, 505 and 531. > > * it defines a new operator and adjusts the definition of chained > comparison > rather than impacting the existing ``and`` and ``or`` operators > * the changes to the ``not`` unary operator are defined in such a way that > control flow optimisations based on the existing semantics remain valid > * rather than the cryptic ``??``, it uses ``else`` as the operator keyword > in > exactly the same sense as it is already used in conditional expressions > * it defines a general purpose short-circuiting binary operator that can > even > be used to express the existing semantics of ``and`` and ``or`` rather > than > focusing solely and inflexibly on existence checking > * it names the proposed builtins in such a way that they provide a strong > mnemonic hint as to when the expression containing them will > short-circuit > and skip evaluating the right operand > > > Possible confusion with conditional expressions > ----------------------------------------------- > > The proposal in this PEP is essentially for an "implied ``if``" where if > you > omit the ``if`` clause from a conditional expression, you invoke the > circuit > breaking protocol instead. That is:: > > exists(foo) else calculate_default() > > invokes the new protocol, but:: > > foo.field.of.interest if exists(foo) else calculate_default() > > bypasses it entirely, *including* the non-short-circuiting ``__else__`` > method. > > This mostly wouldn't be a problem for the proposed ``types.CircuitBreaker`` > implementation (and hence the ``exists`` and ``missing`` builtins), as the > only purpose the extended protocol serves in that case is to remove the > wrapper in the short-circuiting case - the ``__else__`` method passes the > right operand through unchanged. > > However, this discrepancy could potentially be eliminated entirely by also > updating conditional expressions to use the circuit breaking protocol if > the condition defines those methods. In that case, ``__then__`` would need > to be updated to accept the left operand as a parameter, with > short-circuiting > indicated by passing in the circuit breaker itself:: > > class CircuitBreaker: > """Base circuit breaker type (available as types.CircuitBreaker)""" > def __init__(self, value, condition, inverse_type): > self.value = value > self._condition = condition > self._inverse_type = inverse_type > def __bool__(self): > return self._condition > def __not__(self): > return self._inverse_type(self.value) > def __then__(self, other): > if other is not self: > return other > return self.value # Short-circuit, remove the wrapper > def __else__(self, other): > if other is not self: > return other > return self.value # Short-circuit, remove the wrapper > > With this symmetric protocol, the definition of conditional expressions > could be updated to also make the ``else`` clause optional:: > > test: else_test ['if' or_test ['else' test]] | lambdef > else_test: or_test ['else' test] > > (We would avoid the apparent simplification to ``else_test ('if' > else_test)*`` > in order to make it easier to correctly preserve the semantics of normal > conditional expressions) > > Given that expanded definition, the following statements would be > functionally equivalent:: > > foo = calculate_default() if missing(foo) > foo = calculate_default() if foo is None else foo > > Just as the base proposal already makes the following equivalent:: > > foo = exists(foo) else calculate_default() > foo = foo if foo is not None else calculate_default() > > The ``if`` based circuit breaker form has the virtue of reading > significantly > better when used for conditional imperative commands like debug messages:: > > print(some_expensive_query()) if verbosity > 2 > > If we went down this path, then ``operator.true`` would need to be declared > as the nominal implicit circuit breaker when the condition didn't define > the > circuit breaker protocol itself (so the above example would produce > ``None`` > if the debugging message was printed, and ``False`` otherwise) > > The main objection to this expansion of the proposal is that it makes it a > more intrusive change that may potentially affect the behaviour of existing > code, while the main point in its favour is that allowing both ``if`` and > ``else`` as circuit breaking operators and also supporting the circuit > breaking > protocol for normal conditional expressions would be significantly more > self-consistent than special-casing a bare ``else`` as a stand-alone > operator. > > > Design Discussion > ================= > > Arbitrary sentinel objects > -------------------------- > > Unlike PEPs 505 and 531, this proposal readily handles custom sentinel > objects:: > > class defined(types.CircuitBreaker): > MISSING = object() > def __init__(self, value): > super().__init__(self, value is not self.MISSING, undefined) > > class undefined(types.CircuitBreaker): > def __init__(self, value): > super().__init__(self, value is defined.MISSING, defined) > > # Using the sentinel to check whether or not an argument was supplied > def my_func(arg=defined.MISSING): > arg = defined(arg) else calculate_default() > > > Implementation > ============== > > As with PEP 505, actual implementation has been deferred pending > in-principle > interest in the idea of making these changes - aside from the possible > syntactic ambiguity concerns covered by the grammer proposals above, the > implementation isn't really the hard part of these proposals, the hard part > is deciding whether or not this is a change where the long term benefits > for > new and existing Python users outweigh the short term costs involved in the > wider ecosystem (including developers of other implementations, language > curriculum developers, and authors of other Python related educational > material) adjusting to the change. > > ...TBD... > > > Acknowledgements > ================ > > Thanks go to Mark E. Haase for feedback on and contributions to earlier > drafts > of this proposal. However, his clear and exhaustive explanation of the > original > protocol design that modified the semantics of ``if-else`` conditional > expressions to use an underlying ``__then__``/``__else__`` protocol helped > convince me it was too complicated to keep, so this iteration contains > neither > that version of the protocol, nor Mark's explanation of it. > > > References > ========== > > .. [1] PEP 335 rejection notification > (http://mail.python.org/pipermail/python-dev/2012-March/117510.html) > > Copyright > ========= > > This document has been placed in the public domain under the terms of the > CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/ > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 12:33:10 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 09:33:10 -0800 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: References: Message-ID: Just spend the extra two characters to do this with existing syntax: w('field1 field2 field3'). Implementation of the w() function is trivial. On Nov 12, 2016 9:04 AM, "Gary Godfrey" wrote: > Hi, > > I looked around for a while but didn't see this proposed anywhere. I > apologize if I missed an existing discussion. > > I do a fair amount of work with pandas and data munging. This means that > I'm often doing things like: > > mydf = df[ ['field1', 'field2', 'field3' ] ] > > This is a little ugly, so if the list is long enough, I do: > > mydf=df[ 'field1 field2 field3'.split() ] > > This is a little more readable, but still a bit ugly. What I'm proposing > here is: > > mydf = df[ w'field1 field2 field3' ] > > This would be identical in all ways (compile-time) to: > > mydf = df[ ('field1', 'field2', 'field3') ] > > This should work with all the python quote variations (w''', w""", etc). > The only internal escapes are \\ indicating a \ and > indicating a non-splitting space: > > songs = w'My\ Bloody\ Valentine Blue\ Suede\ Shoes' > > One question is whether to have w'' be a list or a tuple. I leaned > slightly towards tuple because it's faster on internal loops: > > In [1]: %timeit a=('this','is','a','test') > 100000000 loops, best of 3: 11.3 ns per loop > > In [2]: %timeit a=['this','is','a','test'] > 10000000 loops, best of 3: 74.3 ns per loop > > However, I mostly see lists used in the data science community, so it's a > little less convenient: > > other_fields = df.columns[-3:] > new_columns = w'field1 field2' + other_fields > # ERROR - can't concatenate list to tuple > new_columns = list(w'field1 field2') + other_fields > > I honestly could go either way with lists or tuples. > > Other Languages: > > perl has the qw operator: > > @a = qw(field1 field2 field3); > > ruby has %w > > a=%w{field1 field2} > > Thanks for reading this far :-) > > Regards, > Gary Godfrey > Austin, TX > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Nov 12 13:05:59 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 Nov 2016 05:05:59 +1100 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: References: Message-ID: <20161112180556.GP3365@ando.pearwood.info> On Sat, Nov 12, 2016 at 05:01:00PM +0000, Gary Godfrey wrote: > I do a fair amount of work with pandas and data munging. This means that > I'm often doing things like: > > mydf = df[ ['field1', 'field2', 'field3' ] ] > > This is a little ugly, so if the list is long enough, I do: > > mydf=df[ 'field1 field2 field3'.split() ] I consider the need for that to indicate a possibly poor design of pandas. Unless there is a good reason not to, I believe that any function that requires a list of strings should also accept a single space-delimited string instead. Especially if the strings are intended as names or labels. So that: func(['fe', 'fi', 'fo', 'fum']) and func('fe fi fo fum') should be treated the same way. Of course, it may be that pandas has a good reason for not supporting that. But in general, we don't change the language to make up for deficiencies in third-party library functionality. > This is a little more readable, but still a bit ugly. I don't agree that its ugly. I think that 'fe fi fo fum'.split() is nicely explicit about what it is doing. It's also a candidate for compile-time optimization since the argument is a literal. > What I'm proposing here is: > > mydf = df[ w'field1 field2 field3' ] > > This would be identical in all ways (compile-time) to: > > mydf = df[ ('field1', 'field2', 'field3') ] Are your field names usually constants known when you write the script? I would have thought they'd more often be variables that you read from your data. > This should work with all the python quote variations (w''', w""", etc). > The only internal escapes are \\ indicating a \ and > indicating a non-splitting space: So not only do we have to learn yet another special kind of string: - unicode strings - byte strings - raw strings (either unicode or bytes) - f-strings - and now w-strings but this one has different escaping rules from the others. I expect that there will be a huge number of confused questions about why people cannot use standard escapes in their "word" strings. > songs = w'My\ Bloody\ Valentine Blue\ Suede\ Shoes' I think that escaping spaces like that will be an attractive nuisance. I had to read your example three times before I noticed that the space between Valentine and Blue was not escaped. I would prefer a simple, straight-forward rule: it unconditionally splits on whitespace. If you need to include non-splitting spaces, use a proper non-breaking space \u00A0, or split the words into a tuple by hand, like you're doing now. I don't think it is worth complicating the feature to support non-splitting spaces. (Hmmm... I see that str.split() currently splits on non-breaking spaces. That feels wrong to me: although the NBSP character is considered whitespace, its whole purpose is to avoid splitting.) > Other Languages: > > perl has the qw operator: > > @a = qw(field1 field2 field3); > > ruby has %w > > a=%w{field1 field2} The fact that other languages do something like this is a (weak) point in its favour. But I see that there are a few questions on Stackoverflow asking what %w means, how it is different from %W, etc. For example: http://stackoverflow.com/questions/1274675/what-does-warray-mean http://stackoverflow.com/questions/690794/ruby-arrays-w-vs-w and I notice this comment from the second link: "%w" is my usual retort to people who get a little too cocky about the readability of Ruby. Works every time. That's a point against this proposal: the feature seems to be a bit puzzling to users in languages that implement it (at least Ruby). I'm rather luke-warm on this proposal, although I might be convinced to support it if: - w'...' unconditionally split on any whitespace (possibly excluding NBSP); - and normal escapes worked. Even then I'm not really convinced this needs to be a language feature. -- Steve From mertz at gnosis.cx Sat Nov 12 14:10:13 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 11:10:13 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: <20161112092624.GN3365@ando.pearwood.info> References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: +100 I like this idea of giving `slice` a metaclass that defines a `.__getitem__()` allowing us to construct slices on the slice type itself. FWIW, this is exactly what pandas.IndexSlice does. E.g., from http://pandas.pydata.org/pandas-docs/stable/advanced.html: In [51]: dfmi.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:] In [52]: idx = pd.IndexSlice In [53]: dfmi.loc[idx[:,:,['C1','C3']],idx[:,'foo']] This is one of those nifty things that's buried in Pandas but not well documented. I'd rather spell the above simply as: dfmi.loc[slice[:,:,['C1','C3']], slice[:,'foo']] I like the change proposed to `str(slice(10))` also... and it would be way better if `slice[:10]` were actual "syntax." In fact, in that case it could even be the repr(). Note: Notwithstanding my scare quotes, Steven isn't actually asking for new syntax. "slice" is already a name, and names can already be followed by square brackets. He's just asking for a new method on a metaclass. On Sat, Nov 12, 2016 at 1:26 AM, Steven D'Aprano wrote: > On Thu, Oct 06, 2016 at 04:19:17PM -0700, Neil Girdhar wrote: > > Currently str(slice(10)) returns "slice(None, 10, None)" > > > > If the start and step are None, consider not emitting them. Similarly > > slice(None) is rendered slice(None, None, None). > > > > When you're printing a lot of slices, it's a lot of extra noise. > > I have an alternative suggestion. Wouldn't it be nice if slice objects > looked something like the usual slice syntax? > > If you think the answer is No, then you'll hate my suggestion :-) > > Let's keep the current repr() of slice objects as they are, using the > full function-call syntax complete with all three arguments show > explicitly: > > repr(slice(None, None, None)) => "slice(None, None, None)" > > But let's make str() of a slice more suggestive of actual slicing, and > as a bonus, make slices easier to create too. > > str(slice(None, None, None)) => "slice[:]" > > Let the slice type itself be sliceable, as an alternate constuctor: > > slice[:] => returns slice(None) > slice[start:] => returns slice(start, None) > slice[:end] => returns slice(None, end) > slice[start::step] => returns slice(start, None, step) > > and so forth. (This probably would require changing the type of slice to > a new metaclass.) > > And then have str() return the compact slice syntax. > > At worst, the compact slice syntax is one character longer than the > optimal function syntax: > > # proposed slice str() > slice[:7] # 9 characters > > # proposed compact str() > slice(7) # 8 characters > > # current str() > slice(None, 7, None) # 20 characters > > > but it will be more compact more often: > > slice[1:] # 9 characters > > versus: > > slice(1, None) # 14 characters > slice(None, 1, None) # 20 characters > > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Sat Nov 12 14:20:18 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sat, 12 Nov 2016 19:20:18 +0000 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 9:10 PM David Mertz wrote: > > dfmi.loc[slice[:,:,['C1','C3']], slice[:,'foo']] > > > I like the change proposed to `str(slice(10))` also... and it would be way > better if `slice[:10]` were actual "syntax." In fact, in that case it > could even be the repr(). > > Indexing operator for classes already has a meaning, for generic types. It is a possibility that slice will become a generic type (see here: https://github.com/python/mypy/issues/2410#issuecomment-258898836) and this syntax will make it either impossible or require Slice[] to be different from slice[] in a potentially confusing way. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.pythonideas at wamp.us Sat Nov 12 13:20:33 2016 From: g.pythonideas at wamp.us (Gary Godfrey) Date: Sat, 12 Nov 2016 18:20:33 +0000 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: References: Message-ID: On Sat, Nov 12, 2016 at 11:33 AM David Mertz wrote: > Just spend the extra two characters to do this with existing syntax: > w('field1 field2 field3'). Implementation of the w() function is trivial. > > I've done that as well and see that done. Sometimes with s('field1 field2'), sometimes with other characters. Sometimes with two letter functions. Part of what I'd like to do here is make it so there's one way that most people do this. If you look through data analytics books and examples, the default is ['field1', 'field2' 'field3']. That's a lot of little characters to type correctly and it's error prone (for instance, did you immediately notice that I was missing a "," in that example? Python will silently make that ['field1','field2field3']). Cheers, Gary -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 14:23:59 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 11:23:59 -0800 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: <20161112180556.GP3365@ando.pearwood.info> References: <20161112180556.GP3365@ando.pearwood.info> Message-ID: > > > mydf = df[ ['field1', 'field2', 'field3' ] ] > > mydf=df[ 'field1 field2 field3'.split() ] > > I consider the need for that to indicate a possibly poor design of > pandas. Unless there is a good reason not to, I believe that any > function that requires a list of strings should also accept a single > space-delimited string instead. > Of course, it may be that pandas has a good reason for not supporting > that. But in general, we don't change the language to make up for > deficiencies in third-party library functionality. > Yes... Pandas has a very good reason. And the general principle is wrong too. A list of strings can contain strings with spaces in them. It's only in rare cases like collections.namedtuple where the whole point of strings is to be valid Python identifiers that you can rule out spaces being inside them. E.g.: >>> df = pd.DataFrame({"Quant":[1,2,3], "Approx Price":[4,5,6], "Weight":[7,8,9]}) >>> df[['Quant','Approx Price']] Quant Approx Price 0 1 4 1 2 5 2 3 6 >>> df['Quant Approx Price'.split()]] ... KeyError: "['Approx' 'Price'] not in index" The hypothetical w-string would have the same ambiguity with w'Quant Approx Price'. Even with namedtuple, you sometimes have to massage strings to get rid of spaces (and other stray punctuation), e.g.: >>> from collections import namedtuple >>> Item = namedtuple('Item', 'Quant Approx_Price Weight') >>> Item(4,5,6) Item(Quant=4, Approx_Price=5, Weight=6) I use namedtuple fairly often dynamically, for example to read a novel CSV file or query a novel SQL table. When I do so, I need to check the "columns" for being valid identifiers, and then massage them to make them so. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.pythonideas at wamp.us Sat Nov 12 14:00:12 2016 From: g.pythonideas at wamp.us (Gary Godfrey) Date: Sat, 12 Nov 2016 19:00:12 +0000 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: <20161112180556.GP3365@ando.pearwood.info> References: <20161112180556.GP3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 12:06 PM Steven D'Aprano wrote: > I consider the need for that to indicate a possibly poor design of > pandas. Unless there is a good reason not to, I believe that any > function that requires a list of strings should also accept a single > space-delimited string instead. Especially if the strings are intended > as names or labels. So that: > > func(['fe', 'fi', 'fo', 'fum']) > > and > > func('fe fi fo fum') > > should be treated the same way. > They don't because df[ 'Column Name'] is a valid way to get a single column worth of data when the column name contains spaces (not encouraged, but it is valid). > > mydf = df[ ('field1', 'field2', 'field3') ] > > Are your field names usually constants known when you write the script? > Yes. All the time. When I'm on the side of creating APIs for data analysts to use, I think of the columns abstractly. When they're writing scripts to analyze data, it's all very explicit and in the domain of the data. Things like: df [df.age > 10] adf = df.pivot_table( ['runid','block'] ) Are common and the "right" way to do things in the problem domain. > So not only do we have to learn yet another special kind of string: > > - unicode strings > - byte strings > - raw strings (either unicode or bytes) > - f-strings > - and now w-strings > Very valid point. I also was considering (and rejected) a 'wb' for tuple of bytes. > I would prefer a simple, straight-forward rule: it unconditionally > splits on whitespace. If you need to include non-splitting spaces, use a > proper non-breaking space \u00A0, or split the words into a tuple by > hand, like you're doing now. I don't think it is worth complicating the > feature to support non-splitting spaces. > You're right there. If there are spaces in the columns, make it explicit and don't use the w''. I withdraw the "feature". And I think you're right that all the existing escape rules should work in the same way they do for regular unicode strings (don't go the raw strings route). Basically, w'foo bar' == tuple('foo bar'.split()) > The fact that other languages do something like this is a (weak) point > in its favour. But I see that there are a few questions on Stackoverflow > asking what %w > means, how it is different from %W, etc. For example: > > http://stackoverflow.com/questions/1274675/what-does-warray-mean > > http://stackoverflow.com/questions/690794/ruby-arrays-w-vs-w > > Well, I'd lean towards not having a W'fields' that does something funky :-). But your point is well taken. > ... > I'm rather luke-warm on this proposal, although I might be convinced to > support it if: > > - w'...' unconditionally split on any whitespace (possibly > excluding NBSP); > > - and normal escapes worked. > > Even then I'm not really convinced this needs to be a language feature. > > I'm realizing that a lot of the reason that I'm seeing this a lot is that it seems to be particular issue to using python for data science. In some ways, they're pushing the language a bit beyond what it's designed to do (the df[ (df.age > 10) & (df.gender=="F")] idiom is amazing and troubling). Since I'm doing a lot of this, these little language issues loom a bit larger than they would with "normal" programming. Thanks for responding. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 14:27:55 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 11:27:55 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: I thought of the use of `.__getitem__()` in metaclasses in the typing module. I feel like this use is more natural and more useful than that. Should we someday need a slice generic type for PEP 484, the spelling would naturally be `Slice[T]` instead, in my mind. But `slice[1:10,2]` should be a constructor for a concrete slice object. On Sat, Nov 12, 2016 at 11:20 AM, ????? wrote: > On Sat, Nov 12, 2016 at 9:10 PM David Mertz wrote: > >> >> dfmi.loc[slice[:,:,['C1','C3']], slice[:,'foo']] >> >> >> I like the change proposed to `str(slice(10))` also... and it would be >> way better if `slice[:10]` were actual "syntax." In fact, in that case it >> could even be the repr(). >> >> > Indexing operator for classes already has a meaning, for generic types. It > is a possibility that slice will become a generic type (see here: > https://github.com/python/mypy/issues/2410#issuecomment-258898836) and > this syntax will make it either impossible or require Slice[] to be > different from slice[] in a potentially confusing way. > > Elazar > > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Sat Nov 12 14:41:24 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 12 Nov 2016 20:41:24 +0100 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On 12 November 2016 at 20:27, David Mertz wrote: > I thought of the use of `.__getitem__()` in metaclasses in the typing > module. I feel like this use is more natural and more useful than that. > Should we someday need a slice generic type for PEP 484, the spelling would > naturally be `Slice[T]` instead, in my mind. But `slice[1:10,2]` should be > a constructor for a concrete slice object. > Slice[T] vs slice[::-1] is coherent with what we have now for List[T] vs list, etc. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sat Nov 12 15:08:00 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 12 Nov 2016 20:08:00 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) Message-ID: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Hello, I would like to propose some reversed assignment operators. Python already has some assignment operators: a += c is the same as a = a + c a -= c is the same as a = a - c a *= c is the same as a = a * c a /= c is the same as a = a / c a //= c is the same as a = a // c a %= c is the same as a = a % c a **= c is the same as a = a ** c What I would like to propose is the creation of the reverse: a =+ c is the same as a = c + a a =- c is the same as a = c - a a =* c is the same as a = c * a a =/ c is the same as a = c / a a =// c is the same as a = c // a a =% c is the same as a = c % a a =** c is the same as a = c ** a For addition (+= and =+), multiplication (*= and =*) and subtraction (-= and =-) of numbers it would give the same result, but addition (+= and =+) and multiplication (*= and =*) when used with strings return a different result. All the other operations (=/, =%, =** and =//) return a different result. I think it is simple and easy to understand, therefore pythonic. :) Best regards, JM From elazarg at gmail.com Sat Nov 12 15:15:19 2016 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Sat, 12 Nov 2016 20:15:19 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: On Sat, Nov 12, 2016 at 10:08 PM Jo?o Matos wrote: > > What I would like to propose is the creation of the reverse: > a =+ c is the same as a = c + a > a =+5 already means a becomes 5 > a =- c is the same as a = c - a > a =-5 already means a becomes -5 Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcrmatos at gmail.com Sat Nov 12 15:18:27 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 12 Nov 2016 20:18:27 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: Hello, What I propose is not a = +5 (see the space between the = and + sign) but a =+ 5 (no space between the = and + sign, and there is a space between + and 5) Exactly the same notation (only reversed) as the current assignment operators. Best regards, JM On 12-11-2016 20:15, ????? wrote: > > On Sat, Nov 12, 2016 at 10:08 PM Jo?o Matos > wrote: > > > What I would like to propose is the creation of the reverse: > a =+ c is the same as a = c + a > > a =+5 already means a becomes 5 > > a =- c is the same as a = c - a > > a =-5 already means a becomes -5 > > Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Sat Nov 12 15:19:03 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Sat, 12 Nov 2016 18:19:03 -0200 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: <24845c28-755b-5ad4-34be-9cb7745106db@gmail.com> On 2016-11-12 18:08, Jo?o Matos wrote: > a =- c is the same as a = c - a This would break compatibility. a =- c is a = -c -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From brenbarn at brenbarn.net Sat Nov 12 15:22:20 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sat, 12 Nov 2016 12:22:20 -0800 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: <582779FC.1030805@brenbarn.net> On 2016-11-12 12:18, Jo?o Matos wrote:> Hello, > > What I propose is not > a = +5 (see the space between the = and + sign) > but > a =+ 5 (no space between the = and + sign, and there is a space between > + and 5) > > Exactly the same notation (only reversed) as the current assignment > operators. The thing is that a =+ 5 is already valid syntax, and means the same as a = +5. You don't need spaces around operators in Python. So your proposal would have the change the behavior of existing syntax, which pretty much makes it a nonstarter. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From jcrmatos at gmail.com Sat Nov 12 15:23:39 2016 From: jcrmatos at gmail.com (=?UTF-8?Q?Jo=c3=a3o_Matos?=) Date: Sat, 12 Nov 2016 20:23:39 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <24845c28-755b-5ad4-34be-9cb7745106db@gmail.com> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <24845c28-755b-5ad4-34be-9cb7745106db@gmail.com> Message-ID: Hello, I understand that, but in my view a =- c and a = -c should not be same. Maybe an ideia for Python 4. Best regards, JM On 12-11-2016 20:19, Bernardo Sulzbach wrote: > On 2016-11-12 18:08, Jo?o Matos wrote: >> a =- c is the same as a = c - a > > This would break compatibility. > > a =- c > > is > > a = -c > > From levkivskyi at gmail.com Sat Nov 12 15:37:32 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 12 Nov 2016 21:37:32 +0100 Subject: [Python-ideas] Built-in function to run coroutines Message-ID: Hi, async/await syntax is a very nice recent feature, but there is something that I miss for coroutines defined with async def, as compared to generators. Coroutines represent an interesting mental model that goes beyond only asynchronous IO, so that I play with them in REPL often. But there is no built-in function to actually run a coroutine, so that typically I use something like: >>> def run(coro): ... try: ... coro.send(None) ... except StopIteration as e: ... return e.value >>> async def f(): ... return 42 >>> run(f()) 42 There is a simple yet useful function for interactive play with generators - ``next``, but not for coroutines. There is an option to do: >>> import asyncio >>> loop = asyncio.get_event_loop() >>> loop.run_until_complete(f()) 42 But this feels a bit redundant for an interactive play. I would propose to add something like an above described ``run`` function to built-ins. Yes, I know, there is a very high bar for adding a built-in function, but I believe such a function will help to promote async/await to a wider community (especially to novices). -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Nov 12 15:45:58 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 12 Nov 2016 20:45:58 +0000 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: References: Message-ID: On other hand having builtin for making toy examples in interactive mode looks very redundant. On Sat, Nov 12, 2016 at 10:38 PM Ivan Levkivskyi wrote: > Hi, > > async/await syntax is a very nice recent feature, but there is something > that I miss for coroutines defined with async def, as compared to > generators. Coroutines represent an interesting mental model that goes > beyond only asynchronous IO, so that I play with them in REPL often. But > there is no built-in function to actually run a coroutine, so that > typically I use something like: > > >>> def run(coro): > ... try: > ... coro.send(None) > ... except StopIteration as e: > ... return e.value > > >>> async def f(): > ... return 42 > > >>> run(f()) > 42 > > There is a simple yet useful function for interactive play with generators > - ``next``, but not for coroutines. There is an option to do: > > >>> import asyncio > >>> loop = asyncio.get_event_loop() > >>> loop.run_until_complete(f()) > 42 > > But this feels a bit redundant for an interactive play. I would propose to > add something like an above described ``run`` function to built-ins. > > Yes, I know, there is a very high bar for adding a built-in function, but > I believe such a function will help to promote async/await to a wider > community (especially to novices). > > -- > Ivan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 12 16:24:20 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 12 Nov 2016 13:24:20 -0800 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: References: Message-ID: I think there's a plan to add a run() function to asyncio, which would be something akin to def run(coro): return get_event_loop().run_until_complete(coro) (but perhaps with better cleanup). Then you could start with `from asyncio import run` and from then on you'd have your handy little function. When using standard Python you can even put such handy imports in your $PYTHONSTARTUP file. Presumably with Jupyter there are other, more powerful mechanisms. --Guido On Sat, Nov 12, 2016 at 12:45 PM, Andrew Svetlov wrote: > On other hand having builtin for making toy examples in interactive mode > looks very redundant. > > On Sat, Nov 12, 2016 at 10:38 PM Ivan Levkivskyi > wrote: > >> Hi, >> >> async/await syntax is a very nice recent feature, but there is something >> that I miss for coroutines defined with async def, as compared to >> generators. Coroutines represent an interesting mental model that goes >> beyond only asynchronous IO, so that I play with them in REPL often. But >> there is no built-in function to actually run a coroutine, so that >> typically I use something like: >> >> >>> def run(coro): >> ... try: >> ... coro.send(None) >> ... except StopIteration as e: >> ... return e.value >> >> >>> async def f(): >> ... return 42 >> >> >>> run(f()) >> 42 >> >> There is a simple yet useful function for interactive play with >> generators - ``next``, but not for coroutines. There is an option to do: >> >> >>> import asyncio >> >>> loop = asyncio.get_event_loop() >> >>> loop.run_until_complete(f()) >> 42 >> >> But this feels a bit redundant for an interactive play. I would propose >> to add something like an above described ``run`` function to built-ins. >> >> Yes, I know, there is a very high bar for adding a built-in function, but >> I believe such a function will help to promote async/await to a wider >> community (especially to novices). >> >> -- >> Ivan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > Thanks, > Andrew Svetlov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 12 16:32:48 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 12 Nov 2016 13:32:48 -0800 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <24845c28-755b-5ad4-34be-9cb7745106db@gmail.com> Message-ID: I think a stronger argument to reject this (even for Python 4, and even if we could somehow get over the syntactic ambiguity) is that it's confusing. `+=` and `=+` are just visually too close, and the distinction is not all that intuitive. At least for `a += b` the expansion to `a = a + b` is a simple contraction. But for the proposed `a =+ b` the expansion would be a transposition: `a = b + a`. Given the uncommon use case this just isn't worth the potential confusion. (Also, there are other possible meanings one could guess for `a =+ b`, given the meaning of `a += b`. E.g., because it's the exact mirror image, you could guess that it means `b = a + b`.) On Sat, Nov 12, 2016 at 12:23 PM, Jo?o Matos wrote: > Hello, > > I understand that, but in my view a =- c and a = -c should not be same. > Maybe an ideia for Python 4. > > Best regards, > > JM > > > On 12-11-2016 20:19, Bernardo Sulzbach wrote: > >> On 2016-11-12 18:08, Jo?o Matos wrote: >> >>> a =- c is the same as a = c - a >>> >> >> This would break compatibility. >> >> a =- c >> >> is >> >> a = -c >> >> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 12 20:08:02 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 12 Nov 2016 17:08:02 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 11:41 AM, Ivan Levkivskyi wrote: > On 12 November 2016 at 20:27, David Mertz wrote: > >> I thought of the use of `.__getitem__()` in metaclasses in the typing >> module. I feel like this use is more natural and more useful than that. >> Should we someday need a slice generic type for PEP 484, the spelling would >> naturally be `Slice[T]` instead, in my mind. But `slice[1:10,2]` should be >> a constructor for a concrete slice object. >> > > Slice[T] vs slice[::-1] is coherent with what we have now for List[T] vs > list, etc. > Not really. We have List[T] but list[x] is invalid -- it doesn't have a different meaning (it's list instances that support indexing). And in fact the distinction between List and list is intentionally minimal -- List is simply what list wants to become when it grows up. :-) Honestly I think the use case of wanting to create a slice object is rare enough that we can continue to write slice(x, y, z). If you really find yourself wanting something shorter, I believe in the past it's been pointed out that you could create a helper, e.g. like this: class S: def __getitem__(self, x): return x s = S() a = s[:():] -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Sat Nov 12 20:13:58 2016 From: shoyer at gmail.com (Stephan Hoyer) Date: Sat, 12 Nov 2016 17:13:58 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 11:10 AM, David Mertz wrote: > +100 > > I like this idea of giving `slice` a metaclass that defines a > `.__getitem__()` allowing us to construct slices on the slice type itself. > > FWIW, this is exactly what pandas.IndexSlice does. E.g., from > http://pandas.pydata.org/pandas-docs/stable/advanced.html: > Indeed, this would be really nice! The requirement to use a special object to construct slices outside of indexing is a repeated pain-point for me and users of pandas/xarray. To non-experts, it's not at all obvious what slice(None, 3, None) means, but slice[:3] has the familiar syntax. In xarray, we encourage passing around slice objects to do indexing with keyword arguments [1], e.g., data.sel(time=slice(100)) to pull out the first 100 values along the time axis. data.sel(time=slice[:100]) would be a significant improvement. Even if we ever get indexing with keyword arguments in Python, I still like the readability of data[time=slice[:100]] better than data[time=:100], where the colon gets lost. [1] http://xarray.pydata.org/en/stable/indexing.html#indexing-with-labeled-dimensions -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 20:28:16 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 17:28:16 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: The very common use case for creating slice objects is in Pandas and similar libraries. Xarray certainly, or Blaze, to a lesser extent NumPy. That said, it's very easy to define a suitable __getitem__, as Guido shows. It's really a question simply of whether that object should be named 'slice' or something else. On Nov 12, 2016 5:08 PM, "Guido van Rossum" wrote: > On Sat, Nov 12, 2016 at 11:41 AM, Ivan Levkivskyi > wrote: > >> On 12 November 2016 at 20:27, David Mertz wrote: >> >>> I thought of the use of `.__getitem__()` in metaclasses in the typing >>> module. I feel like this use is more natural and more useful than that. >>> Should we someday need a slice generic type for PEP 484, the spelling would >>> naturally be `Slice[T]` instead, in my mind. But `slice[1:10,2]` should be >>> a constructor for a concrete slice object. >>> >> >> Slice[T] vs slice[::-1] is coherent with what we have now for List[T] vs >> list, etc. >> > > Not really. We have List[T] but list[x] is invalid -- it doesn't have a > different meaning (it's list instances that support indexing). And in fact > the distinction between List and list is intentionally minimal -- List is > simply what list wants to become when it grows up. :-) > > Honestly I think the use case of wanting to create a slice object is rare > enough that we can continue to write slice(x, y, z). If you really find > yourself wanting something shorter, I believe in the past it's been pointed > out that you could create a helper, e.g. like this: > > class S: > def __getitem__(self, x): return x > s = S() > > a = s[:():] > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 20:46:01 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 17:46:01 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: If we *do* want the name 'slice' as the spelling for the thing that can either be called or indexed to create a slice object, we could probably use some boilerplate like this: In [1]: class Slice: ...: def __init__(self): ...: self.slice = slice ...: def __getitem__(self, x): ...: return x ...: def __call__(self, *args, **kws): ...: return self.slice(*args, **kws) ...: In [2]: slice = Slice() In [3]: slice(1,10,2) Out[3]: slice(1, 10, 2) In [4]: slice[1:10:2] Out[4]: slice(1, 10, 2) I'm sure there are some less common uses of the name 'slice' that would break here. That's why I'd want an official standard behavior. > The very common use case for creating slice objects is in Pandas and > similar libraries. Xarray certainly, or Blaze, to a lesser extent NumPy. > > That said, it's very easy to define a suitable __getitem__, as Guido > shows. It's really a question simply of whether that object should be named > 'slice' or something else. > On Nov 12, 2016 5:08 PM, "Guido van Rossum" wrote: > >> Honestly I think the use case of wanting to create a slice object is rare >> enough that we can continue to write slice(x, y, z). If you really find >> yourself wanting something shorter, I believe in the past it's been pointed >> out that you could create a helper, e.g. like this: >> > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 12 20:58:38 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 12 Nov 2016 17:58:38 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Nov 12, 2016 5:46 PM, "David Mertz" wrote: > > If we *do* want the name 'slice' as the spelling for the thing that can either be called or indexed to create a slice object, we could probably use some boilerplate like this: > >> In [1]: class Slice: >> ...: def __init__(self): >> ...: self.slice = slice >> ...: def __getitem__(self, x): >> ...: return x >> ...: def __call__(self, *args, **kws): >> ...: return self.slice(*args, **kws) >> ...: >> >> In [2]: slice = Slice() >> >> In [3]: slice(1,10,2) >> Out[3]: slice(1, 10, 2) >> >> In [4]: slice[1:10:2] >> Out[4]: slice(1, 10, 2) >> > I'm sure there are some less common uses of the name 'slice' that would break here. That's why I'd want an official standard behavior. isinstance(obj, slice) would be a notable one. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 12 22:07:34 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 12 Nov 2016 19:07:34 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 5:46 PM, David Mertz wrote: > If we *do* want the name 'slice' as the spelling for the thing that can > either be called or indexed to create a slice object, we could probably use > some boilerplate like this: > I can't stop you from doing that in your own session, but I don't want to reuse the builtin slice object for that. If this is so useful with Pandas maybe the Pandas library can define its own helper for this purpose. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at 2sn.net Sat Nov 12 22:35:54 2016 From: python at 2sn.net (Alexander Heger) Date: Sun, 13 Nov 2016 14:35:54 +1100 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: <20161112092624.GN3365@ando.pearwood.info> References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: > > But let's make str() of a slice more suggestive of actual slicing, and > as a bonus, make slices easier to create too. > > str(slice(None, None, None)) => "slice[:]" > following all the later discussion, I'd like to come back to this initial part of this thread: An even briefer for for just "str" would be to omit the 'slice' part str(slice(None, None, None)) => "[:]" etc. Since it has colons in there, it's clear it is a slice. And it is very brief notation. -Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Nov 12 23:32:02 2016 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 Nov 2016 20:32:02 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: Indeed. I mentioned up-thread that pandas names this IndexSlicer, so it does exist. And maybe that's a perfectly fine spelling (users can always give it a shorter name, like ndx) On Nov 12, 2016 7:07 PM, "Guido van Rossum" wrote: > On Sat, Nov 12, 2016 at 5:46 PM, David Mertz wrote: > >> If we *do* want the name 'slice' as the spelling for the thing that can >> either be called or indexed to create a slice object, we could probably use >> some boilerplate like this: >> > > I can't stop you from doing that in your own session, but I don't want to > reuse the builtin slice object for that. If this is so useful with Pandas > maybe the Pandas library can define its own helper for this purpose. > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 13 01:51:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Nov 2016 16:51:14 +1000 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: On 13 November 2016 at 02:57, David Mertz wrote: > If recommend 'valued(foo)'. Without the final 'd' I think of "the value of > foo" rather than "does foo have a value?" Obviously, the value of foo is > just spelled 'foo' in Python, but it seems confusing. > > 'exists(foo)' is even more confusing since almost everyone will read it as > "is foo defined?" I know you can't do that with a call in Python, but you > can in lots of other languages. Right, I was actually persuaded by Steven's argument that PEP 532 would work better if it didn't propose competing directly with PEP 505 at all, and instead positioned itself as providing the underlying conceptual unification between that PEP and the existing short-circuiting operators. Updating the PEP draft to work along those lines is making me thing it's a good direction to take. The key benefit of that approach in relation to `??` specifically is that rather than trying to find circuit breaker names that are short and suggestive, we can just name them literally in the operator module such that "LHS ?? RHS" becomes equivalent to "operator.is_not_none(LHS) else RHS". At that point, if we did decide to offer a builtin instead of dedicated syntax, the option I'd argue for is actually SQL's "coalesce": coalesce(EXPR1) else coalesce(EXPR2) else EXPR3 Yes, it's computer science jargon, but the operation itself is an odd one that doesn't really have an established mathematical precedent or grammatical English equivalent. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From levkivskyi at gmail.com Sun Nov 13 06:25:08 2016 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sun, 13 Nov 2016 12:25:08 +0100 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On 13 November 2016 at 04:07, Guido van Rossum wrote: > > On Sat, Nov 12, 2016 at 5:46 PM, David Mertz wrote: > >> If we *do* want the name 'slice' as the spelling for the thing that can >> either be called or indexed to create a slice object, we could probably use >> some boilerplate like this: >> > > I can't stop you from doing that in your own session, but I don't want to > reuse the builtin slice object for that. If this is so useful with Pandas > maybe the Pandas library can define its own helper for this purpose. > This reminds me @ vs .dot() for matrix multiplication a bit. Pandas has IndexSlicer, NumPy has index_exp, etc. I think it would be nice to have a simple common way to express this. But here we have an additional ingredient -- generic types. I think that a reasonable compromise would be to simply continue the way proposed in http://bugs.python.org/issue24379 -- just add operator.subscript for this purpose. Pros: * subscript is not a class, so that subscript[...] will be not confused with generics; * this does not require patching built-ins; * all libraries that need this will get a "common interface" in stdlib, operator module seems to be good place for this. The patch for operator.subscript was already applied, but it is now reverted because it caused a refleak. I have submitted a new patch that does not cause refleaks, I just replaced empty __slots__ with a __setattr__: it looks like __slots__ are not needed here to save memory (there is only a singleton instance), they were used just to create an immutable object. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 13 10:53:02 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 13 Nov 2016 07:53:02 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: Sounds good. On Sun, Nov 13, 2016 at 3:25 AM, Ivan Levkivskyi wrote: > On 13 November 2016 at 04:07, Guido van Rossum wrote: > >> >> On Sat, Nov 12, 2016 at 5:46 PM, David Mertz wrote: >> >>> If we *do* want the name 'slice' as the spelling for the thing that can >>> either be called or indexed to create a slice object, we could probably use >>> some boilerplate like this: >>> >> >> I can't stop you from doing that in your own session, but I don't want to >> reuse the builtin slice object for that. If this is so useful with Pandas >> maybe the Pandas library can define its own helper for this purpose. >> > > This reminds me @ vs .dot() for matrix multiplication a bit. Pandas has > IndexSlicer, NumPy has index_exp, etc. I think it would be nice to have a > simple common way to express this. > But here we have an additional ingredient -- generic types. I think that a > reasonable compromise would be to simply continue the way proposed in > http://bugs.python.org/issue24379 -- just add operator.subscript for this > purpose. Pros: > * subscript is not a class, so that subscript[...] will be not confused > with generics; > * this does not require patching built-ins; > * all libraries that need this will get a "common interface" in stdlib, > operator module seems to be good place for this. > > The patch for operator.subscript was already applied, but it is now > reverted because it caused a refleak. > I have submitted a new patch that does not cause refleaks, I just replaced > empty __slots__ with a __setattr__: > it looks like __slots__ are not needed here to save memory (there is only > a singleton instance), > they were used just to create an immutable object. > > -- > Ivan > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 13 10:55:18 2016 From: guido at python.org (Guido van Rossum) Date: Sun, 13 Nov 2016 07:55:18 -0800 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On Sat, Nov 12, 2016 at 7:35 PM, Alexander Heger wrote: > > But let's make str() of a slice more suggestive of actual slicing, and >> as a bonus, make slices easier to create too. >> >> str(slice(None, None, None)) => "slice[:]" >> > > following all the later discussion, I'd like to come back to this initial > part of this thread: > An even briefer for for just "str" would be to omit the 'slice' part > > str(slice(None, None, None)) => "[:]" > > etc. Since it has colons in there, it's clear it is a slice. And it is > very brief notation. > It is also confusing -- it is surrounded by [] but it is not a list (not even close). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Sun Nov 13 16:00:19 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 13 Nov 2016 22:00:19 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: On 12 November 2016 at 21:08, Jo?o Matos wrote: > What I would like to propose is the creation of the reverse: > a =+ c is the same as a = c + a > a =- c is the same as a = c - a > a =* c is the same as a = c * a > a =/ c is the same as a = c / a > a =// c is the same as a = c // a > a =% c is the same as a = c % a > a =** c is the same as a = c ** a A good syntax example: a = sum (a, c) a = mul (a, c) a = div (a, c) Another good syntax, I'm not a fan of, but at least intuitive and learnt in school: a = a + c a = a * c a = a / c Bad syntax, not readable: a += c a -= c a *= c And your proposal: slightly different looking, but as bad as above, at least I don't see any improvement of the look, actually it makes even harder to see the beginning of the right part of assignment, so IMO it is merely worsening of already bad syntax. As for me, I would even prohibit all these += for the sake of readability. Mikhail From toddrjen at gmail.com Sun Nov 13 18:08:22 2016 From: toddrjen at gmail.com (Todd) Date: Sun, 13 Nov 2016 18:08:22 -0500 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: On Sun, Nov 13, 2016 at 4:00 PM, Mikhail V wrote: > On 12 November 2016 at 21:08, Jo?o Matos wrote: > > > What I would like to propose is the creation of the reverse: > > a =+ c is the same as a = c + a > > a =- c is the same as a = c - a > > a =* c is the same as a = c * a > > a =/ c is the same as a = c / a > > a =// c is the same as a = c // a > > a =% c is the same as a = c % a > > a =** c is the same as a = c ** a > > A good syntax example: > > a = sum (a, c) > a = mul (a, c) > a = div (a, c) > Except that there is no "div" in python 3.x, and "sum(a, c)" does not add "a" and "c". > Another good syntax, I'm not a fan of, but at least intuitive and > learnt in school: > > a = a + c > a = a * c > a = a / c > How is using a function better than using an operator? Especially considering the ambiguity issues you just demonstrated. The fact the operator version is part of the core language while the function versions aren't even builtins (they are in the "operator" module) suggests to me that the function version is not the preferred version in Python. > Bad syntax, not readable: > > a += c > a -= c > a *= c > > What, specifically, is not readable about this syntax? > As for me, I would even prohibit all these += for the sake of readability. > Great, so I will have to make a copy of my 500 MB array every time I want to do a simple mathematical operation on it rather than being able to do the operation in-place. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Sun Nov 13 19:19:30 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 14 Nov 2016 01:19:30 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: On 14 November 2016 at 00:08, Todd wrote: > > On Sun, Nov 13, 2016 at 4:00 PM, Mikhail V wrote: >> >> On 12 November 2016 at 21:08, Jo?o Matos wrote: >> >> > What I would like to propose is the creation of the reverse: >> > a =+ c is the same as a = c + a >> > a =- c is the same as a = c - a >> > a =* c is the same as a = c * a >> > a =/ c is the same as a = c / a >> > a =// c is the same as a = c // a >> > a =% c is the same as a = c % a >> > a =** c is the same as a = c ** a >> >> A good syntax example: >> >> a = sum (a, c) >> a = mul (a, c) >> a = div (a, c) > > > Except that there is no "div" in python 3.x, and "sum(a, c)" does not add > "a" and "c". > >> >> Another good syntax, I'm not a fan of, but at least intuitive and >> learnt in school: >> >> a = a + c >> a = a * c >> a = a / c > > > How is using a function better than using an operator? Especially > considering the ambiguity issues you just demonstrated. > > The fact the operator version is part of the core language while the > function versions aren't even builtins (they are in the "operator" module) > suggests to me that the function version is not the preferred version in > Python. > >> >> Bad syntax, not readable: >> >> a += c >> a -= c >> a *= c >> > > What, specifically, is not readable about this syntax? > >> >> As for me, I would even prohibit all these += for the sake of >> readability. > > > Great, so I will have to make a copy of my 500 MB array every time I want to > do a simple mathematical operation on it rather than being able to do the > operation in-place. It is kind of clear from the context, that I am speaking of syntax and not how things are working under the hood, or? If a compiler cannot optimize "a = a + 1" into an in-place operation, that is misfortune. In numpy one can define in-place calulations, if I am not mistaken, with np.vectorize. A better option would be to support unary operators so the user can write directly without assignment: inc (a, 1) Would mean in-place increment "a" with 1 >> a += c >> a -= c >> a *= c >> > > What, specifically, is not readable about this syntax? Well, if you find this syntax readable, then I am afraid we are on the opposite sides of barricades ;) Mikhail From danilo.bellini at gmail.com Mon Nov 14 01:03:52 2016 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Mon, 14 Nov 2016 04:03:52 -0200 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> Message-ID: 2016-11-06 23:27 GMT-02:00 Wes Turner : > - So, IIUC, for recursive list comprehensions > - "prev" = x_(n-1) > - there is a need to define an initial value > - chain([1000], [...]) > - sometimes, we actually need window function > - __[0] = x_(n-1) > - __[1] = x_(n-2) # this > - __[-1] = x_(n-2) # or this > - this can be accomplished with dequeue > - __= dequeue([1000], maxlen) > - for recursive list comprehensions, we'd want to bind e.g. __ to a > dequeue > > [f(__[0], x) for x in y with __ = dequeue((1000,), 1)] > If I understood correctly, that's an alternative to get a general recursive list comprehension with a syntax like: [f(hist, x) for x in y with hist = deque([start_values], size)] You're trying to solve the output lag/window problem using a circular queue with random/indexed access to its values (a collections.deque instance). You're using "with" instead of "from" to distinguish it from my first proposal. That's not a scan anymore, but something more general. Some information can be removed from that syntax, for example the size can be defined to be the starting iterable/memory/history data size, and the deque can be something internal. Also, using the negative indices would be more explicit as hist[-1] would be the previous iteration result, hist[-2] would be its former result and so on. The syntax would be: [func(hist, target) for target in iterable with hist = start_iterable] i.e., this idea is about a new "with hist = start_iterable" at the end (or " from" instead of "with"). The resulting list size would be len(list(start_iterable)) + len(list(iterable)). As a generator instead of a list, that can be implemented as this "windowed scan" generator function: >>> import collections >>> def wscan(func, iterable, start_iterable): ... pre_hist = [] ... for item in start_iterable: ... yield item ... pre_hist.append(item) ... hist = collections.deque(pre_hist, len(pre_hist)) ... for target in iterable: ... item = func(hist, target) ... yield item ... hist.append(item) The Fibonacci example would be written as: >>> list(wscan(lambda fibs, unused: fibs[-1] + fibs[-2], range(10), [0, 1])) [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] With the "windowed scan" syntax proposal, it would become: >>> [fibs[-1] + fibs[-2] for unused in range(10) with fibs = [0, 1]] [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] Or: >>> [fibs[-1] + fibs[-2] for unused in range(10) from fibs = [0, 1]] [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Nov 14 02:30:53 2016 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 14 Nov 2016 01:30:53 -0600 Subject: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions In-Reply-To: References: <20161023155920.GR22471@ando.pearwood.info> <20161024002939.GV22471@ando.pearwood.info> <22555.3340.450045.611347@turnbull.sk.tsukuba.ac.jp> <22559.35790.868083.267230@turnbull.sk.tsukuba.ac.jp> Message-ID: I think I understand now. Still, for the more general case, I think it could be useful to be able to specify a max window size for the the dequeue (or indexable list-like structure); though defaulting to None and storing all values may be additionally useful. On Monday, November 14, 2016, Danilo J. S. Bellini wrote: > 2016-11-06 23:27 GMT-02:00 Wes Turner >: > >> - So, IIUC, for recursive list comprehensions >> - "prev" = x_(n-1) >> - there is a need to define an initial value >> - chain([1000], [...]) >> - sometimes, we actually need window function >> - __[0] = x_(n-1) >> - __[1] = x_(n-2) # this >> - __[-1] = x_(n-2) # or this >> - this can be accomplished with dequeue >> - __= dequeue([1000], maxlen) >> - for recursive list comprehensions, we'd want to bind e.g. __ to a >> dequeue >> >> [f(__[0], x) for x in y with __ = dequeue((1000,), 1)] >> > > If I understood correctly, that's an alternative to get a general > recursive list comprehension with a syntax like: > > [f(hist, x) for x in y with hist = deque([start_values], size)] > > You're trying to solve the output lag/window problem using a circular > queue with random/indexed access to its values (a collections.deque > instance). You're using "with" instead of "from" to distinguish it from > my first proposal. > > That's not a scan anymore, but something more general. Some information > can be removed from that syntax, for example the size can be defined to be > the starting iterable/memory/history data size, and the deque can be > something internal. Also, using the negative indices would be more explicit > as hist[-1] would be the previous iteration result, hist[-2] would be its > former result and so on. The syntax would be: > > [func(hist, target) for target in iterable with hist = start_iterable] > > i.e., this idea is about a new "with hist = start_iterable" at the end > (or "from" instead of "with"). The resulting list size would be len(list(start_iterable)) > + len(list(iterable)). As a generator instead of a list, that can be > implemented as this "windowed scan" generator function: > > >>> import collections > >>> def wscan(func, iterable, start_iterable): > ... pre_hist = [] > ... for item in start_iterable: > ... yield item > ... pre_hist.append(item) > ... hist = collections.deque(pre_hist, len(pre_hist)) > ... for target in iterable: > ... item = func(hist, target) > ... yield item > ... hist.append(item) > > The Fibonacci example would be written as: > > >>> list(wscan(lambda fibs, unused: fibs[-1] + fibs[-2], range(10), [0, > 1])) > [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] > > With the "windowed scan" syntax proposal, it would become: > > >>> [fibs[-1] + fibs[-2] for unused in range(10) with fibs = [0, 1]] > [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] > > Or: > > >>> [fibs[-1] + fibs[-2] for unused in range(10) from fibs = [0, 1]] > [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at rcfox.ca Mon Nov 14 04:01:37 2016 From: ryan at rcfox.ca (Ryan Fox) Date: Mon, 14 Nov 2016 04:01:37 -0500 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol Message-ID: (Hi, I'm a first-time poster. I was inspired by Raymond Hettinger's keynote at PyCon CA to look at new PEPs and comment on them. Hopefully, I'm not committing any faux-pas in my post!) 1) I don't think this proposal sufficiently handles falsy values. What is the expected value of x in the following? x = exists(0) else 1 Since the PEP specifically states that exists() is to check that the input is not None, as a user, I would expect x == 0. However, from my interpretation, it appears that the value 0 would still be directly evaluated for truthiness and we would get x == 1. I don't fully get what the purpose of __then__ and __else__ are meant to be, but it seems like instead of this: type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) you would want: LHS if type(_lhs).__then__(_lhs) else RHS Where __then__ returns a simple True/False result. (Maybe the name __then__ doesn't make sense in that case.) 2) My initial reaction was that `else` doesn't belong in an expression, but I guess there's already precedent for that. (I actually wasn't aware of the `x if y else z` expression until I read this PEP!) I'm already not a fan of the overloading of else in the cases of for/else and try/else. (Are there other uses? It's a hard thing to search on Google...) Now, we're going to have `else`s that aren't anchored to other statements. You've always known that an `else` belonged to the preceding if/for/try at the same level of indentation. (Or in the same expression.) Is there a chance of a missing colon silently changing the meaning of code? if foo(): a = 1 else bar() (Probably not...) Are two missing spaces too outrageous? x = yiffoo() else bar() I'm not 100% sure, but I think that a current parser would see that as a syntax error very early, as opposed to having to wait for it to try to find 'yiffoo' at run-time. 3) Towards the end, you propose some very Perl-like syntax: print(some_expensive_query()) if verbosity > 2 This seems completely unrelated to the rest of the PEP, and will likely invite people to propose an `unless` operator if implemented. (Followed by an `until` statement.) :) While it does read more naturally like English, it seems superfluous in a programming language. 4) The proposal shows how it fixes some common pain points: value = missing(obj) else obj.field.of.interest value = missing(obj) else obj["field"]["of"]["interest"] But it doesn't address very similar ones: missing(obj) else missing(obj.field) else missing(obj.field.of) else obj.field.of.interest obj.get('field', {}).get('of', {}).get('interest') (The first example shows how it would be handled with the PEP in its current state.) Maybe these are too far out of scope, I'm not sure. They feel very similar to me though. I hope these are useful comments and not too nit-picky. Thanks, Ryan Fox -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 14 06:16:27 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 14 Nov 2016 22:16:27 +1100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> Message-ID: <20161114111627.GR3365@ando.pearwood.info> On Mon, Nov 14, 2016 at 01:19:30AM +0100, Mikhail V wrote: [...] > >> A good syntax example: > >> > >> a = sum (a, c) There is a reason why mathematicians and accountants use symbols as a compact notation for common functions, instead of purely functional notation. Here is a real-world example, the formula for compound interest: A = P*(1+r/100)**n Compare that to: A = mul(P, pow(sum(1, div(r, 100)), n)) Even Reverse Polish Notation is easier to read than that series of functions: P 1 r 100 / + n ** * Don't misunderstand me, functions are important, and over-use of cryptic symbols that are only meaningful to an expert will hurt readability and maintainability of code. But people have needed to do arithmetic for over six thousand years, and there is no good substitute for compact operators for basic operations. [...] > It is kind of clear from the context, that I am speaking of syntax and > not how things are working under the hood, or? > If a compiler cannot optimize "a = a + 1" into an in-place operation, > that is misfortune. That's not how Python works. Or at least, not without an extremely powerful and smart compiler, like PyPy. In Python, integers (and floats) are immutable objects. They have to be immutable, otherwise you would have things like this: x = 1 y = x x = x + 1 # changes the object 1 in place print(y*10) # expect 10, but get 20 That's how lists work, because they are mutable: py> x = [1] py> y = x py> x[0] = x[0] + 1 py> print(y[0]*10) # expect 1*10 = 10 20 A "sufficiently smart" compiler can work around this, as PyPy does under some circumstances, but you shouldn't expect this optimization to be simple. [...] > A better option would be to support unary operators so the user > can write directly without assignment: > > inc (a, 1) > > Would mean in-place increment "a" with 1 That is impossible with Python's executation model, in particular the "pass by object sharing" calling convention. I expect that this would require some extremely big changes to the way the compiler works, possibly a complete re-design, in order to allow pass by reference semantics. The biggest problem is that even if the compiler could choose between calling conventions, it would have to make that decision at runtime. It won't know until runtime that inc() requires pass by reference. That would make function calls even slower than they are now. -- Steve From ncoghlan at gmail.com Mon Nov 14 08:35:02 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Nov 2016 23:35:02 +1000 Subject: [Python-ideas] str(slice(10)) should return "slice(10)" In-Reply-To: References: <18c9fb30-566a-4213-a066-7ea5c9f5c44e@googlegroups.com> <20161112092624.GN3365@ando.pearwood.info> Message-ID: On 13 November 2016 at 21:25, Ivan Levkivskyi wrote: > This reminds me @ vs .dot() for matrix multiplication a bit. Pandas has > IndexSlicer, NumPy has index_exp, etc. I think it would be nice to have a > simple common way to express this. > But here we have an additional ingredient -- generic types. I think that a > reasonable compromise would be to simply continue the way proposed in > http://bugs.python.org/issue24379 -- just add operator.subscript for this > purpose. Pros: > * subscript is not a class, so that subscript[...] will be not confused with > generics; > * this does not require patching built-ins; > * all libraries that need this will get a "common interface" in stdlib, > operator module seems to be good place for this. >From an educational point of view, it also makes it a bit easier to give interactive examples of how slicing syntax maps to the subscript parameter on __getitem__, __setitem__ and __delitem__: you can just do "print(operator.subscript[EXPR])" rather than having to build a dummy __getitem__ implementation of your own. If an actual use case is found for it, that approach would also leave "operator.subscript('EXPR')" available for a micro-eval implementation that evaluated a given string as a subscript rather than as a normal top level expression. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 14 09:11:54 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Nov 2016 00:11:54 +1000 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: On 14 November 2016 at 19:01, Ryan Fox wrote: > (Hi, I'm a first-time poster. I was inspired by Raymond Hettinger's keynote > at PyCon CA to look at new PEPs and comment on them. Hopefully, I'm not > committing any faux-pas in my post!) > > 1) I don't think this proposal sufficiently handles falsy values. > > What is the expected value of x in the following? > > x = exists(0) else 1 > > Since the PEP specifically states that exists() is to check that the input > is not None, as a user, I would expect x == 0. However, from my > interpretation, it appears that the value 0 would still be directly > evaluated for truthiness and we would get x == 1. No, the conditional branching would be based on exists.__bool__ (or, in the current working draft, is_not_none.__bool__), and that would be "0 is not None", which would be True and hence short-circuit. > I don't fully get what the purpose of __then__ and __else__ are meant to be, > but it seems like instead of this: > > type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS) > > you would want: > > LHS if type(_lhs).__then__(_lhs) else RHS `__then__` is responsible for *unwrapping* the original value from the circuit breaker when it short-circuits: it's what allows the overall expression to return "0", even though the truth check is done based on "0 is not None". > Where __then__ returns a simple True/False result. (Maybe the name __then__ > doesn't make sense in that case.) We already have a method for that: __bool__. However, it has exactly the problem you describe, which is why "0 or expr" will always short-circuit, and why "(0 is not None) or expr" will return "True". > 2) My initial reaction was that `else` doesn't belong in an expression, but > I guess there's already precedent for that. (I actually wasn't aware of the > `x if y else z` expression until I read this PEP!) > > I'm already not a fan of the overloading of else in the cases of for/else > and try/else. (Are there other uses? It's a hard thing to search on > Google...) Now, we're going to have `else`s that aren't anchored to other > statements. You've always known that an `else` belonged to the preceding > if/for/try at the same level of indentation. (Or in the same expression.) > Is there a chance of a missing colon silently changing the meaning of code? > > if foo(): > a = 1 > else > bar() > > (Probably not...) No, due to Python's line continuation rules - you'd also need parentheses or a backslash to avoid getting a SyntaxError on the unfinished line. It does create amibiguities around conditional expressions though, hence why that comes up as one of the main pragmatic concerns with the idea. > Are two missing spaces too outrageous? > > x = yiffoo() else bar() > > I'm not 100% sure, but I think that a current parser would see that as a > syntax error very early, as opposed to having to wait for it to try to find > 'yiffoo' at run-time. That style of error is already possible with the other keyword based operators: x = notfoo() y = xorfoo() z = yandfoo() As you not, the main defense is that this will usually be a name error, picked up either at runtime or by a static code analyser. > 3) Towards the end, you propose some very Perl-like syntax: > > print(some_expensive_query()) if verbosity > 2 > > This seems completely unrelated to the rest of the PEP, and will likely > invite people to propose an `unless` operator if implemented. (Followed by > an `until` statement.) :) > > While it does read more naturally like English, it seems superfluous in a > programming language. De Morgan's laws [1] mean that 'and' and 'or' are technically redundant with each other, as given 'not', you can always express one in terms of the other: X and Y --> not ((not X) or (not Y)) X or Y --> not ((not X) and (not Y)) However, writing out the laws like that also makes it clear why they're not redundant in practice: the inverted forms involve double-negatives that make them incredibly hard to read. Those rules impact this PEP by way of the fact that in "LHS if COND else RHS", the "if" and "else" are actually in the same logical relation to each other as "and" and "or" are in "COND and LHS or RHS". Accordingly, if "LHS if COND else RHS" were to be reformulated as a compound instruction built from two binary instructions (akin to the way comparison chaining works) as considered in the "Risks and Concerns" section about the language level inconsistencies that the current draft introduces, then we'd expect De Morgan's laws to hold there as well: Y if X --> not ((not X) else (not Y)) X else Y --> not ((not Y) if (not X)) It hadn't occurred to me to include that observation in the PEP while updating it to switch to that base design, but it really should be there as an additional invariant that well-behaved symmetric circuit breakers should adhere to. [1] https://en.wikipedia.org/wiki/De_Morgan%27s_laws > 4) The proposal shows how it fixes some common pain points: > > value = missing(obj) else obj.field.of.interest > value = missing(obj) else obj["field"]["of"]["interest"] > > But it doesn't address very similar ones: > > missing(obj) else missing(obj.field) else missing(obj.field.of) else > obj.field.of.interest > obj.get('field', {}).get('of', {}).get('interest') > > (The first example shows how it would be handled with the PEP in its current > state.) > > Maybe these are too far out of scope, I'm not sure. They feel very similar > to me though. The current draft already indicates it doesn't aim to compete with the "?." or "?[]" proposals in PEP 505 (which handle these two cases), and the next draft drops the competition with "??" as well. That way, the proposed circuit breakers for the PEP 505 cases can just be "operator.is_none" and "operator.is_not_none". > I hope these are useful comments and not too nit-picky. They were very helpful, and picked up a key technical point that I'd missed in the revised proposal I'm currently working on. I think I also need to restore a diagram that Mark E. Haase drew for an earlier draft of the PEP that may make it easier for folks to visualise the related control flow. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mikhailwas at gmail.com Mon Nov 14 13:21:25 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 14 Nov 2016 19:21:25 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <20161114111627.GR3365@ando.pearwood.info> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 14 November 2016 at 12:16, Steven D'Aprano wrote: > On Mon, Nov 14, 2016 at 01:19:30AM +0100, Mikhail V wrote: > > [...] >> >> A good syntax example: >> >> >> >> a = sum (a, c) > > There is a reason why mathematicians and accountants use symbols as a > compact notation for common functions, instead of purely functional > notation. > > Here is a real-world example, the formula for compound interest: > > A = P*(1+r/100)**n > > Compare that to: > > A = mul(P, pow(sum(1, div(r, 100)), n)) > > Even Reverse Polish Notation is easier to read than that series of > functions: > > P 1 r 100 / + n ** * > > > Don't misunderstand me, functions are important, and over-use of cryptic > symbols that are only meaningful to an expert will hurt readability and > maintainability of code. But people have needed to do arithmetic for > over six thousand years, and there is no good substitute for compact > operators for basic operations. I agree. Actually I meant that both are good examples. In some cases I still find function-style better, so this: A = P*(1+r/100)**n I would tend to write it like: A = P * pow((1 + r/100) , n) For me it is slightly more readable, since ** already makes the equation inconsistent. And of course you cannot use glyphs for all possible operations, it will be total mess. Most of problems with function-style equations come from limitations of representation so for Courier font for examples the brackets are too small and makes it hard to read. With good font and rendering there no such problems. Some equation editors allow even different sized brackets - the outer extend more and more when I add nested equations, so it looks way better. > [...] >> It is kind of clear from the context, that I am speaking of syntax and >> not how things are working under the hood, or? >> If a compiler cannot optimize "a = a + 1" into an in-place operation, >> that is misfortune. > > That's not how Python works. Or at least, not without an extremely > powerful and smart compiler, like PyPy. > > In Python, integers (and floats) are immutable objects. They have to be > immutable, otherwise you would have things like this: > > x = 1 > y = x > x = x + 1 # changes the object 1 in place > print(y*10) # expect 10, but get 20 > > That's how lists work, because they are mutable: > > py> x = [1] > py> y = x > py> x[0] = x[0] + 1 > py> print(y[0]*10) # expect 1*10 = 10 > 20 > > > A "sufficiently smart" compiler can work around this, as PyPy does under > some circumstances, but you shouldn't expect this optimization to be > simple. > > [...] >> A better option would be to support unary operators so the user >> can write directly without assignment: >> >> inc (a, 1) >> >> Would mean in-place increment "a" with 1 > > That is impossible with Python's executation model, in particular the > "pass by object sharing" calling convention. I expect that this would > require some extremely big changes to the way the compiler works, > possibly a complete re-design, in order to allow pass by reference > semantics. Thanks a lot for a great explanation! So for current Python behavior, writing x = x + 1 and x += 1 Would mean the same in runtime? Namely creates a copy and assigns the value x+1 to x. Or there is still some overhead at parsing stage? Then however I see even less sense in using such a shortcut, it would be good to dismiss this notation, since it makes hard to read the code. As for Numpy, it uses anyway its own approaches, so in-place should use own syntax, e.g. like numpy.sum(x_, a) to do it in-place. Mikhail From yselivanov.ml at gmail.com Mon Nov 14 13:30:35 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 14 Nov 2016 13:30:35 -0500 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: References: Message-ID: <9099c61e-c464-55ab-738a-ea7727ba7719@gmail.com> Hi Guido, On 2016-11-12 4:24 PM, Guido van Rossum wrote: > I think there's a plan to add a run() function to asyncio, which would be > something akin to > > def run(coro): > return get_event_loop().run_until_complete(coro) > > (but perhaps with better cleanup). Please see https://github.com/python/asyncio/pull/465. Yury From yselivanov.ml at gmail.com Mon Nov 14 13:39:24 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 14 Nov 2016 13:39:24 -0500 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: <2ba7f318-e93f-0826-2184-dcd7989a2966@mail.de> References: <9099c61e-c464-55ab-738a-ea7727ba7719@gmail.com> <2ba7f318-e93f-0826-2184-dcd7989a2966@mail.de> Message-ID: <499c0d2c-efb8-14c1-e2e8-b943709f39d3@gmail.com> On 2016-11-14 1:35 PM, Sven R. Kunze wrote: > What about making "run" an instance method of coroutines? That would require coroutines to be aware of the loop that is running them. Not having them aware of that is what makes the design simple and allows alternatives to asyncio. All in all I'd be strong -1 to do that. Yury From srkunze at mail.de Mon Nov 14 13:35:32 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 14 Nov 2016 19:35:32 +0100 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: <9099c61e-c464-55ab-738a-ea7727ba7719@gmail.com> References: <9099c61e-c464-55ab-738a-ea7727ba7719@gmail.com> Message-ID: <2ba7f318-e93f-0826-2184-dcd7989a2966@mail.de> What about making "run" an instance method of coroutines? On 14.11.2016 19:30, Yury Selivanov wrote: > Hi Guido, > > > On 2016-11-12 4:24 PM, Guido van Rossum wrote: >> I think there's a plan to add a run() function to asyncio, which >> would be >> something akin to >> >> def run(coro): >> return get_event_loop().run_until_complete(coro) >> >> (but perhaps with better cleanup). > > Please see https://github.com/python/asyncio/pull/465. Best, Sven From prometheus235 at gmail.com Mon Nov 14 13:57:36 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Mon, 14 Nov 2016 12:57:36 -0600 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: Currently, Numpy takes advantage of __iadd__ and friends by performing the operation in-place; there is no copying or other object created. Numpy is very thinly C, for better and worse (which is also likely where the += syntax came from). If you're doing vast amounts of numeric computation, it quickly pays off to learn a little about how C likes to store arrays in memory and that doing things like creating a whole new array of the same size and having to free up the old one for every operation isn't a great idea. I strongly dislike the notion that I'd have to use arbitrary function calls to add one to an entire array, or multiply it by 2, or add it to another array, etc. As a minor nit, np.vectorize doesn't make in-place functions, it simply makes a naive, element-wise function act as though it was vectorized. The name is unfortunate, because it does nothing to speed it up, and it usually slows it down (because of that layer). Reworking code to avoid copying large arrays by doing in-place operations is the preferred method, though not always possible. Nick On Mon, Nov 14, 2016 at 12:21 PM, Mikhail V wrote: > On 14 November 2016 at 12:16, Steven D'Aprano wrote: > > On Mon, Nov 14, 2016 at 01:19:30AM +0100, Mikhail V wrote: > > > > [...] > >> >> A good syntax example: > >> >> > >> >> a = sum (a, c) > > > > There is a reason why mathematicians and accountants use symbols as a > > compact notation for common functions, instead of purely functional > > notation. > > > > Here is a real-world example, the formula for compound interest: > > > > A = P*(1+r/100)**n > > > > Compare that to: > > > > A = mul(P, pow(sum(1, div(r, 100)), n)) > > > > Even Reverse Polish Notation is easier to read than that series of > > functions: > > > > P 1 r 100 / + n ** * > > > > > > Don't misunderstand me, functions are important, and over-use of cryptic > > symbols that are only meaningful to an expert will hurt readability and > > maintainability of code. But people have needed to do arithmetic for > > over six thousand years, and there is no good substitute for compact > > operators for basic operations. > > I agree. Actually I meant that both are good examples. > In some cases I still find function-style better, so this: > > A = P*(1+r/100)**n > > I would tend to write it like: > > A = P * pow((1 + r/100) , n) > > For me it is slightly more readable, since ** already makes the > equation inconsistent. > And of course you cannot use glyphs for all possible operations, it > will be total mess. > > Most of problems with function-style equations come from limitations > of representation > so for Courier font for examples the brackets are too small and makes > it hard to read. > With good font and rendering there no such problems. Some equation editors > allow even different sized brackets - the outer extend more and more > when I add nested equations, so it looks way better. > > > > [...] > >> It is kind of clear from the context, that I am speaking of syntax and > >> not how things are working under the hood, or? > >> If a compiler cannot optimize "a = a + 1" into an in-place operation, > >> that is misfortune. > > > > That's not how Python works. Or at least, not without an extremely > > powerful and smart compiler, like PyPy. > > > > In Python, integers (and floats) are immutable objects. They have to be > > immutable, otherwise you would have things like this: > > > > x = 1 > > y = x > > x = x + 1 # changes the object 1 in place > > print(y*10) # expect 10, but get 20 > > > > That's how lists work, because they are mutable: > > > > py> x = [1] > > py> y = x > > py> x[0] = x[0] + 1 > > py> print(y[0]*10) # expect 1*10 = 10 > > 20 > > > > > > A "sufficiently smart" compiler can work around this, as PyPy does under > > some circumstances, but you shouldn't expect this optimization to be > > simple. > > > > [...] > >> A better option would be to support unary operators so the user > >> can write directly without assignment: > >> > >> inc (a, 1) > >> > >> Would mean in-place increment "a" with 1 > > > > That is impossible with Python's executation model, in particular the > > "pass by object sharing" calling convention. I expect that this would > > require some extremely big changes to the way the compiler works, > > possibly a complete re-design, in order to allow pass by reference > > semantics. > > Thanks a lot for a great explanation! > So for current Python behavior, writing > > x = x + 1 > > and > > x += 1 > > Would mean the same in runtime? Namely creates a copy and assigns > the value x+1 to x. Or there is still some overhead at parsing stage? > Then however I see even less sense in using such a shortcut, > it would be good to dismiss this notation, since it makes hard > to read the code. > As for Numpy, it uses anyway its own approaches, so > in-place should use own syntax, e.g. like numpy.sum(x_, a) to > do it in-place. > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Nov 14 14:02:07 2016 From: toddrjen at gmail.com (Todd) Date: Mon, 14 Nov 2016 14:02:07 -0500 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On Mon, Nov 14, 2016 at 1:21 PM, Mikhail V wrote: > On 14 November 2016 at 12:16, Steven D'Aprano wrote: > > On Mon, Nov 14, 2016 at 01:19:30AM +0100, Mikhail V wrote: > > > > [...] > >> >> A good syntax example: > >> >> > >> >> a = sum (a, c) > > > > There is a reason why mathematicians and accountants use symbols as a > > compact notation for common functions, instead of purely functional > > notation. > > > > Here is a real-world example, the formula for compound interest: > > > > A = P*(1+r/100)**n > > > > Compare that to: > > > > A = mul(P, pow(sum(1, div(r, 100)), n)) > > > > Even Reverse Polish Notation is easier to read than that series of > > functions: > > > > P 1 r 100 / + n ** * > > > > > > Don't misunderstand me, functions are important, and over-use of cryptic > > symbols that are only meaningful to an expert will hurt readability and > > maintainability of code. But people have needed to do arithmetic for > > over six thousand years, and there is no good substitute for compact > > operators for basic operations. > > I agree. Actually I meant that both are good examples. > In some cases I still find function-style better, so this: > > A = P*(1+r/100)**n > > I would tend to write it like: > > A = P * pow((1 + r/100) , n) > > For me it is slightly more readable, since ** already makes the > equation inconsistent. > > Okay, then make a proposal to get the operators included as built-ins. Once you have gotten that approved, then we can start talking about which syntax is better. But the idea that people should import a module to do basic mathematical operations is a non-starter. > Most of problems with function-style equations come from limitations > of representation > so for Courier font for examples the brackets are too small and makes > it hard to read. > With good font and rendering there no such problems. Some equation editors > allow even different sized brackets - the outer extend more and more > when I add nested equations, so it looks way better. > > You shouldn't be using a variable-width font for coding anyway. That is going to cause all sorts of problems (indentation not matching up, for example). You should use a fixed-width font. But if brackets are a problem, I don't see how using more brackets is a solution (which is the case with function calls). On the contrary, I would think you would want to minimize the number of brackets. > > > [...] > >> It is kind of clear from the context, that I am speaking of syntax and > >> not how things are working under the hood, or? > >> If a compiler cannot optimize "a = a + 1" into an in-place operation, > >> that is misfortune. > > > > That's not how Python works. Or at least, not without an extremely > > powerful and smart compiler, like PyPy. > > > > In Python, integers (and floats) are immutable objects. They have to be > > immutable, otherwise you would have things like this: > > > > x = 1 > > y = x > > x = x + 1 # changes the object 1 in place > > print(y*10) # expect 10, but get 20 > > > > That's how lists work, because they are mutable: > > > > py> x = [1] > > py> y = x > > py> x[0] = x[0] + 1 > > py> print(y[0]*10) # expect 1*10 = 10 > > 20 > > > > > > A "sufficiently smart" compiler can work around this, as PyPy does under > > some circumstances, but you shouldn't expect this optimization to be > > simple. > > > > [...] > >> A better option would be to support unary operators so the user > >> can write directly without assignment: > >> > >> inc (a, 1) > >> > >> Would mean in-place increment "a" with 1 > > > > That is impossible with Python's executation model, in particular the > > "pass by object sharing" calling convention. I expect that this would > > require some extremely big changes to the way the compiler works, > > possibly a complete re-design, in order to allow pass by reference > > semantics. > > Thanks a lot for a great explanation! > So for current Python behavior, writing > > x = x + 1 > > and > > x += 1 > > Would mean the same in runtime? Namely creates a copy and assigns > the value x+1 to x. Or there is still some overhead at parsing stage? > Then however I see even less sense in using such a shortcut, > it would be good to dismiss this notation, since it makes hard > to read the code. > No, that is only the case for immutable types like floats. For mutable types like lists then x = x + y and x += y Are not the same thing. The first makes a new object, while the second does an in-place operation. Sometimes you want a new object, sometimes you don't. Having both versions allows you to control that. > As for Numpy, it uses anyway its own approaches, so > in-place should use own syntax, e.g. like numpy.sum(x_, a) to > do it in-place. > > First, there is no "_" suffix for variables that numpy could use. The syntax you are suggesting isn't possible without a major change to the basic python grammar and interpreter. Second, again, "sum(x, a)" does not add "x" and "a". Third, again, why? You still haven't given any reason we should prefer this syntax. If this is just your personal preference, please say that. But if you expect people to add a bunch of new built-ins, implement your new suffix syntax, and implement your massive changes to how python operations and function calls work, then you are going to need something more than personal preference. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Nov 14 15:42:02 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 14 Nov 2016 21:42:02 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 14 November 2016 at 19:57, Nick Timkovich wrote: > Currently, Numpy takes advantage of __iadd__ and friends by performing the > operation in-place; there is no copying or other object created. Numpy is > very thinly C, for better and worse (which is also likely where the += > syntax came from). If you're doing vast amounts of numeric computation, it > quickly pays off to learn a little about how C likes to store arrays in > memory and that doing things like creating a whole new array of the same > size and having to free up the old one for every operation isn't a great > idea. I strongly dislike the notion that I'd have to use arbitrary function > calls to add one to an entire array, or multiply it by 2, or add it to > another array, etc. > > As a minor nit, np.vectorize doesn't make in-place functions, it simply > makes a naive, element-wise function act as though it was vectorized. The > name is unfortunate, because it does nothing to speed it up, and it usually > slows it down (because of that layer). Reworking code to avoid copying large > arrays by doing in-place operations is the preferred method, though not > always possible. > > Nick I can understand you good. But imagine, if Numpy would allow you to simply write: A = A + 1 Which would bring you directly to same internal procedure as A += 1. So it does not currently, why? I've tested now A += 99 against A = A + 99 and there is indeed a 30% speed difference. So it functions different. Would you then still want to write += ? I never would. Also I think to implement this syntax would be almost trivial, it should just take the A = A part and do the rest as usual. And this for equally sized arrays: A = A + B Should just add B values to A values in-place. Now it gives me also ~30% speed difference compared to A += B. Mikhail From tomuxiong at gmx.com Mon Nov 14 15:52:20 2016 From: tomuxiong at gmx.com (Thomas Nyberg) Date: Mon, 14 Nov 2016 15:52:20 -0500 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 11/14/2016 03:42 PM, Mikhail V wrote: > On 14 November 2016 at 19:57, Nick Timkovich wrote: > > I can understand you good. But imagine, if Numpy would allow you to > simply write: > A = A + 1 > Which would bring you directly to same internal procedure as A += 1. > So it does not currently, why? > I've tested now A += 99 against A = A + 99 and there is indeed a 30% speed > difference. So it functions different. > > Would you then still want to write += ? > I never would. Also I think to implement this syntax would be almost > trivial, it should > just take the A = A part and do the rest as usual. > > And this for equally sized arrays: > A = A + B > > Should just add B values to A values in-place. Now it gives me also > ~30% speed difference > compared to A += B. > If you take this file: test.py ---------------------------------------------- a = a + 1 a += 1 ---------------------------------------------- And you look at the bytecode it produces you get the following: ---------------------------------------------- $ python -m dis test.py 1 0 LOAD_NAME 0 (a) 3 LOAD_CONST 0 (1) 6 BINARY_ADD 7 STORE_NAME 0 (a) 2 10 LOAD_NAME 0 (a) 13 LOAD_CONST 0 (1) 16 INPLACE_ADD 17 STORE_NAME 0 (a) 20 LOAD_CONST 1 (None) 23 RETURN_VALUE ---------------------------------------------- That shows that the first and second lines are compiled _differently_. The point is that in the first line, numpy does not have the information necessary to know that "a + 1" will be assigned back to a. In the second case it does. This is presumably why they couldn't do the optimization you desire without large changes in cpython. On the second point, personally I prefer writing "a += 1" to "a = a + 1". I think it's clearer and would keep using it even if the two were equally efficient. But we are all allowed our opinions... Cheers, Thomas From toddrjen at gmail.com Mon Nov 14 15:52:01 2016 From: toddrjen at gmail.com (Todd) Date: Mon, 14 Nov 2016 15:52:01 -0500 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On Mon, Nov 14, 2016 at 3:42 PM, Mikhail V wrote: > On 14 November 2016 at 19:57, Nick Timkovich > wrote: > > Currently, Numpy takes advantage of __iadd__ and friends by performing > the > > operation in-place; there is no copying or other object created. Numpy is > > very thinly C, for better and worse (which is also likely where the += > > syntax came from). If you're doing vast amounts of numeric computation, > it > > quickly pays off to learn a little about how C likes to store arrays in > > memory and that doing things like creating a whole new array of the same > > size and having to free up the old one for every operation isn't a great > > idea. I strongly dislike the notion that I'd have to use arbitrary > function > > calls to add one to an entire array, or multiply it by 2, or add it to > > another array, etc. > > > > As a minor nit, np.vectorize doesn't make in-place functions, it simply > > makes a naive, element-wise function act as though it was vectorized. The > > name is unfortunate, because it does nothing to speed it up, and it > usually > > slows it down (because of that layer). Reworking code to avoid copying > large > > arrays by doing in-place operations is the preferred method, though not > > always possible. > > > > Nick > > I can understand you good. But imagine, if Numpy would allow you to > simply write: > A = A + 1 > Which would bring you directly to same internal procedure as A += 1. > So it does not currently, why? > First, because the language doesn't allow it. But more fundamentally, sometimes we don't want A = A + 1 to be the same as A += 1. Making a copy is sometimes what you want. Having both versions lets you control when you make a copy rather than being forced to always make a copy or always not. > I never would. Also I think to implement this syntax would be almost > trivial, it should > just take the A = A part and do the rest as usual. > The Python language doesn't allow it. numpy can only work with the information provided to it be the language, and the information needed to do that sort of thing is not provided to classes. Nor should it in my opinion, this is one of those fundamental operations that I think absolutely must be consistent. What you are talking about is an enormous backwards-compatibility break. It is simply not going to happen, it would break every mutable class. And you still have not provided any reason we should want to do it this way. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Nov 14 17:13:02 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 14 Nov 2016 23:13:02 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 14 November 2016 at 21:52, Todd wrote: >> I can understand you good. But imagine, if Numpy would allow you to >> simply write: >> A = A + 1 >> Which would bring you directly to same internal procedure as A += 1. >> So it does not currently, why? > > > First, because the language doesn't allow it. > > But more fundamentally, sometimes we don't want A = A + 1 to be the same as > A += 1. Making a copy is sometimes what you want. Having both versions > lets you control when you make a copy rather than being forced to always > make a copy or always not. > we don't want A = A + 1 to be the same as A += 1 " This sounds quite frustrating for me, what else could this be but A += 1? Would I want a copy with same name? How that will make sense? Making a copy is how it works now, ok. But even now there are differences, eg: A = B [1:10] is not a copy but a reference creation operation. So there are different meanings of =. > >> >> I never would. Also I think to implement this syntax would be almost >> trivial, it should >> just take the A = A part and do the rest as usual. > > > The Python language doesn't allow it. numpy can only work with the > information provided to it be the language, and the information needed to do > that sort of thing is not provided to classes. Nor should it in my opinion, > this is one of those fundamental operations that I think absolutely must be > consistent. > > What you are talking about is an enormous backwards-compatibility break. It > is simply not going to happen, it would break every mutable class. > I don't want to break anything, no. I don't have such deep knowledge of all cases, but that is very interesting how actually it must break *everything*. Say I scan through lines and find "= " and then see that on the left is an numpy array, which already indicates a unusual variable. And if look on the right side and first argument is again A, cannot I decide to make an exception and redirect it to A.__add__() or what must be there. It does already do an overloading of all these "+=" for numpy. so what will fundamentally break everything I am not sure. > And you still have not provided any reason we should want to do it this way. If let alone numpy for now and take only numeric Python variables, I find the += syntax very bad readable. I must literally stop and lean towards the monitor to decipher the line. And I am not even telling that you *cannot* do it with all possible operations: there are standard math, ORing ,XORing, conditionals, bitshifting etc. Is not this obvious? And what other reason apart from readability there can be at all? Again, how about TOOWTDI principle? some write a = a + 1, some a += 1 ? It does not add any pleasure for reading code. >> Most of problems with function-style equations come from limitations >> of representation >> so for Courier font for examples the brackets are too small and makes >> it hard to read. >> With good font and rendering there no such problems. Some equation editors >> allow even different sized brackets - the outer extend more and more >> when I add nested equations, so it looks way better. >You shouldn't be using a variable-width font for coding anyway. >That is going to cause all sorts of problems (indentation not matching up, for example). You should use a fixed-width font. No no, I should not use monowidth fonts and nobody ever should. Most of those "indentation" problems are just though-out, All you need is IDE with good tabulation support. The readability difference between monowidth and real fonts is *huge*. Currently I am forced to use monowidth, since it is VIM's limitation and it bothers me a lot. Exception is tables with numbers, but those should not be inputed in same manner as strings, it is other medium type. Mikhail From mertz at gnosis.cx Mon Nov 14 17:36:48 2016 From: mertz at gnosis.cx (David Mertz) Date: Mon, 14 Nov 2016 14:36:48 -0800 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: This is just a matter of understanding the actual semantics of Python, which is not the same as other languages such as C. In Python the "assignment operator" is a way of *binding a value*. So writing: a = some_expression Is ALWAYS and ONLY binding the value of `some_expression` to the name `a`. The fact that `a` might occur within `some_expression` is irrelevant to these semantics (although it *does* mean that subsequent code won't use `a` to refer to whatever value it used to have. In clear and distinct contrast, "augmented assignment" is a way of calling a method on an object. So writing: a += some_expression Means EXACTLY: a.__iadd__(some_expression) I recognize this can have the same effect if `__iadd__()` is not explicitly defined, and when that is true, it will fall back to using `__add__()` as the implementation. This is described in: https://www.python.org/dev/peps/pep-0203/ To see the difference, e.g.: >>> class Foo: .... def __add__(self, other): .... print("Adding", other) .... return other .... def __iadd__(self, other): .... print("Inplace assignment", other) .... return self .... >>> foo = Foo() >>> foo += 3 Inplace assignment 3 >>> foo + 3 Adding 3 3 >>> foo = foo + 3 Adding 3 >>> foo # I'm bound to an int now 3 On Mon, Nov 14, 2016 at 2:13 PM, Mikhail V wrote: > On 14 November 2016 at 21:52, Todd wrote: > > >> I can understand you good. But imagine, if Numpy would allow you to > >> simply write: > >> A = A + 1 > >> Which would bring you directly to same internal procedure as A += 1. > >> So it does not currently, why? > > > > > > First, because the language doesn't allow it. > > > > But more fundamentally, sometimes we don't want A = A + 1 to be the same > as > > A += 1. Making a copy is sometimes what you want. Having both versions > > lets you control when you make a copy rather than being forced to always > > make a copy or always not. > > > we don't want A = A + 1 to be the same as A += 1 " > > This sounds quite frustrating for me, what else could this be but A += 1? > Would I want a copy with same name? How that will make sense? > > Making a copy is how it works now, ok. But even now there are differences, > eg: > A = B [1:10] > is not a copy but a reference creation operation. So there are > different meanings of =. > > > > >> > >> I never would. Also I think to implement this syntax would be almost > >> trivial, it should > >> just take the A = A part and do the rest as usual. > > > > > > The Python language doesn't allow it. numpy can only work with the > > information provided to it be the language, and the information needed > to do > > that sort of thing is not provided to classes. Nor should it in my > opinion, > > this is one of those fundamental operations that I think absolutely must > be > > consistent. > > > > What you are talking about is an enormous backwards-compatibility > break. It > > is simply not going to happen, it would break every mutable class. > > > > I don't want to break anything, no. I don't have such deep knowledge of all > cases, but that is very interesting how actually it must break > *everything*. Say I > scan through lines and find "= " and then see that on the left is an numpy > array, which already indicates a unusual variable. > And if look on the right side and first argument is again A, cannot I > decide to make an exception and redirect it to A.__add__() or what > must be there. > It does already do an overloading of all these "+=" for numpy. > so what will fundamentally break everything I am not sure. > > > And you still have not provided any reason we should want to do it this > way. > > If let alone numpy for now and take only numeric Python variables, I > find the += syntax > very bad readable. I must literally stop and lean towards the monitor > to decipher > the line. And I am not even telling that you *cannot* do it with all > possible operations: > there are standard math, ORing ,XORing, conditionals, bitshifting etc. > Is not this obvious? > And what other reason apart from readability there can be at all? > Again, how about TOOWTDI principle? some write a = a + 1, some a += 1 ? > It does not add any pleasure for reading code. > > > >> Most of problems with function-style equations come from limitations > >> of representation > >> so for Courier font for examples the brackets are too small and makes > >> it hard to read. > >> With good font and rendering there no such problems. Some equation > editors > >> allow even different sized brackets - the outer extend more and more > >> when I add nested equations, so it looks way better. > > >You shouldn't be using a variable-width font for coding anyway. > >That is going to cause all sorts of problems (indentation not matching > up, for example). You should use a fixed-width font. > > No no, I should not use monowidth fonts and nobody ever should. Most > of those "indentation" problems are just though-out, > All you need is IDE with good tabulation support. The readability > difference between monowidth and real fonts > is *huge*. Currently I am forced to use monowidth, since it is VIM's > limitation and it bothers me a lot. > Exception is tables with numbers, but those should not be inputed in > same manner as strings, it is other medium type. > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 14 18:34:58 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 14 Nov 2016 23:34:58 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 14 November 2016 at 22:13, Mikhail V wrote: >> we don't want A = A + 1 to be the same as A += 1 " > > This sounds quite frustrating for me, what else could this be but A += 1? > Would I want a copy with same name? How that will make sense? >>> A = [1,2,3] >>> B = A >>> A = A + [1] >>> A [1, 2, 3, 1] >>> B [1, 2, 3] >>> A = [1,2,3] >>> B = A >>> A += [1] >>> A [1, 2, 3, 1] >>> B [1, 2, 3, 1] For a mutable class, A = A + something is fundamentally different from A += something. Before proposing fundamental changes to Python's semantics, please make sure that you at least understand those semantics first, and explain why your proposal is justified. There are a lot of people on this list, and the cumulative time spent reading your posts is therefore quite significant. You owe it to all those readers to ensure that your proposals are at least feasible *in the context of the Python language*. Paul From mikhailwas at gmail.com Mon Nov 14 21:08:21 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 15 Nov 2016 03:08:21 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 15 November 2016 at 00:34, Paul Moore wrote: > On 14 November 2016 at 22:13, Mikhail V wrote: >>> we don't want A = A + 1 to be the same as A += 1 " >> >> This sounds quite frustrating for me, what else could this be but A += 1? >> Would I want a copy with same name? How that will make sense? > >>>> A = [1,2,3] >>>> B = A >>>> A = A + [1] >>>> A > [1, 2, 3, 1] >>>> B > [1, 2, 3] >>>> A = [1,2,3] >>>> B = A >>>> A += [1] >>>> A > [1, 2, 3, 1] >>>> B > [1, 2, 3, 1] > > For a mutable class, A = A + something is fundamentally different from > A += something. > > Before proposing fundamental changes to Python's semantics, please > make sure that you at least understand those semantics first, and > explain why your proposal is justified. There are a lot of people on > this list, and the cumulative time spent reading your posts is > therefore quite significant. You owe it to all those readers to ensure > that your proposals are at least feasible *in the context of the > Python language*. > > Paul Ok I am calmed down already. But how do you jump to lists already? I started an example with integers. I just want to increment an integer and I don't want to see any += in my code, it strains me. And that is exact reason I hate C syntax. Then Todd started about Numpy arrays. Ok, I just commented what I find for clearer syntax with array increment. And now lists, mutable classes... I don't use classes in my programs. I could propose something but how, if we mix everything in one big pile? Then my proposal is to make typed variables first, so I could at least consider use cases. Mikhail From klahnakoski at mozilla.com Mon Nov 14 21:57:39 2016 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Mon, 14 Nov 2016 21:57:39 -0500 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: <278678d1-1624-41e4-646e-cda4bb24d251@mozilla.com> It would be nice if ....coalesce(EXPR1, EXPR2, EXPR3) Evaluated the N+1 argument only if the Nth evaluated to None. Of course this may break the general patterns in the Python language. Maybe we can fake it by wrapping the expressions in lambdas: ....coalesce(lambda: EXPR1(), lambda EXPR2(), lambda EXPR3()) and defining a `coalesce` as def coalesce(*args): ....for a in args: ........a_val=a() ........if a_val is not None: ............return a_val ....return None Making a coalesce call looks painful, but allowing the called function to control the evaluation of its parameters may be useful. Suppose we can pass methods to functions; using generators to do so: Let & refer to lazy-evaluated parameters: ....def coalesce(&a, &b): ........if a is None: ............return b ........else: ............return a .... ....c = coalesce(expr1(), expr2()) Would be converted to : ....def coalesce(a, b): ........a = yield ........a_val = a() ........if a_val is None: ............b = yield ............b_val = b() ............return b_val ........else: ............return a_val .... ....exprs = [expr1, expr2] ....temp = coalesce() ....for e in exprs: ........try: ............c = temp.next(e) ........except StopIteration: ............break Or, even better... ....def coalesce(*&args): ........for a_val in args: ............if a_val is not None: ................return a_val ........return None .... ....c = coalesce(expr1(), expr2()) Gets converted to ....def coalesce(*args): ........for a in args: ............a_val = a() ............if a_val is not None: ................return a_val ........return None .... ....exprs = [expr1, expr2] ....temp = coalesce() ....for e in exprs: ........try: ............c = temp.next(e) ........except StopIteration: ............break ...or something like that. I can not think of other reasons for this type of expansion; maybe logical `and` can be given a magic method: "__logand__": ....def __logand__(self, &other): ........if self: ............return True ........return o .... ....c = my_object and some_other() which has a combination of immediately-evaluated parameters, and lazy-evaluated parameters: class MyClass(object): ....def __logand__(self): ........if self: ............yield True ............return ........other = yield ........return other() .... ....exprs = [some_other] ....temp = MyClass.__logand__(my_object) ....for e in exprs: ........try: ............c = temp.next(e) ........except StopIteration: ............break I hope that the acrobatics shown here might be easier to implement at a lower level; where in-line generator code collapses to simple branched logic. On 11/13/2016 1:51 AM, Nick Coghlan wrote: > > At that point, if we did decide to offer a builtin instead of > dedicated syntax, the option I'd argue for is actually SQL's > "coalesce": coalesce(EXPR1) else coalesce(EXPR2) else EXPR3 Yes, it's > computer science jargon, but the operation itself is an odd one that > doesn't really have an established mathematical precedent or > grammatical English equivalent. Cheers, Nick. From boekewurm at gmail.com Mon Nov 14 21:59:28 2016 From: boekewurm at gmail.com (Matthias welp) Date: Tue, 15 Nov 2016 03:59:28 +0100 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: >> For a mutable class, A = A + something is fundamentally different from >> A += something. >> >> Paul > Ok I am calmed down already. > But how do you jump to lists already? I started an example with integers. > I just want to increment an integer and I don't want to see any += in my code, > it strains me. And that is exact reason I hate C syntax. > Then Todd started about Numpy arrays. Ok, I just commented what I find for > clearer syntax with array increment. > And now lists, mutable classes... I don't use classes in my programs. > I could propose something but how, if we mix everything in one big pile? > Then my proposal is to make typed variables first, so I could > at least consider use cases. > Mikhail Mikhail, what Paul probably means here is that python 'operators' are actually 'syntactic sugar' for functions (it is not recommended to call these functions directly, but it is possible): e.g. if you use 'a = a + 1', python under the hood interprets it as 'a = a.__add__(1)', but if you use 'a += 1' it uses 'a.__iadd__(1)'. All python operators are implementable under functions, which is part of the spec. For integers or strings, that may not be as visible to the end user, but if you want to change the behaviour of operators, please first look at the docs [1] and try to understand them, and understand why they exists (inconclusive examples: [2). I hope that this clears up the misconceptions you may have about how python operators work. -Matthias [1]: https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types [2]: e.g. to make your library more efficient (like the operators in numpy for more efficient array operations), or adding operator functionality for numerical classes (like the decimal.Decimal class). From python at mrabarnett.plus.com Mon Nov 14 22:40:12 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 15 Nov 2016 03:40:12 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: <00927579-1dda-d1ad-c417-230ee3e86f4e@mrabarnett.plus.com> On 2016-11-15 02:59, Matthias welp wrote: [snip] > e.g. if you use 'a = a + 1', python under the hood interprets it as > 'a = a.__add__(1)', but if you use 'a += 1' it uses 'a.__iadd__(1)'. FTR, 'a += 1' is interpreted as 'a = a.__iadd__(1)'. From jelle.zijlstra at gmail.com Mon Nov 14 22:55:43 2016 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Mon, 14 Nov 2016 19:55:43 -0800 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: <00927579-1dda-d1ad-c417-230ee3e86f4e@mrabarnett.plus.com> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> <00927579-1dda-d1ad-c417-230ee3e86f4e@mrabarnett.plus.com> Message-ID: 2016-11-14 19:40 GMT-08:00 MRAB : > On 2016-11-15 02:59, Matthias welp wrote: > [snip] > > e.g. if you use 'a = a + 1', python under the hood interprets it as >> 'a = a.__add__(1)', but if you use 'a += 1' it uses 'a.__iadd__(1)'. >> > > FTR, 'a += 1' is interpreted as 'a = a.__iadd__(1)'. Or to be even more pedantic, `a = type(a).__iadd__(a, 1)`. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at rcfox.ca Tue Nov 15 02:13:25 2016 From: ryan at rcfox.ca (Ryan Fox) Date: Tue, 15 Nov 2016 02:13:25 -0500 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: > > No, the conditional branching would be based on exists.__bool__ (or, > in the current working draft, is_not_none.__bool__), and that would be > "0 is not None", which would be True and hence short-circuit. > > > `__then__` is responsible for *unwrapping* the original value from the > circuit breaker when it short-circuits: it's what allows the overall > expression to return "0", even though the truth check is done based on > "0 is not None". > I see. I somehow missed that exists was a wrapping class rather than an evaluating function. --- Should you be concerned about cases where a CircuitBreaker class is used without the `else` operator? For example, if a user accidentally does something like: x = exists(input_value) or default_value If input_value is not None, the user will get an `exists` object instead of their input value. I'm worried that the distinction between `or` and `else` will not be obvious. It seems like `else` will effectively just be `or`, but with more functionality. --- I'm also still not convinced about the reasons to avoid implementing this on `or`. I'll address the points from the rationale: > defining a shared protocol for both and and or was confusing, as __then__ was the short-circuiting outcome for or , while__else__ was the short-circuiting outcome for and I wonder: Could the protocol be defined in terms of `or`, with DeMorgan's law applied behind the scenes in the case of `and`? ie: existing(x) and existing(y) => missing(y) or missing(x) > the and and or operators have a long established and stable meaning, so readers would inevitably be surprised if their meaning now became dependent on the type of the left operand. Even new users would be confused by this change due to 25+ years of teaching material that assumes the current well-known semantics for these operators With basic pass-through implementations for __then__ and __else__ attached to all classes by default, and the existing __bool__, it seems like `or` would continue to function in the same way it currently does. There are plenty of current dunder methods that are already redefined in ways that might confuse people: % on strings, set operators, etc. > Python interpreter implementations, including CPython, have taken advantage of the existing semantics of and and or when defining runtime and compile time optimisations, which would all need to be reviewed and potentially discarded if the semantics of those operations changed I can't really speak to any of this, not being familiar with the internals of any implementation. Though, it might work out that some of the code for handling `and` and `or` could be thrown out, since those operators would be transformed into conditional expressions. I very much understand the desire to not break working, optimized implementations. However, this feels a little flimsy as a reason for introducing new syntax. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Nov 15 02:52:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Nov 2016 17:52:15 +1000 Subject: [Python-ideas] Built-in function to run coroutines In-Reply-To: <499c0d2c-efb8-14c1-e2e8-b943709f39d3@gmail.com> References: <9099c61e-c464-55ab-738a-ea7727ba7719@gmail.com> <2ba7f318-e93f-0826-2184-dcd7989a2966@mail.de> <499c0d2c-efb8-14c1-e2e8-b943709f39d3@gmail.com> Message-ID: On 15 November 2016 at 04:39, Yury Selivanov wrote: > On 2016-11-14 1:35 PM, Sven R. Kunze wrote: >> >> What about making "run" an instance method of coroutines? > > That would require coroutines to be aware of the loop that is running them. > Not having them aware of that is what makes the design simple and allows > alternatives to asyncio. It also helps minimise the additional work needed for Tornado, Cython, etc to provide their own coroutine implementations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 15 03:18:37 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Nov 2016 18:18:37 +1000 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: On 15 November 2016 at 17:13, Ryan Fox wrote: > I'm worried that the distinction between `or` and `else` will not be > obvious. It seems like `else` will effectively just be `or`, but with more > functionality. The next draft makes that explicit: "and", "or" and PEP 505's "??" would all just be syntactic sugar for "else" combined with particular circuit breakers. > I'm also still not convinced about the reasons to avoid implementing this on > `or`. I'll address the points from the rationale: > >> defining a shared protocol for both and and or was confusing, as __then__ >> was the short-circuiting outcome for or , while__else__ was the >> short-circuiting outcome for and > > I wonder: Could the protocol be defined in terms of `or`, with DeMorgan's > law applied behind the scenes in the case of `and`? > > ie: > existing(x) and existing(y) => missing(y) or missing(x) The next draft reverts to the symmetric API proposal from pre-publication drafts, so this part of the rationale is gone. >> the and and or operators have a long established and stable meaning, so >> readers would inevitably be surprised if their meaning now became dependent >> on the type of the left operand. Even new users would be confused by this >> change due to 25+ years of teaching material that assumes the current >> well-known semantics for these operators > > With basic pass-through implementations for __then__ and __else__ attached > to all classes by default, and the existing __bool__, it seems like `or` > would continue to function in the same way it currently does. Except it would be a *lot* slower (as in, an-order-of-magnitude slower, not a-few-percent slower). The forced call to __bool__() in the second example below hints at the likely cost of bypassing the existing optimised fast paths for conditions that produce a boolean result: $ python -m perf timeit -s "lhs = True; rhs = False" "lhs and rhs" ..................... Median +- std dev: 16.6 ns +- 3.3 ns $ python -m perf timeit -s "lhs = True; rhs = False" "lhs.__bool__() and rhs" ..................... Median +- std dev: 113 ns +- 18 ns Accordingly, we want interpreter implementations to be able to readily distinguish between "normal" conditions (which would continue to just be evaluated as boolean values in order to determine which branch to take) and circuit breakers (which want to be able to further influence the result *after* the interpreter has determined which branch to evaluate) > There are plenty of current dunder methods that are already redefined in > ways that might confuse people: % on strings, set operators, etc. None of those cases introduced a protocol method into an operation that didn't previously use one - they instead borrowed existing protocol driven operators for their own purposes. >> Python interpreter implementations, including CPython, have taken >> advantage of the existing semantics of and and or when defining runtime and >> compile time optimisations, which would all need to be reviewed and >> potentially discarded if the semantics of those operations changed > > I can't really speak to any of this, not being familiar with the internals > of any implementation. Though, it might work out that some of the code for > handling `and` and `or` could be thrown out, since those operators would be > transformed into conditional expressions. That's exactly the kind of outcome we *don't* want. > I very much understand the desire to not break working, optimized > implementations. However, this feels a little flimsy as a reason for > introducing new syntax. The language-design-driven reason is that "and" and "or" are terms drawn from boolean logic, and hence can reasonably be expected to implement that. We absolutely *could* say that they don't *necessarily* implement boolean logic anymore, just as mathematical operators don't necessarily represent the traditional arithmetic operations, but I'd personally prefer the status quo to that possible outcome. The first draft of PEP 532 *did* propose doing things that way, though: https://github.com/python/peps/commit/3378b942747604be737eb627df085979ff61b621 I never posted that version here, as I didn't really like it myself, and had in fact already rewritten it to the current proposal by the time I merged it into the main PEPs repo: https://github.com/python/peps/commit/8f095cf8c0ccd4bf770e933a21e04b37afc53cfe :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Nov 15 04:46:01 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Nov 2016 09:46:01 +0000 Subject: [Python-ideas] Reverse assignment operators (=+, =-, =*, =/, =//, =**, =%) In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: On 15 November 2016 at 02:59, Matthias welp wrote: > Mikhail, what Paul probably means here is that python 'operators' are actually > 'syntactic sugar' for functions (it is not recommended to call > these functions directly, but it is possible): More specifically, the meaning of the syntax a = a + b does not depend on the types of the operators - it always means a = a.__add__(b) (with some added complexity for __radd__). So you can't make it do something different "for integers" because at this stage of interpretation, Python hasn't looked at the types of the objects in the expression. Hence my example, which uses lists to show an "obvious" case where a = a + b and a += b differ. It's harder to show a convincing example when a has an immutable type like int, because the fundamental difference between the 2 constructs is how they treat mutable values. If you're proposing a = a + b to introspect at runtime the type of a, and produce different bytecode depending on the answer, you're proposing a fundamental change to the runtime semantics of Python (such that the resulting language is arguably not even Python any more). That's not to say it can't be done, just that it's not in scope for "ideas about new features for Python" in any practical sense. Paul From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 15 08:04:02 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 15 Nov 2016 22:04:02 +0900 Subject: [Python-ideas] How we think about change [was: Reverse assignment operators...] In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> Message-ID: <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> Mikhail V writes: > But how do you jump to lists already? Because "+" and "+=" are operators which may be defined for any objects. Paul explained *why* he chose to to do that elsewhere. My point is that the semantics "a += b" *is* "type(a).__iadd__(a, b)" is true for all objects. It is a fundamental consistency in Python. It may make what one might consider odd behavior fall into its proper place. Thus, in explaining this kind of thing it is often useful (YMMV) to "jump" to a different type that supports the same behavior to see how a proposed change can cause inconsistency. > I started an example with integers. I just want to increment an > integer and I don't want to see any += in my code, it strains > me. If you *really* want to limit this to *integers* in *your* code, that's easy enough. Just don't use it. If by "my code" you mean "code I might have to read"[1], you're just out of luck, even for the very special case of integers. (1) Guido himself preferred *not* to have augmented assignment way back when, but bowed to popular demand. You are in a tiny minority, opposing the preference of a large and loud majority. (2) Guido could remove those operators, but it would break a lot of existing code that uses them, and that would not be acceptable.[2] (3) It would be a gratuitous special case, a fundamental inconsistency in Python semantics. So it's not just calming down. You need to understand that if you want to use a popular language with batteries included and a huge range of battle-tested very useful add-ons, you're going to have to accept that you will often be reading code that you would refuse to write unless forced to by court order. You see, you're wasting your own time here by trying to explain. It's just your preference, which in this matter is as valid as any other *so no explanation is necessary*. But the opinion that "+=" for integers is useful and readable was the majority when it was installed, and backward compability rules out changing it now. And I really don't think you have to worry about "=+". Footnotes: [1] And IMHO it is Pythonic to interpret "in my code" that way. Ie, it is Pythonic to be readable to as many Python developers as possible. But it would smooth discussion to distinguish the "my code" that you *write* from the "my code" that you *read* in your posts. (I guess English is a second language for you, it may not be worth it and *we* will adapt. But it does help, if you would like to try.) [2] Theoretically, he has the power. But he also has promised no such backward incompatible changes "ever again", so practically impossible. Yes, IMO this would be a breaking change big enough to compare to Unicode in 3.0. From chris.barker at noaa.gov Tue Nov 15 11:52:56 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 15 Nov 2016 08:52:56 -0800 Subject: [Python-ideas] How we think about change [was: Reverse assignment operators...] In-Reply-To: <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> Message-ID: <-8105712888627315773@unknownmsgid> > Because "+" and "+=" are operators which may be defined for any > objects. Paul explained *why* he chose to to do that elsewhere. My > point is that the semantics "a += b" *is* "type(a).__iadd__(a, b)" is > true for all objects. Well, yes. But it is defined in particular ways in the built in types and the stdlib. And those ways are consistent-- I.e. They do the same thing as __add__. And we REALLY wouldn't want it any other way. But the __i*__ operators are a bit of a wart -- they mean different things for mutable and immutable types. This is because they were added to satisfy two use-cases: Syntactic sugar for common and simple operations like incrementing an integer. Compact syntax for in-place operations. As numbers are immutable, you can't have in-place operations. Period. So it was handy to use the same notation for both. And I don't think anyone wanted to add TWO new sets of operators. This dual use does cause confusion occasionally, but not that often in practice. -CHB From dmoisset at machinalis.com Tue Nov 15 12:34:22 2016 From: dmoisset at machinalis.com (Daniel Moisset) Date: Tue, 15 Nov 2016 17:34:22 +0000 Subject: [Python-ideas] Proposal: Tuple of str with w'list of words' In-Reply-To: References: Message-ID: On 12 November 2016 at 17:01, Gary Godfrey wrote: > Hi, > > This is a little more readable, but still a bit ugly. What I'm proposing > here is: > > mydf = df[ w'field1 field2 field3' ] > > This would be identical in all ways (compile-time) to: > > mydf = df[ ('field1', 'field2', 'field3') ] > If using a tuple as an index expression, wouldn't it be ok for you to use: mydf = df['field1', 'field2', 'field3'] ? That should be equivalent to the second example, but without the doble bracketing Best, D. -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Tue Nov 15 16:02:32 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 15 Nov 2016 22:02:32 +0100 Subject: [Python-ideas] How we think about change [was: Reverse assignment operators...] In-Reply-To: <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> Message-ID: On 15 November 2016 at 14:04, Stephen J. Turnbull wrote: > > Mikhail V writes: > > But how do you jump to lists already? > Thus, in explaining this kind of thing it is often useful > (YMMV) to "jump" to a different type that supports the same behavior > to see how a proposed change can cause inconsistency. First, thank you for the input, and time spent to clear things here. I must admit I cannot "jump" for this time. Not only because I don't know inner nature of *all* possible implementions and cannot analyse them all in coverable amount of time, but also because I cannot draw logical connection between "appending to a list" and "incrementing an integer value" expressings, even if for some reason we want to use same syntax for both. Otherwise it would make extremely hard to contribute ideas for syntax improvement from not "python-dev" people. Of course only if one wants to recieve such ideas. > > I started an example with integers. I just want to increment an > > integer and I don't want to see any += in my code, it strains > > me. > Just don't use it. > If by "my code" you mean "code I might have to read"[1], you're just out > of luck, even for the very special case of integers. I tend to use word *my* since I can carry responsibility only for things I know from my experience. I hear often *we* in western political TV shows and this sounds not so good for my ears. But it is more of cultural thing I think. And I must read not only *my own* code of course. I personally never use += operators if I can. > (1) Guido himself preferred *not* to have augmented assignment way back when, > but bowed to popular demand. You are in a tiny minority, opposing the > preference of a large and loud majority. (2) Guido could remove those > operators, but it would break a lot of existing code that uses them, > and that would not be acceptable.[2] Good to know that at least this being discussed, but not so good that majority wins. As for Python's numbers I personally would without hesitation remove the += and Co. I was jus curious how bad would it "break" things? Currently I cannot run my 2.7 code with 3.x Python, so I need to convert it. Probably not so obvious, how to convert the code with numeric operators automatically, since there are no type annotations in code... All in all, the issues of compatibility and majority with habits are there, I just think of "python-ideas" merely as discussion about possible future syntax improvements, simply because it relates to my specialization. But I also try to learn the technical part. To summarize things: I may be well losing my time explaining why += is bad, that is true. It is next to impossible to explain it, unless one can just *see* it. But apart from minority/majority, there is only one truth, so I would say, if "we" could consider such a slight rearrangement of how we *call* the things: Now I see many people call += a "syntactic sugar", and for me it is rather a "syntactic defect", so it is something on the opposite. One can say it is only opinion, ok, but if at least a pair of people will stop writing +=, then I could say, my mission was not a fail. Mikhail From mertz at gnosis.cx Tue Nov 15 22:56:30 2016 From: mertz at gnosis.cx (David Mertz) Date: Tue, 15 Nov 2016 19:56:30 -0800 Subject: [Python-ideas] How we think about change [was: Reverse assignment operators...] In-Reply-To: References: <8a14d1ef-a9b3-0ae1-d454-4751d9b208b3@gmail.com> <20161114111627.GR3365@ando.pearwood.info> <22571.1986.527135.127692@turnbull.sk.tsukuba.ac.jp> Message-ID: The "just see it" isn't likely to go very far. My impression is that maybe 5% of people find '+=' and friends confusing or bothersome. Probably 15% are neutral (including me). And 80% think 'a += 1' is simply MORE readable and more easily understood than 'a = a+1'. Folks who come from C-family languages especially like the augmented assignment. It's very common for the inplace and binary operators to do the same thing. For those cases, it really is just syntax sugar. But enough importent libraries like NumPy make a distinction tray a highly value having __iadd__() and friends. On Nov 15, 2016 1:03 PM, "Mikhail V" wrote: > On 15 November 2016 at 14:04, Stephen J. Turnbull > wrote: > > > > Mikhail V writes: > > > > But how do you jump to lists already? > > > Thus, in explaining this kind of thing it is often useful > > (YMMV) to "jump" to a different type that supports the same behavior > > to see how a proposed change can cause inconsistency. > > First, thank you for the input, and time spent to clear things here. > > I must admit I cannot "jump" for this time. Not only because > I don't know inner nature of *all* possible implementions and > cannot analyse them all in coverable amount of time, > but also because I cannot draw logical connection between > "appending to a list" and "incrementing an integer value" expressings, > even if for some reason we want to use same syntax for both. > Otherwise it would make extremely hard to contribute ideas for > syntax improvement from not "python-dev" people. > Of course only if one wants to recieve such ideas. > > > > I started an example with integers. I just want to increment an > > > integer and I don't want to see any += in my code, it strains > > > me. > > > Just don't use it. > > If by "my code" you mean "code I might have to read"[1], you're just out > > of luck, even for the very special case of integers. > > I tend to use word *my* since I can carry responsibility only for things > I know from my experience. I hear often *we* in western political TV shows > and this sounds not so good for my ears. But it is more of cultural > thing I think. > And I must read not only *my own* code of course. > I personally never use += operators if I can. > > > (1) Guido himself preferred *not* to have augmented assignment way back > when, > > but bowed to popular demand. You are in a tiny minority, opposing the > > preference of a large and loud majority. (2) Guido could remove those > > operators, but it would break a lot of existing code that uses them, > > and that would not be acceptable.[2] > > Good to know that at least this being discussed, but not so good > that majority wins. As for Python's numbers I personally would without > hesitation > remove the += and Co. > I was jus curious how bad would it "break" things? Currently I cannot run > my 2.7 > code with 3.x Python, so I need to convert it. Probably not so obvious, > how to convert the code with numeric operators automatically, > since there are no type annotations in code... > > All in all, the issues of compatibility and majority with habits > are there, I just think of "python-ideas" merely as discussion about > possible future syntax improvements, simply because it relates to my > specialization. > But I also try to learn the technical part. > > To summarize things: > > I may be well losing my time explaining why += is bad, that is true. > It is next to impossible to explain it, unless one can just *see* it. > But apart from minority/majority, there is only one truth, so I would say, > if "we" could consider such a slight rearrangement of how we *call* the > things: > > Now I see many people call += a "syntactic sugar", and for me it is > rather a "syntactic defect", so it is something on the opposite. > One can say it is only opinion, ok, but if at least a pair of people will > stop writing +=, then I could say, my mission was not a fail. > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Wed Nov 16 03:51:14 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 16 Nov 2016 09:51:14 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] Message-ID: On 15 November 2016 at 10:46, Paul Moore wrote: > If you're proposing a = a + b to introspect at runtime the type of a, > and produce different bytecode depending on the answer, you're > proposing a fundamental change to the runtime semantics of Python > (such that the resulting language is arguably not even Python any > more). That's not to say it can't be done, just that it's not in scope > for "ideas about new features for Python" in any practical sense. I thank you all again for informing me about assignment operator and its further consequences. I read accurately and try to absorb the information, still however I feel like I need just a little bit of more understanding in this particular case. So I can do better if we imagine, just imagine, such a scenario: 1. += and friends are now not allowed for integers, but all _add()_, _iadd()_ are still of course there, it does not concern me. 2. Somebody says: now I want to write something like: halfofanextremelylongvariable = halfofanextremelylongvariable + 1 3. I come up, scratch my head and say: Ok, now we can write that as: halfofanextremelylongvariable = self + 1 So, considering previous comments, there was an impression that I wanted to shake the fundament of all Python world or something. But is it necesserily so? So I don't know how the Python parser works exactly, but why cannot the above roughly speaking "translated", e.g.: x = self + 1 would mean exactly the same that was before x += 1 and call __iadd()__ in turn. Isn't it a very simple thing technically seen? Since augmented assignment is always applied to exactly one argument I believe (correct me if I am wrong), would not it be simple and non-ambigios operation at parsing stage? So in other words sequence " = self + " is same as sequence " += " was. Another example, lets take a Numpy array for example. So it was also stated that += is vital for doing in-place sum. Now I say, we write it in such way, having "A+=1" in mind: exec A + 1 since again, += can be only applied only on one operand, cannot I just say that a sequence "exec {1} + {2}" is what sequence "{1}+={2}" was? What semantics it will fundamentally break or so hard to implement? I was actually trying to give similar ideas in previous comments, but I was not precise in my ideas, so it came to misunderstanding, or it can be that I'm still missing something very important here. Mikhail From p.f.moore at gmail.com Wed Nov 16 04:27:12 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Nov 2016 09:27:12 +0000 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On 16 November 2016 at 08:51, Mikhail V wrote: > What semantics it will fundamentally break or so hard to implement? I'm afraid I don't have time at the moment to fully review your email, but my first impression is that you are proposing that the name "self" be treated specially. It's going to be very difficult to convince people that this is a good idea. In the first instance, if you're talking about "self" being a keyword, that would break the current usage of "self" as a normal variable name, which is used by convention in all class methods to represent the current object. If you're not aware, in class A: def method(self): ... the use of self as the variable name is *convention*. You could use any other name just as well. If self were a keyword, it couldn't be used here. So making "self" a keyword would be a major change. In addition, consider code like this: class A(int): def method(self): myvar = 1 myvar = self + 1 return myvar foo = A(5) foo.method() This is legitimate code at the moment. It sets "myvar" to 6 (self is an integer subclass with value 5, which the code adds 1 to) and returns that. Your proposal would change that to return 2. The problem here is that when considering language changes, you can't just consider code that you consider "correct". You have to take account of the fact that other people may be writing code that is to your eyes bizarre and unmaintainable - but if it does what they want it to, and it works according to the language specification, they have a right to expect that the behaviour won't get changed (at least not without due consideration of the backward compatibility process). So proposals that change the behaviour of currently working code have to be *very* careful to consider all edge cases. This is probably the biggest hurdle for people proposing changes to Python to get over. Backward compatibility isn't pleasant to deal with - it's frustrating and demotivating to have a good idea blocked because code that seems senseless "might break". But that's part of the price we have to pay for Python being a hugely popular language. Hope this is useful. Paul From roland at catalogix.se Wed Nov 16 06:51:41 2016 From: roland at catalogix.se (Roland Hedberg) Date: Wed, 16 Nov 2016 13:51:41 +0200 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? Message-ID: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> Hi! Why am I bringing this up: Security is hard ! Implementing a standard correctly is not easy. I know about the later because the last 2 years I?ve been involved in certifying OpenID Connect Provider instances. Lately I?ve been doing the same for OpenID Connect Relying Party libraries. All of what I?ve done in written in Python and on Github. Regarding the first opinion that has been shown time and time again so I won?t go into that here. Now, voices has been raise within the OpenID Foundation that it would pick a number of implementations, one per language, and stamp them with a sign of approval. Those implementations would all be thoroughly tested for compliance and usability before approved. My Python implementation (https://github.com/rohe/pyoidc) is probably the forerunner when it comes to being the chosen Python implementation. It?s been around for a number of years and it?s the basis for the test tools. Which means, it has been thoroughly tested by many independent parties. My question to you is if it would be possible to get an OAuth2/OIDC implementation like mine to be part of the Python standard distribution. I realise that I will have to rewrite parts of pyoidc because presently it uses modules (for instance pycryptdome and requests) that are not part of the standard distribution. The bottom line is of course that it would benefit the community to have a high quality OAuth2/OIDC implementation within easy reach. ? Roland From cory at lukasa.co.uk Wed Nov 16 07:50:28 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 16 Nov 2016 12:50:28 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> Message-ID: <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> > On 16 Nov 2016, at 11:51, Roland Hedberg wrote: > > The bottom line is of course that it would benefit the community to have a > high quality OAuth2/OIDC implementation within easy reach. I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? Cory From ncoghlan at gmail.com Wed Nov 16 08:16:06 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 16 Nov 2016 23:16:06 +1000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> Message-ID: On 16 November 2016 at 22:50, Cory Benfield wrote: > >> On 16 Nov 2016, at 11:51, Roland Hedberg wrote: >> >> The bottom line is of course that it would benefit the community to have a >> high quality OAuth2/OIDC implementation within easy reach. > > I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? >From an architectural point of view, I'd also note that anyone trying to do modern web and network service development *without* the ability to use additional components beyond the standard library toolkit has many more problems than just the lack of a readily accessible OAuth2/OIDC implementation (such as the absence of 'requests' itself). I do think it could be useful for you to ask the requests developers if they'd be willing to explicitly recommend a particular approach to implementing OIDC atop requests and provide a pointer from their documentation. Searching on Google for "python oidc" indicates both "pip install oidc" and "pip install oic" are available (with the latter being the case discussed here), but of the two, only yours appears to provide API usage documentation. The protocol walkthrough at http://pyoidc.readthedocs.io/en/latest/howto/rp.html seems like it would be particularly useful to many folks as a hands-on introduction to the steps involved in OIDC based client authentication. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From roland at catalogix.se Wed Nov 16 08:30:19 2016 From: roland at catalogix.se (Roland Hedberg) Date: Wed, 16 Nov 2016 15:30:19 +0200 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> Message-ID: > On 16 Nov 2016, at 14:50, Cory Benfield wrote: > > >> On 16 Nov 2016, at 11:51, Roland Hedberg wrote: >> >> The bottom line is of course that it would benefit the community to have a >> high quality OAuth2/OIDC implementation within easy reach. > > I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? Basically, I think it?s a matter of visibility. If someone tells you you have to add OIDC RP capabilities to your service what do you do. If the needed batteries are already included in Python it?s easy, it?s there. If it isn?t then the best case scenario is that you find all the implementations there is and based on objective evaluations (like the OIDF standards compliant tests) chose the ?best? one. Is that the most common scenario ? I doubt it. The OIDF will publish information about their preferred set of libraries but I still think there will be a substantial portion of coders we won?t reach. If you have any idea about how we could reach more coders I?m all ears. ? Roland From roland at catalogix.se Wed Nov 16 08:32:21 2016 From: roland at catalogix.se (Roland Hedberg) Date: Wed, 16 Nov 2016 15:32:21 +0200 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> Message-ID: <190B1F83-C298-423E-9A5D-D46865A3D0F5@catalogix.se> > On 16 Nov 2016, at 15:16, Nick Coghlan wrote: > > On 16 November 2016 at 22:50, Cory Benfield wrote: >> >>> On 16 Nov 2016, at 11:51, Roland Hedberg wrote: >>> >>> The bottom line is of course that it would benefit the community to have a >>> high quality OAuth2/OIDC implementation within easy reach. >> >> I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? > > From an architectural point of view, I'd also note that anyone trying > to do modern web and network service development *without* the ability > to use additional components beyond the standard library toolkit has > many more problems than just the lack of a readily accessible > OAuth2/OIDC implementation (such as the absence of 'requests' itself). > > I do think it could be useful for you to ask the requests developers > if they'd be willing to explicitly recommend a particular approach to > implementing OIDC atop requests and provide a pointer from their > documentation. That?s a very good idea! I?ll act on it. > Searching on Google for "python oidc" indicates both > "pip install oidc" and "pip install oic" are available (with the > latter being the case discussed here), but of the two, only yours > appears to provide API usage documentation. The protocol walkthrough > at http://pyoidc.readthedocs.io/en/latest/howto/rp.html seems like it > would be particularly useful to many folks as a hands-on introduction > to the steps involved in OIDC based client authentication. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pavol.lisy at gmail.com Wed Nov 16 08:38:59 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Wed, 16 Nov 2016 14:38:59 +0100 Subject: [Python-ideas] PEP 532: A circuit breaking operator and protocol In-Reply-To: References: Message-ID: On 11/12/16, Mark E. Haase wrote: > 1. It is easier to Google a name. E.g., Google "c# ??" and you'll get > nothing related to null coalescing in c#". ("C# question marks" does find > the right content, however.) python has nice (*) help system and would have help('??')... (where we could get better keywords to search on web - for example "None coalescing operator") (*) although it could be better: 1. for example help('**') show nothing about unpacking. help('+') show info about operator precedence and not about +. 2. there is not info about dunder methods ... From cory at lukasa.co.uk Wed Nov 16 08:55:20 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 16 Nov 2016 13:55:20 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> Message-ID: <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> > On 16 Nov 2016, at 13:30, Roland Hedberg wrote: > > >> On 16 Nov 2016, at 14:50, Cory Benfield wrote: >> >> I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? > > Basically, I think it?s a matter of visibility. > If someone tells you you have to add OIDC RP capabilities to your service what do you do. > If the needed batteries are already included in Python it?s easy, it?s there. Ultimately I think that this is a bit misleading. Python?s included batteries are often not the best way to add certain bits of function to your program, and Python programmers are extremely accustomed to needing to get their batteries from elsewhere. Certainly web developers are: web developers cannot even really get started without pip installing *something* unless they plan to hand-roll WSGI using wsgiref or use SimpleHTTPServer, which they overwhelmingly do not. > The OIDF will publish information about their preferred set of libraries but I still think > there will be a substantial portion of coders we won?t reach. > > If you have any idea about how we could reach more coders I?m all ears. Coders who need OIDC will go looking for it and will find their options. Ultimately, a huge number of projects haven?t suffered from being outside the standard library. Some of these are even replacements for Python?s included batteries, which means they?re competing with the ?just there? options users already have. It should be noted that I believe that Python?s standard library is already too big, and has had a tendency in the past to expand into cases that were not warranted. I think that saying that you?re worried that users won?t find your module and so it should go into the standard library is solving the wrong problem. We shouldn?t just shove useful things into the standard library because they?re useful: that leads to a massive, bloated standard library that is an enormous maintenance burden for the Python core developers who frankly have more than enough to be doing already. Instead, we should aim to solve the actual problem: how do we provide tools to allow users to find the best-in-class solutions to their problems from the third-party Python ecosystem? That there is a much harder problem, unfortunately, but I think we should aim to provide a bit of impetus towards solving it by refusing to add things to the standard library that aren?t likely to be extremely broadly useful. Cory From roland at catalogix.se Wed Nov 16 09:09:07 2016 From: roland at catalogix.se (Roland Hedberg) Date: Wed, 16 Nov 2016 16:09:07 +0200 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> Message-ID: > On 16 Nov 2016, at 15:55, Cory Benfield wrote: > > >> On 16 Nov 2016, at 13:30, Roland Hedberg wrote: >> >> >>> On 16 Nov 2016, at 14:50, Cory Benfield wrote: >>> >>> I think the core question you need to answer for this proposal is: why is ?pip install oic? not easy-enough reach? > > It should be noted that I believe that Python?s standard library is already too big, and has had a tendency in the past to expand into cases that were not warranted. I think that saying that you?re worried that users won?t find your module and so it should go into the standard library is solving the wrong problem. We shouldn?t just shove useful things into the standard library because they?re useful: that leads to a massive, bloated standard library that is an enormous maintenance burden for the Python core developers who frankly have more than enough to be doing already. Instead, we should aim to solve the actual problem: how do we provide tools to allow users to find the best-in-class solutions to their problems from the third-party Python ecosystem? I agree that is the real question. For instance, I remember someone raising at a PyCon US the concern about modules that no longer has a maintainer. That would just be the one among several things you need to know about modules you consider using. > That there is a much harder problem, unfortunately, but I think we should aim to provide a bit of impetus towards solving it by refusing to add things to the standard library that aren?t likely to be extremely broadly useful. Granted ? Roland From p.f.moore at gmail.com Wed Nov 16 09:53:34 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Nov 2016 14:53:34 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> Message-ID: On 16 November 2016 at 13:55, Cory Benfield wrote: >> If you have any idea about how we could reach more coders I?m all ears. > > Coders who need OIDC will go looking for it and will find their options. Ultimately, a huge number of projects haven?t suffered from being outside the standard library. Some of these are even replacements for Python?s included batteries, which means they?re competing with the ?just there? options users already have. I'm not a web developer as such, although I do write code that consumes web services on occasion. I don't know what OIDC is, but I do know, for example, that some services use OAuth. So I can imagine being in a situation of saying "I want to get data from a web API xxx, and it needs OAuth identification, how do I do that in Python?" Typically, the API docs are in terms of something like Javascript, with a browser UI, so don't help much for a command line script which is the sort of thing I'd be writing. In that situation, a well-known, easy to use module for OAuth in Python would be fantastic. Agreed that it could as easily be on PyPI as in the stdlib, but discoverability isn't as good with PyPI - I can scan the stdlib docs, but for PyPI I'd end up scanning Google, and what I found that way was oauthlib - I didn't see any mention of pyoidc. I can't comment on what that implies, though. In my brief search though I didn't find any sort of command line "Hello world" level example. > It should be noted that I believe that Python?s standard library is already too big, and has had a tendency in the past to expand into cases that were not warranted. I should also note that I rely heavily on the stdlib, and for a non-trivial amount of the work I do (which is one-off scripts, not full-blown applications) having to go outside the stdlib, other than for a very few select modules, is a step change in complexity. So I'm a fan of the "batteries included" approach. I don't know whether OAuth is a sufficiently common requirement to warrant going into the stdlib. My instinct is that if you're integrating it into a web app, then there's no value in it being in the stdlib as you'll already need 3rd party modules. If it's practical to use OAuth from a simple Python script (say, to integrate with a web API like github) then being in the stdlib could be a benefit. But how many people write Python scripts that use/need OAuth? I've no feel for that. Paul From ethan at stoneleaf.us Wed Nov 16 11:37:25 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Nov 2016 08:37:25 -0800 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> Message-ID: <582C8B45.7080606@stoneleaf.us> On 11/16/2016 03:51 AM, Roland Hedberg wrote: > My question to you is if it would be possible to get an OAuth2/OIDC implementation like mine > to be part of the Python standard distribution. > > I realise that I will have to rewrite parts of pyoidc because presently it uses modules > (for instance pycryptdome and requests) that are not part of the standard distribution. My concern with a rewrite is that your code is no longer thoroughly tested. Of course, this depends in large part on the completeness of your unit tests. I agree with Cory that discoverability is the problem to solve. You can help with that somewhat by seeding sites like StackOverflow with the information.* -- ~Ethan~ * SO is not very tolerant of questions looking for opinion-based answers such as software recommendations, so make sure you phrase your question as "How do I do XXX using pyoidc?" and your answer can also include other links on the OIDC subject. From mikhailwas at gmail.com Wed Nov 16 11:48:39 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 16 Nov 2016 17:48:39 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On 16 November 2016 at 10:27, Paul Moore wrote: > On 16 November 2016 at 08:51, Mikhail V wrote: >> What semantics it will fundamentally break or so hard to implement? > > I'm afraid I don't have time at the moment to fully review your email, > but my first impression is that you are proposing that the name "self" > be treated specially. It's going to be very difficult to convince > people that this is a good idea. Oh Paul, come on, pleeeeease. I am making an **example**. Write a "poo" instead or a poo pile character, whatever. You think I am an idiot and don't know that there is "self" in OOP. When I was writing that I just thought, should I make a special note that I am making it only for example, but then thought, oh that would be too pedantic. You say you have no time and write a whole page about it, so don't blame me that I take too much time from you. Mikhail From Nikolaus at rath.org Wed Nov 16 12:24:24 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 16 Nov 2016 09:24:24 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: (Mikhail V.'s message of "Wed, 16 Nov 2016 17:48:39 +0100") References: Message-ID: <87oa1fmj7b.fsf@thinkpad.rath.org> On Nov 16 2016, Mikhail V wrote: > On 16 November 2016 at 10:27, Paul Moore wrote: >> On 16 November 2016 at 08:51, Mikhail V wrote: >>> What semantics it will fundamentally break or so hard to implement? >> >> I'm afraid I don't have time at the moment to fully review your email, >> but my first impression is that you are proposing that the name "self" >> be treated specially. It's going to be very difficult to convince >> people that this is a good idea. > > Oh Paul, come on, pleeeeease. > I am making an **example**. Write a "poo" instead or a poo pile > character, ...and all of Paul's points still apply. > You think I am an idiot and don't know that there is "self" in OOP. No, that wouldn't be the reason. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From p.f.moore at gmail.com Wed Nov 16 12:28:07 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Nov 2016 17:28:07 +0000 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On 16 November 2016 at 16:48, Mikhail V wrote: > On 16 November 2016 at 10:27, Paul Moore wrote: >> On 16 November 2016 at 08:51, Mikhail V wrote: >>> What semantics it will fundamentally break or so hard to implement? >> >> I'm afraid I don't have time at the moment to fully review your email, >> but my first impression is that you are proposing that the name "self" >> be treated specially. It's going to be very difficult to convince >> people that this is a good idea. > > Oh Paul, come on, pleeeeease. > I am making an **example**. Write a "poo" instead or a poo pile character, > whatever. You think I am an idiot and don't know that there is "self" in OOP. No I don't think you're an idiot. I thought you were offering a proposal. > When I was writing that I just thought, should I make a special note > that I am making it only for example, but then thought, oh that would > be too pedantic. Well, if it's "only an example", there's nothing to comment on. You need to offer a proper proposal with the details filled in - otherwise it's not worth replying. > You say you have no time and write a whole page about it, > so don't blame me that I take too much time from you. I'm sorry. I thought I was trying to help by explaining how your proposal could be improved. My mistake - I'll keep quiet next time. Paul From rymg19 at gmail.com Wed Nov 16 12:50:23 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 16 Nov 2016 11:50:23 -0600 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On Wed, Nov 16, 2016 at 10:48 AM, Mikhail V wrote: > On 16 November 2016 at 10:27, Paul Moore wrote: > > On 16 November 2016 at 08:51, Mikhail V wrote: > >> What semantics it will fundamentally break or so hard to implement? > > > > I'm afraid I don't have time at the moment to fully review your email, > > but my first impression is that you are proposing that the name "self" > > be treated specially. It's going to be very difficult to convince > > people that this is a good idea. > > Oh Paul, come on, pleeeeease. > I am making an **example**. Write a "poo" instead or a poo pile character, > whatever. You think I am an idiot and don't know that there is "self" in > OOP. > Maybe I should just point out a sec that saying this is a great way to get people to not listen to you. As for your actual question: I'm not sure if I understand correctly, but I think you got things a little mixed up, or I'm not reading this the right way (also likely). Let's not use self anymore as an example, since that ended in fire and brimstone. Let's use something like this: A = source_var + 1 where `source_var` is a magic thing that refers to the target variable. Does that get the same idea across? Yes? Okay. The main reason that this would be difficult to translate isn't difficulty, but just readability. It's weird that using source_var would use __iadd__, but anything else would use __add__. Now, what I think everyone was telling you about as for "difficulty translating" is this: A = A + 1 ==> A += 1 Similar problem: semantics change. If someone decided to be weird and have __add__ and __iadd__ do two different things, this would completely break that. Granted, that's a stupid idea to begin with, but it's still poor justification for the code breakage. > When I was writing that I just thought, should I make a special note > that I am making it only for example, but then thought, oh that would > be too pedantic. > You say you have no time and write a whole page about it, > so don't blame me that I take too much time from you. > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan (????) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 16 13:21:58 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Nov 2016 10:21:58 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <582CA3C6.2090204@stoneleaf.us> On 11/16/2016 08:48 AM, Mikhail V wrote: > On 16 November 2016 at 10:27, Paul Moore wrote: >> On 16 November 2016 at 08:51, Mikhail V wrote: >>> What semantics it will fundamentally break or so hard to implement? >> >> I'm afraid I don't have time at the moment to fully review your email, >> but my first impression is that you are proposing that the name "self" >> be treated specially. It's going to be very difficult to convince >> people that this is a good idea. > > Oh Paul, come on, pleeeeease. > I am making an **example**. Write a "poo" instead or a poo pile character, > whatever. You think I am an idiot and don't know that there is "self" in OOP. > When I was writing that I just thought, should I make a special note > that I am making it only for example, but then thought, oh that would > be too pedantic. Next time, include the "just an example" disclaimer. We don't know what you do and don't know. > You say you have no time and write a whole page about it, > so don't blame me that I take too much time from you. Hopefully you were joking, but it's really hard to tell. -- ~Ethan~ From Nikolaus at rath.org Wed Nov 16 13:26:12 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 16 Nov 2016 10:26:12 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: (Ryan Gonzalez's message of "Wed, 16 Nov 2016 11:50:23 -0600") References: Message-ID: <87inrnmgcb.fsf@thinkpad.rath.org> On Nov 16 2016, Ryan Gonzalez wrote: > A = A + 1 ==> A += 1 > > > Similar problem: semantics change. If someone decided to be weird and have > __add__ and __iadd__ do two different things, this would completely break > that. Granted, that's a stupid idea to begin with, but it's still poor > justification for the code breakage. Aeh, that's used e.g. in numpy and most certaintly not weird. x = np.range(5) y = x**2 y_int = interpolate(x, y, copy=False) y = y+1 print(y_int(3)) If you replace 'y = y + 1' with 'y += 1', then instead of creating a new array and assigning it to y, you modify the existing array in place, which will change the result of y_int(3). That is a feature. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From mertz at gnosis.cx Wed Nov 16 14:10:54 2016 From: mertz at gnosis.cx (David Mertz) Date: Wed, 16 Nov 2016 11:10:54 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: > > Similar problem: semantics change. If someone decided to be weird and have > __add__ and __iadd__ do two different things, this would completely break > that. Granted, that's a stupid idea to begin with, but it's still poor > justification for the code breakage. > The most notable example of something that is "weird" and "a stupid idea" is NumPy... and *everything* in the Scientific Python ecosystem thereby. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 16 14:36:04 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Nov 2016 19:36:04 +0000 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On 16 November 2016 at 17:50, Ryan Gonzalez wrote: > If someone decided to be weird and have __add__ and __iadd__ do two > different things, this would completely break that. Granted, that's a stupid > idea to begin with, but it's still poor justification for the code breakage. If you think of a = a + 1 as "assigning the value of the expression a + 1 to a" and a += 1 as "incrementing a by 1" it's neither weird nor stupid. There are some well-known invariants (for example, that the value of a after the operation should be one more than the value before) that would be confusing to violate, but there are plenty of other aspects of the behaviour that are not so fundamental, and which can safely and sensibly differ between the two operations. Paul PS Note for anyone who wants to take this off on a wild tangent - my above comment is *in the context of Python as it has been defined for 20+ years*. If you want to write a new language, you can make your own judgement about any or all of this, and the success of your language will be the measure of how reasonable your ideas are... From greg.ewing at canterbury.ac.nz Wed Nov 16 16:08:06 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Nov 2016 10:08:06 +1300 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <582CCAB6.2030001@canterbury.ac.nz> David Mertz wrote: > The most notable example of something that is "weird" and "a stupid > idea" is NumPy... and *everything* in the Scientific Python ecosystem > thereby. Also the built-in list type. Guido clearly disagrees about the stupidity level of this idea! -- Greg From prometheus235 at gmail.com Wed Nov 16 16:57:34 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Wed, 16 Nov 2016 15:57:34 -0600 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <582CCAB6.2030001@canterbury.ac.nz> References: <582CCAB6.2030001@canterbury.ac.nz> Message-ID: I think the best way to remove compound operators would be to go back in time and hit Dennis Ritchie with a bat at the exact moment when the notion of them crosses his mind. In all seriousness, compound operators are in practically every modern language and aren't going away. While you don't need to be a savant polyglot and build a database-analysis-webapp in every language known to man, everyone should break out of their own "language bubble" once they have the basics of programming down. Do a tutorial and make a weekend project with a couple other *disparate* languages with different paradigms (not just close relatives like Python and Ruby, or Java and C and C++). There's always something good in other perspectives and worth learning from. Except Perl. On Wed, Nov 16, 2016 at 3:08 PM, Greg Ewing wrote: > David Mertz wrote: > >> The most notable example of something that is "weird" and "a stupid idea" >> is NumPy... and *everything* in the Scientific Python ecosystem thereby. >> > > Also the built-in list type. Guido clearly disagrees about > the stupidity level of this idea! > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Wed Nov 16 16:58:37 2016 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Wed, 16 Nov 2016 13:58:37 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: Hi all, Please be mindful when replying, even if some of the lurker know who some of you are and can figure out that some of the reply to this thread below this message are sarcastic, not all readers can. Your messages can also be cited out of context. Thus many messages in this thread can be misinterpreted, and can be/are hurtful (regardless of previous misinterpretation). Please try to give the example. Some of the recent exchanges are typically the kind of things that are driving contributors away, or even just repel users from following what's happening on Python-*. Honestly I like to hear about ideas here, but if it's to have to read these kind of exchanges, I'm starting to lean toward unsubscribing. I'll remind you that Brett Cannon wrote an email to this list only 9 days ago to remind people to be civil. Also when you are subscribed to the mailing list a link to the code of conduct is appended to every mails: Code of Conduct: http://python.org/psf/codeofconduct/ Regards, -- Matthias On Wed, Nov 16, 2016 at 1:27 AM, Paul Moore wrote: > On 16 November 2016 at 08:51, Mikhail V wrote: >> What semantics it will fundamentally break or so hard to implement? > > I'm afraid I don't have time at the moment to fully review your email, > but my first impression is that you are proposing that the name "self" > be treated specially. It's going to be very difficult to convince > people that this is a good idea. > > In the first instance, if you're talking about "self" being a keyword, > that would break the current usage of "self" as a normal variable > name, which is used by convention in all class methods to represent > the current object. If you're not aware, in > > class A: > def method(self): > ... > > the use of self as the variable name is *convention*. You could use > any other name just as well. If self were a keyword, it couldn't be > used here. > > So making "self" a keyword would be a major change. > > In addition, consider code like this: > > class A(int): > def method(self): > myvar = 1 > myvar = self + 1 > return myvar > > foo = A(5) > foo.method() > > This is legitimate code at the moment. It sets "myvar" to 6 (self is > an integer subclass with value 5, which the code adds 1 to) and > returns that. Your proposal would change that to return 2. > > The problem here is that when considering language changes, you can't > just consider code that you consider "correct". You have to take > account of the fact that other people may be writing code that is to > your eyes bizarre and unmaintainable - but if it does what they want > it to, and it works according to the language specification, they have > a right to expect that the behaviour won't get changed (at least not > without due consideration of the backward compatibility process). So > proposals that change the behaviour of currently working code have to > be *very* careful to consider all edge cases. > > This is probably the biggest hurdle for people proposing changes to > Python to get over. Backward compatibility isn't pleasant to deal with > - it's frustrating and demotivating to have a good idea blocked > because code that seems senseless "might break". But that's part of > the price we have to pay for Python being a hugely popular language. > > Hope this is useful. > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mikhailwas at gmail.com Wed Nov 16 17:24:01 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 16 Nov 2016 23:24:01 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: On 16 November 2016 at 18:28, Paul Moore wrote: > No I don't think you're an idiot. I thought you were offering a proposal. > >> When I was writing that I just thought, should I make a special note >> that I am making it only for example, but then thought, oh that would >> be too pedantic. > > Well, if it's "only an example", there's nothing to comment on. You > need to offer a proper proposal with the details filled in - otherwise > it's not worth replying. > >> You say you have no time and write a whole page about it, >> so don't blame me that I take too much time from you. > > I'm sorry. I thought I was trying to help by explaining how your > proposal could be improved. My mistake - I'll keep quiet next time. Paul, ok, I apologize for the latter. But can you see, I spent also quite a bit of time trying to think out use cases, I spent over an hour and looked for similar ideas here. Then I accurately made the subject for the question "Technical possibilities" which already implies that it is about my previous misunderstanding of typed vs. non-typed execution and its possible consequences and it is hard to do without exact knowledge how it all works. Therefore I start small and wanted actually that next time I don't ask stupid questions and other new users can also raise their knowledge for making better quality post. My only fault is that I choose wrong name for the keyword and naively supposed that it will not cause big misunderstanding. Ok? So lets say I made a typo and lets take other keyword, it is not so important in this case, which one. Mikhail From steve at pearwood.info Wed Nov 16 19:06:55 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 Nov 2016 11:06:55 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <20161117000654.GS3365@ando.pearwood.info> On Wed, Nov 16, 2016 at 11:24:01PM +0100, Mikhail V wrote: [...] > My only fault is that I choose wrong name > for the keyword and naively supposed that it will not cause big > misunderstanding. > > Ok? So lets say I made a typo and lets take other keyword, it is not > so important in this case, which one. It doesn't matter what keyword you come up with, you still have the problem that this idea introduces a new keyword. New keywords always break backwards compatibility, which means there has to be a good reason to do it. We could make up a random keyword, XusBeooSwqxy3nf7TRx say, and expect that probably nobody in the world has ever used that in their code before. But it would be hard for people to remember and spell correctly, and that goes against the idea that keywords should be meaningful. Any meaningful name you pick as a keyword will probably clash with somebody's code, which means you will break their working program by introducing a new keyword. We take our responsibility seriously: we try very hard to not break people's code when they upgrade, so there has to be a really, really good reason before new keywords are introduced. If this was, say, Python 2.2 or so, and you proposed a new keyword: really_long_variable_name = KEYWORD + 1 then it might be treated a bit more seriously. But the first question we'd ask is, what do other languages do? Python rarely invents new programming ideas itself, but it does copy good ideas from other languages. And of course, the most obvious answer is: Most C-based languages have a += augmented assignment symbol, which avoids the need for a new keyword: really_long_variable_name += 1 Now that we have symbols for augmented assignment, there's *zero* chance that we will remove them, and very little chance that we'll introduce a parallel syntax for doing the same thing by keyword. To get an idea of the standard we require before adding major new syntax or keywords, you should read the historical PEPs: https://www.python.org/dev/peps/ You don't have to write up a full PEP before proposing an idea here, but if the idea involves a large change (especially a new keyword) it almost certainly will require a PEP. But either way, reading the PEPs will give you an idea of what proposals are likely to be accepted, which are likely to be rejected, and which are still open for discussion, as well as the sorts of objections you are likely to receive. -- Steve From steve at pearwood.info Wed Nov 16 19:16:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 Nov 2016 11:16:18 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <20161117001617.GT3365@ando.pearwood.info> On Wed, Nov 16, 2016 at 07:36:04PM +0000, Paul Moore wrote: > On 16 November 2016 at 17:50, Ryan Gonzalez wrote: > > If someone decided to be weird and have __add__ and __iadd__ do two > > different things, this would completely break that. Granted, that's a stupid > > idea to begin with, but it's still poor justification for the code breakage. > > If you think of a = a + 1 as "assigning the value of the expression a > + 1 to a" and a += 1 as "incrementing a by 1" it's neither weird nor > stupid. Indeed. Suppose Python had both mutable and immutable ints (as we have mutable lists and immutable tuples, which are kinda-lists). Then given assignment of immutable ints the two forms will be the same. (This is the status quo.) a = b = 99 a = a + 1 assert a == 100 and b == 99 and likewise for a += 1. But for *mutable* ints, they may not be. We might expect: a = b = mutable(99) a = a + 1 assert a == 100 and b == 99 no different from the status quo, but: a = b = mutable(99) a += 1 # modify a in-place assert a == b == 100 as is the status quo for lists. Of course, if we're designing mutable ints, we're free to pick whatever behaviour we want for both __add__ and __iadd__. But we needn't feel that they are *necessarily* identical. -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 16 21:42:14 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 17 Nov 2016 11:42:14 +0900 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> Message-ID: <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> Cory Benfield writes: > I think the core question you need to answer for this proposal is: > why is ?pip install oic? not easy-enough reach? My first guess would be "some enterprises use OAuth internally for the same reason they have draconian approval policies". More straightforwardly, this is the kind of battery that enterprises which make it hard to use pip seem likely to value. Nick's response is formally correct, but if requests is so important, it's likely to already be on the approved list. I would assume that pyoic also implements the client side, so it is useful even if requests is absent. But I am not a draconian security policy QA/security reviewer. I'd take anything Paul Moore says pretty seriously, as he operates in such an environment. From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 16 21:50:59 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 17 Nov 2016 11:50:59 +0900 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> Matthias Bussoni writes: > Please be mindful when replying, even if some of the lurker know > who some of you are and can figure out that some of the reply to > this thread below this message are sarcastic, not all readers > can. Your messages can also be cited out of context. Mikhail has long since gone past the point where further posts from him deserve such consideration. Replying to Paul Moore when deprecating "sarcastic" replies is just bad manners; Paul is never intentionally sarcastic that I can remember. Mikhail V writes: > So I can do better if we imagine, just imagine, such a scenario: I can imagine it, and immediately reject it for reasons already given for precisely that scenario. Evidently you have not understood the answers that have been given already. Please just stop. You are trying to enforce your preferences on the Python world, and that is just not going to happen because the Python world has already demonstrated strong preference in the opposite direction, starting at > 1. += and friends are now not allowed for integers, > but all _add()_, _iadd()_ are still of course there, > it does not concern me. Both I and David Mertz, among others, have addressed this point directly. It is not going to happen. Of course I am not authoritative; if 6 or 7 different posters does not convince you, feel free to ask Guido directly. But it is useless to continue here. Your questions have all already been answered in full. Your preference has not been denied; it's simply impossible to satisfy it and the preference of the great majority (David estimated 80%, sort of humorously I think; I'm stodgy, I'll just say significantly more than 50%) at the same time. Your persistence given those responses is rude, because you're simply ignoring the effort that others have taken to address your posts. Please stop both the repeat posting and the failure to acknowledge the effort others have made -- words are not enough, you must try harder to understand to show proper respect. > What semantics it will fundamentally break or so hard to implement? This has been explained, in detail, in several ways, from at least two points of view. It is rude of you to continue asking. Please stop, reread the posts explaining this point, and ask about specific statements you don't understand rather than whining that nobody seems interested in doing things your way. This is my last post on this matter, and my last reply to you until your behavior indicates an interest in cooperating to solve problems, rather than self-centered demands. Sincerely yours, Stephen Turnbull -- Associate Professor Department of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From bussonniermatthias at gmail.com Wed Nov 16 22:01:25 2016 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Wed, 16 Nov 2016 19:01:25 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> References: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Nov 16, 2016 at 6:50 PM, Stephen J. Turnbull wrote: > Replying to Paul Moore when > deprecating "sarcastic" replies is just bad manners; Paul is never > intentionally sarcastic that I can remember. Apologies, if my message looked like targetting Paul, and was a reply to Paul. Paul answers were always well written and I never meant to target his message or him, but other messages in the thread. I didn't wanted to look like my comment was about anyone especially any may miss-used my mail client when replying. Sorry about how it could have been perceived, and for doing that. -- M From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 16 22:07:34 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 17 Nov 2016 12:07:34 +0900 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <582CCAB6.2030001@canterbury.ac.nz> Message-ID: <22573.7926.125503.317807@turnbull.sk.tsukuba.ac.jp> Nick Timkovich writes: > I think the best way to remove compound operators would be to go > back in time and hit Dennis Ritchie with a bat at the exact moment > when the notion of them crosses his mind. True enough as a concept, but Python didn't have to implement them for immutables. It might have been interesting to allow in-place operations only for mutables. Dunno if Guido thought of that at the time, and I suspect that a lot of the voices that convinced him to go against his instincts wanted them specifically for integers (a lot of people wanted "++" and "--", too). So even if he did think of it, it might not have satisfied the proponents that their needs were being addressed. > Except Perl. Please don't. Perl has its weaknesses, but it has its strengths too. I suspect that Python regexps were strongly influenced by PCRE, and that the Cheeseshop has borrowed a feature or two from CPAN. From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 16 22:07:48 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 17 Nov 2016 12:07:48 +0900 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: Message-ID: <22573.7940.882876.507327@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > PS Note for anyone who wants to take this off on a wild tangent - my > above comment is *in the context of Python as it has been defined for > 20+ years*. Not quite. Augmented assignment operators were added in Python 2.0 according to What's New, and they were quite controversial (== Guido didn't like them) at the time. In that respect, Mikhail is in good company (perhaps for different reasons). But Guido did accept them, and they are immoveable at this point in time. Steve From mikhailwas at gmail.com Wed Nov 16 22:17:12 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 17 Nov 2016 04:17:12 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> References: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 03:50, Stephen J. Turnbull wrote: > Matthias Bussoni writes: > > > Please be mindful when replying, even if some of the lurker know > > who some of you are and can figure out that some of the reply to > > this thread below this message are sarcastic, not all readers > > can. Your messages can also be cited out of context. > > Mikhail has long since gone past the point where further posts from > him deserve such consideration. Replying to Paul Moore when > deprecating "sarcastic" replies is just bad manners; Paul is never > intentionally sarcastic that I can remember. Steven, are you sure that Matthias adressed that only to me? There were indeed pair of quite strange comments regarding Numpy and lists that I personally also did not understand. So let us probably wait and Matthias will clear that. > This is my last post on this matter, and my last reply to you until > your behavior indicates an interest in cooperating to solve problems, > rather than self-centered demands. I must say you re getting it wrong, from my side there are only clear *non* self-centered, humanistic intentions. This is probaly not seen, but that is due to differences between people, their culture, background, etc. Everyone is unique and stylistics of words does not reflect it, but the intentions what is important. Mikhail From greg.ewing at canterbury.ac.nz Wed Nov 16 16:06:31 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Nov 2016 10:06:31 +1300 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> Message-ID: <582CCA57.2000206@canterbury.ac.nz> Cory Benfield wrote: > Instead, we should aim > to solve the actual problem: how do we provide tools to allow users to find > the best-in-class solutions to their problems from the third-party Python > ecosystem? Perhaps there could be a curated area on PyPI, maintained by core developers or people appointed by them, where the packages considered best-in-class would be placed. Instead of lobbying to get a package into the stdlib (a very high barrier to cross) people would then have the option of lobbying to get it stamped as "recommended" on PyPI. -- Greg From rosuav at gmail.com Wed Nov 16 23:04:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 17 Nov 2016 15:04:42 +1100 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <582CCA57.2000206@canterbury.ac.nz> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <582CCA57.2000206@canterbury.ac.nz> Message-ID: On Thu, Nov 17, 2016 at 8:06 AM, Greg Ewing wrote: > Cory Benfield wrote: >> >> Instead, we should aim >> to solve the actual problem: how do we provide tools to allow users to >> find >> the best-in-class solutions to their problems from the third-party Python >> ecosystem? > > > Perhaps there could be a curated area on PyPI, maintained > by core developers or people appointed by them, where > the packages considered best-in-class would be placed. > > Instead of lobbying to get a package into the stdlib (a > very high barrier to cross) people would then have the > option of lobbying to get it stamped as "recommended" on > PyPI. Or on the Python wiki: https://wiki.python.org/moin/FrontPage A page there could highlight the best PyPI packages. ChrisA From ncoghlan at gmail.com Wed Nov 16 23:06:28 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 Nov 2016 14:06:28 +1000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 12:42, Stephen J. Turnbull wrote: > Cory Benfield writes: > > > I think the core question you need to answer for this proposal is: > > why is ?pip install oic? not easy-enough reach? > > My first guess would be "some enterprises use OAuth internally for the > same reason they have draconian approval policies". More > straightforwardly, this is the kind of battery that enterprises which > make it hard to use pip seem likely to value. Nick's response is > formally correct, but if requests is so important, it's likely to > already be on the approved list. I would assume that pyoic also > implements the client side, so it is useful even if requests is > absent. In that context, the problem is the old "batteries that leak acid everywhere can be worse than no batteries at all" one: we know from painful experience with the SSL module that the standard library's typical release and adoption cycle can be seriously problematic when it comes to network security related modules. However, when it comes to draconian security policies, *transitive recommendations have power*: if CPython is approved, and python-dev collectively says "we recommend pip, virtualenv, and requests", then folks in locked down environments can use that as evidence to suggest that those other modules are also trustworthy (particularly when there are commercial software vendors shipping them). In the case of something like OIDC, if the requests, Flask, Django and Pyramid developers were all inclined to recommend the same library for server and client implementations, then that would similarly be sufficient justification in many environments to bring the component in as a new dependency. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Nov 17 04:14:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 09:14:53 +0000 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> References: <22573.6931.867546.667061@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 02:50, Stephen J. Turnbull wrote: > Matthias Bussoni writes: > > > Please be mindful when replying, even if some of the lurker know > > who some of you are and can figure out that some of the reply to > > this thread below this message are sarcastic, not all readers > > can. Your messages can also be cited out of context. > > Mikhail has long since gone past the point where further posts from > him deserve such consideration. Replying to Paul Moore when > deprecating "sarcastic" replies is just bad manners; Paul is never > intentionally sarcastic that I can remember. For the record, I didn't take Matthias' comment personally. I presumed that the main focus of his comment was the sub-thread that was hinting towards "NumPy is weird" - some of the comments in there lacked enough context. But regardless, his point was entirely fair - even something as simple as a trimmed quote in email can make a comment mean something quite different from the original intent, so it pays everyone to think carefully about what they say. Oh, and by the way, if you think I'm never sarcastic, you've never met me face to face ;-) I take way more care with my tone in emails than I do in real life! Paul From p.f.moore at gmail.com Thu Nov 17 04:22:45 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 09:22:45 +0000 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <22573.7940.882876.507327@turnbull.sk.tsukuba.ac.jp> References: <22573.7940.882876.507327@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 03:07, Stephen J. Turnbull wrote: > Paul Moore writes: > > > PS Note for anyone who wants to take this off on a wild tangent - my > > above comment is *in the context of Python as it has been defined for > > 20+ years*. > > Not quite. Augmented assignment operators were added in Python 2.0 > according to What's New, and they were quite controversial (== Guido > didn't like them) at the time. In that respect, Mikhail is in good > company (perhaps for different reasons). But Guido did accept them, > and they are immoveable at this point in time. Fair point. My "20+ years" was a rough guess, and wasn't so much intended to relate specifically to augmented assignment as to the principles involved. Python tends to pick up ideas from other C-like languages, and doesn't arbitrarily deviate from common syntax (the conditional expression being the most obvious example where we did use a different syntax than everyone else). Paul PS Whatever the merit of Mikhail's specific proposals, he is prompting discussion on the principles behind our rejection of his ideas, and that's valuable. So thanks to him for that, at least. From p.f.moore at gmail.com Thu Nov 17 04:35:28 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 09:35:28 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 02:42, Stephen J. Turnbull wrote: > But I am not a draconian security policy QA/security reviewer. I'd > take anything Paul Moore says pretty seriously, as he operates in such > an environment. For context, my environment is one that doesn't formally use Python, but needs a lot of adhoc automation and management solutions, for which Python is a great fit, as long as it doesn't need anything that isn't pure "out of the box" functionality (because once we need that, we get into formal requests for things to be added to supported software lists). There's certain possibilities for "under the radar" additions, but the costs get high pretty quickly and overwhelm the benefits. So in some ways things can be very flexible, but in others I need to think in worst-case "I'm lucky to have Python at all, let's not push my luck" terms. It's likely that this sort of environment is becoming less common as Python becomes more mainstream/popular (it's not that long ago that you were lucky to find Python in a default Unix installation at all, for example), but it is still something we should be considering when looking at what deserves to be in the stdlib (sure requests is better than urllib, but if urllib disappeared, I wouldn't be able to do web requests at all in many of my environments). Paul From mikhailwas at gmail.com Thu Nov 17 05:16:14 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 17 Nov 2016 11:16:14 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <22573.7940.882876.507327@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 10:22, Paul Moore wrote: > On 17 November 2016 at 03:07, Stephen J. Turnbull > wrote: >> Paul Moore writes: >> >> > PS Note for anyone who wants to take this off on a wild tangent - my >> > above comment is *in the context of Python as it has been defined for >> > 20+ years*. >> >> Not quite. Augmented assignment operators were added in Python 2.0 >> according to What's New, and they were quite controversial (== Guido >> didn't like them) at the time. In that respect, Mikhail is in good >> company (perhaps for different reasons). But Guido did accept them, >> and they are immoveable at this point in time. > > Fair point. My "20+ years" was a rough guess, and wasn't so much > intended to relate specifically to augmented assignment as to the > principles involved. Python tends to pick up ideas from other C-like > languages, and doesn't arbitrarily deviate from common syntax (the > conditional expression being the most obvious example where we did use > a different syntax than everyone else). > > Paul > > PS Whatever the merit of Mikhail's specific proposals, he is prompting > discussion on the principles behind our rejection of his ideas, and > that's valuable. So thanks to him for that, at least. Citation from http://legacy.python.org/dev/peps/pep-0203/ """ Expressions of the form = are common enough in those languages to make the extra syntax worthwhile, and Python does not have significantly fewer of those expressions. Quite the opposite, in fact, since in Python you can also concatenate lists with a binary operator, something that is done quite frequently. Writing the above expression as = -> __is both more readable__ and less error prone, because it is instantly obvious to the reader that it is that is being changed, and not that is being replaced by something almost, but not quite, entirely unlike . """ Aha, sure, if "readable" == "suits C programmers". And it's surely not, which in some sence also means that it *more* error prone, since I cannot notice possible error so easy. (I am not trying to start flame again though, that is *not* my intention) Mikhail From mal at egenix.com Thu Nov 17 05:41:42 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 17 Nov 2016 11:41:42 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <22573.7940.882876.507327@turnbull.sk.tsukuba.ac.jp> Message-ID: <582D8966.400@egenix.com> On 17.11.2016 11:16, Mikhail V wrote: > Citation from > http://legacy.python.org/dev/peps/pep-0203/ > > """ > Expressions of the form > > = > > are common enough in those languages to make the extra syntax > worthwhile, and Python does not have significantly fewer of those > expressions. Quite the opposite, in fact, since in Python you can > also concatenate lists with a binary operator, something that is > done quite frequently. Writing the above expression as > > = > > -> __is both more readable__ and less error prone, because it is > instantly obvious to the reader that it is that is being > changed, and not that is being replaced by something almost, > but not quite, entirely unlike . > > """ > > Aha, sure, if "readable" == "suits C programmers". > And it's surely not, which in some sence also means that it *more* > error prone, since I cannot notice possible error so easy. The above follows the general principle in Python that "explicit is better than implicit": 1. "a = a + 1" means (in Python): add 1 to a (via the .__add__() method) and bind the new value to "a". 2. "a += 1" means (in Python): let the object a add 1 to itself (via the .__iadd__() method) and bind the new value to "a". In the first variant, a may well result in a new object being created, even changing the object type. In the second variant, it is immediately clear that the intent is for a to do something to itself. It is much less likely to result in a new object type (even though it may still result in a new object being created). More common is to have a manipulate itself without even changing the object. Examples: --------- >>> l = [1,2,3] >>> id(l) 140431507775856 >>> l = l + [4] >>> id(l) 140431507776864 Here, a new list object was created. >>> l += [5] >>> id(l) 140431507776864 >>> Here, the object itself was manipulated without creating a copy; and that's what you'd typically expect from an in-place operation on mutable objects. The situation is different with immutable objects: >>> a = 1 >>> id(a) 27693400 >>> a = a + 1 >>> id(a) 27693376 >>> a += 1 >>> id(a) 27693352 In both cases, you get new objects (but the object type remains the same). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 17 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From cory at lukasa.co.uk Thu Nov 17 05:58:05 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 17 Nov 2016 10:58:05 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> Message-ID: <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> > On 16 Nov 2016, at 14:53, Paul Moore wrote: > > I'm not a web developer as such, although I do write code that > consumes web services on occasion. I don't know what OIDC is, but I do > know, for example, that some services use OAuth. So I can imagine > being in a situation of saying "I want to get data from a web API xxx, > and it needs OAuth identification, how do I do that in Python?" > Typically, the API docs are in terms of something like Javascript, > with a browser UI, so don't help much for a command line script which > is the sort of thing I'd be writing. In that situation, a well-known, > easy to use module for OAuth in Python would be fantastic. > > Agreed that it could as easily be on PyPI as in the stdlib, but > discoverability isn't as good with PyPI - I can scan the stdlib docs, > but for PyPI I'd end up scanning Google, and what I found that way was > oauthlib - I didn't see any mention of pyoidc. I can't comment on what > that implies, though. In my brief search though I didn't find any sort > of command line "Hello world" level example. I think this is actually another great example of why we should resist attempts to add modules to the standard library without enormous caution. I think that fundamentally in most of these cases the audience on python-dev is not equipped to decide whether an implementation deserves to become a default battery. And this is a surprisingly good example case. With all due respect to Roland, pyoidc is not the incumbent in the synchronous OAuth space: requests-oauthlib is. A quick Google search for ?python oauth? turned up the following client libraries in my top 10 results: oauthlib (discarded because it?s a sans-IO implementation that recommends requests-oauthlib as a client), python-oauth2, requests-oauthlib, and Google?s OAuth 2 client library (discarded because it is bundled into the Google API client libraries module and so distorts my download counts below). A quick query of the PyPI download database for the three months shows the following download counts for those modules: - requests-oauthlib == 1,897,048 - oauth2 == 349,759 - pyoidc == 10,520 This is not intended to be chastening for Roland: all new modules start with low download counts. As the current lead maintainer of requests-oauthlib, let me say publicly and loudly that I?d love to have pyoidc replace requests-oauthlib. I *hate* requests-oauthlib. I maintain it literally only to prevent it falling into disrepair because it is extremely widely used. I would love a better library to come along so that we can sunset requests-oauthlib. I am entirely prepared to believe that Roland?s module is better than requests-oauthlib: it would be hard for it not to be. However, *right now*, pyoidc does not have anything like a majority (or even a plurality) of mindshare amongst Python developers writing OAuth clients. So why should pyoidc be added to the standard library over the competing implementations? The only reason I can see to add it is if it is a better implementation than its competitors, and the python-dev community believe that developers using the competitor implementations would be better served using pyoidc instead. Is that the case? Do we have some objective measure of this? Paul, you mentioned that discovery on PyPI is a problem: I don?t contest that at all. But I don?t think the solution to that problem is to jam modules into the standard library, and I think even less of that idea when there is no formal process available for python-dev to consider the implementations available for the standard library. Instead, I think we need a way to be able to ask the question: ?what does the wider Python development community consider to be the gold standard for solving problem X??. I do not think that adding modules to the standard library is the way to answer that question. The TL;DR of this massive argument is: I think the community of people who actually use OAuth on a regular basis are better placed to judge what the best-in-class battery for OAuth is. What we need is a way to surface their collective opinions to people who don?t know what options are available, rather than to make commitments to long-term support of a module in the standard library. >> It should be noted that I believe that Python?s standard library is already too big, and has had a tendency in the past to expand into cases that were not warranted. > > I should also note that I rely heavily on the stdlib, and for a > non-trivial amount of the work I do (which is one-off scripts, not > full-blown applications) having to go outside the stdlib, other than > for a very few select modules, is a step change in complexity. So I'm > a fan of the "batteries included" approach. I think this is the other problem that needs solving, and because I?m a full-time OSS developer with complete admin rights on my development and production targets I?m badly placed to solve it. What needs to be done to make it easier for people in your position to obtain non-included batteries? Can anything be done at all? > I don't know whether OAuth is a sufficiently common requirement to > warrant going into the stdlib. My instinct is that if you're > integrating it into a web app, then there's no value in it being in > the stdlib as you'll already need 3rd party modules. If it's practical > to use OAuth from a simple Python script (say, to integrate with a web > API like github) then being in the stdlib could be a benefit. But how > many people write Python scripts that use/need OAuth? I've no feel for > that. Yeah, OAuth from Python scripts isn?t entirely uncommon. It?s mostly used to interact with third-party APIs, which usually use OAuth to allow for revocable and granular permissions grants to specific scripts. Cory From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 17 06:12:42 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 17 Nov 2016 20:12:42 +0900 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> Message-ID: <22573.37034.132349.124484@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > In that context, the problem is the old "batteries that leak acid > everywhere can be worse than no batteries at all" one: we know from > painful experience with the SSL module that the standard library's > typical release and adoption cycle can be seriously problematic when > it comes to network security related modules. In that context, SSL is a somewhat special case because of its dependence on OpenSSL and because of Apple's U+1F4A9-headed attitude toward fixing something they distribute. If pyoic is pure Python, I would expect as many releases as we have branches supporting it within 6-12 weeks. Less time, if a truly nasty security bug. No? Sure, that's problematic for any sites that haven't yet approved a Python version with pyoic in it, but they're problematic period. > However, when it comes to draconian security policies, *transitive > recommendations have power*: if CPython is approved, and python-dev > collectively says "we recommend pip, virtualenv, and requests", then > folks in locked down environments can use that as evidence to suggest > that those other modules are also trustworthy (particularly when there > are commercial software vendors shipping them). I understand that argument, but I can assure you that my employer does not. Its security policies tend to be both draconian and ineffective. Is the "python-dev and RHEL recommend it" evidence all that effective for "most" sites with "software must be approved before installing" policies? (This argument could easily go against me, if "draconian" sites tend to drag on approving Python point releases as well as on "new" PyPI modules.) Steve From p.f.moore at gmail.com Thu Nov 17 06:35:31 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 11:35:31 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: On 17 November 2016 at 10:58, Cory Benfield wrote: > Paul, you mentioned that discovery on PyPI is a problem: I don?t contest that at all. But I don?t think the solution to that problem is to jam modules into the standard library, and I think even less of that idea when there is no formal process available for python-dev to consider the implementations available for the standard library. Yeah, in the process of the discussion a certain amount of context was lost. I also don't think that the solution is to "jam" modules into the standard library. I *do* think that part of the solution should be to have good solutions to common programming problems in the standard library. What is a "common problem" changes over time, as well as by problem domain, and we need to take that into account. My feeling is that (client-level, web service consumer focused) OAuth is tending towards being one of those "common problems" (as the authentication side of the whole REST/JSON/etc web API toolset) and warrants consideration for inclusion in the stdlib. I have no experience to judge what is the current "best solution". I'm happy for the community to thrash that out, and settle on a standard. Roland proposed his library as a solution - I can comment on that (from the perspective of "what a non-expert would like to have from the stdlib") but I won't until the question of "is the proposed library really the standard" is resolved. And maybe it *won't* be resolved as the situation isn't sufficiently settled yet. And that's fine. But I don't agree with the principle that we should stop adding solutions to common problems to the stdlib "because PyPI is only a pip install away". There will always be users who can't, won't or simply don't use PyPI and judge Python on what you can do with a base install. And that's a valid judgement to make. One of the reasons I prefer Python over (say) Perl, is that if I go onto a Linux server that's isolated from the internet, both are available but on Python I can do things like compose a MIME email and send it via SMTP, work with dates and times, read and write CSV files, parse XML data from an external program, etc. On Perl I can't because the Perl standard library doesn't have those things available, so I end up having to write my own - which means I take time away from getting my *actual* job done. > Instead, I think we need a way to be able to ask the question: ?what does the wider Python development community consider to be the gold standard for solving problem X??. Agreed, that's the key unsolved question for Python packaging. > I do not think that adding modules to the standard library is the way to answer that question. Again agreed. But I *do* think that once that question is answered (on a case by case basis, not the overall "how do we do it for everything" question) then adding the module that has been identified as the gold standard to the stdlib *may* (depending on the problem domain) be important. Put it another way - being in the stdlib isn't a solution to the discoverability problem, but it is a solution to the access problem (which is a real problem for some people, despite pip and PyPI). Paul From p.f.moore at gmail.com Thu Nov 17 06:44:25 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 11:44:25 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <22573.37034.132349.124484@turnbull.sk.tsukuba.ac.jp> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <22573.6406.483708.135481@turnbull.sk.tsukuba.ac.jp> <22573.37034.132349.124484@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 November 2016 at 11:12, Stephen J. Turnbull wrote: > > However, when it comes to draconian security policies, *transitive > > recommendations have power*: if CPython is approved, and python-dev > > collectively says "we recommend pip, virtualenv, and requests", then > > folks in locked down environments can use that as evidence to suggest > > that those other modules are also trustworthy (particularly when there > > are commercial software vendors shipping them). > > I understand that argument, but I can assure you that my employer does > not. Its security policies tend to be both draconian and ineffective. > Is the "python-dev and RHEL recommend it" evidence all that effective > for "most" sites with "software must be approved before installing" > policies? (This argument could easily go against me, if "draconian" > sites tend to drag on approving Python point releases as well as on > "new" PyPI modules.) In my experience, it's not so much "draconian" policies that cause the issues, as "not interested" policies. You get the standard build because that's what the default install gave, before the application-specific stack was added. If the application in question isn't built with Python (e.g., you use Python for automation, system management, or whatever) then you stand no chance of finding anyone to even *ask* for permission to go beyond the "out of the box" build. At best, on Windows systems, you get to say "we need you to run the following installers for our support tools" once, and likely once only. No internet access, no "please can we have X added", you get what you thought to ask for on day 1, and that's it. Paul From cory at lukasa.co.uk Thu Nov 17 07:27:20 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 17 Nov 2016 12:27:20 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: <8ADA9A2B-2489-4B71-A411-81D6A089922C@lukasa.co.uk> > On 17 Nov 2016, at 11:35, Paul Moore wrote: > > On 17 November 2016 at 10:58, Cory Benfield wrote: >> Paul, you mentioned that discovery on PyPI is a problem: I don?t contest that at all. But I don?t think the solution to that problem is to jam modules into the standard library, and I think even less of that idea when there is no formal process available for python-dev to consider the implementations available for the standard library. > > Yeah, in the process of the discussion a certain amount of context was > lost. I also don't think that the solution is to "jam" modules into > the standard library. Fair enough: ?jam? was probably a more emotive term than I needed to use there, I?ll happily concede that. > I *do* think that part of the solution should be to have good > solutions to common programming problems in the standard library. What > is a "common problem" changes over time, as well as by problem domain, > and we need to take that into account. My feeling is that > (client-level, web service consumer focused) OAuth is tending towards > being one of those "common problems" (as the authentication side of > the whole REST/JSON/etc web API toolset) and warrants consideration > for inclusion in the stdlib. So this argument seems reasonable to me, but my problem with it is that it seems to be fuzzy. It leads me to all kinds of follow-on questions, such as: - What counts as a common problem? Is there an objective measure, or do we decide by gut feel? - How do we scope the problem? Is the problem we?re solving in this specific case OAuth? OpenID Connect? HTTP authentication in general? - How complex does that problem have to be before we decide that the solution doesn?t belong in the standard library? Alternatively, does being complex make it *more* important that we have a standard library solution? - Should the standard library have a greenfield implementation or adopt the third-party one? - If it adopts the third party one what happens to the third-party maintainers, are they expected to keep maintaining? - If they object, does CPython do a hostile fork and take over maintenance itself, or pursue another implementation? - How do we balance the desire to increase the scope of the stdlib with the increased maintenance burden that brings? - Do we ever *remove* modules that are solutions to problems that are no longer common? - Is it acceptable to solve only part of the problem? (For context, OAuth2 is a complex specification that leaves a lot of detail out: requests-oauthlib contains a lot of ?compliance fixes? for specific oauth2 servers that deviate from the specification in unexpected ways. Is it acceptable to write exactly to the spec and to leave anyone who needs custom code to their own devices?) - What about asyncio integration? Is that mandatory for new protocol code? Optional? How important? Can it be asyncio-only? - As a follow-on, what about integration with other stdlib modules? Does the new OAuth module have to work with all stdlib HTTP clients? Only one? Is it its own client you use directly? - What happens if/when the protocol is revised? - What happens if/when the maintainers move on from the project? - Do we also maintain an out-of-tree backport for users on older Pythons? If not, is it acceptable for those users to have older versions of the library unless they upgrade their whole Python distribution? This isn?t me disagreeing with you, just me pointing out that the fuzziness around this makes me nervous. It has been my experience that a large number of protocol implementations in the standard library are already struggling to meet their maintenance goals, and I?d be pretty reluctant about wanting to add to that burden. > But I don't agree with the principle that we should stop adding > solutions to common problems to the stdlib "because PyPI is only a pip > install away". There will always be users who can't, won't or simply > don't use PyPI and judge Python on what you can do with a base > install. And that's a valid judgement to make. One of the reasons I > prefer Python over (say) Perl, is that if I go onto a Linux server > that's isolated from the internet, both are available but on Python I > can do things like compose a MIME email and send it via SMTP, work > with dates and times, read and write CSV files, parse XML data from an > external program, etc. On Perl I can't because the Perl standard > library doesn't have those things available, so I end up having to > write my own - which means I take time away from getting my *actual* > job done. I can understand that. I definitely think you and I have disagreements on the best way to solve this problem (and that we?re unlikely to resolve them in this thread!), but I certainly acknowledge that this use case is real and important. I think my biggest disagreement on this use case is simply about scope: at what point does a use become too niche to be supported by the standard library? > Put it another way - being in the stdlib isn't a solution to the > discoverability problem, but it is a solution to the access problem > (which is a real problem for some people, despite pip and PyPI). Fair enough. Cory From steve at pearwood.info Thu Nov 17 07:30:34 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 Nov 2016 23:30:34 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <22573.7926.125503.317807@turnbull.sk.tsukuba.ac.jp> References: <582CCAB6.2030001@canterbury.ac.nz> <22573.7926.125503.317807@turnbull.sk.tsukuba.ac.jp> Message-ID: <20161117123030.GU3365@ando.pearwood.info> On Thu, Nov 17, 2016 at 12:07:34PM +0900, Stephen J. Turnbull wrote: > Nick Timkovich writes: > > > I think the best way to remove compound operators would be to go > > back in time and hit Dennis Ritchie with a bat at the exact moment > > when the notion of them crosses his mind. > > True enough as a concept, but Python didn't have to implement them for > immutables. It might have been interesting to allow in-place > operations only for mutables. By definition, you can only perform in-place operations on mutables. But I guess you are talking about the syntax. Should += raise an exception if the assignment target is immutable? Problem is, how is the Python interpreter supposed to determine whether or not the target is immutable or mutable? There's no isinstance(obj, Immutable) and I doubt there ever will be. More importantly, banning immutables would immediately prohibit the number one most common use of augmented assignment, and probably the largest motivation for the feature: x += 1 > Dunno if Guido thought of that at the time, and I suspect that a lot > of the voices that convinced him to go against his instincts wanted > them specifically for integers (a lot of people wanted "++" and "--", > too). So even if he did think of it, it might not have satisfied the > proponents that their needs were being addressed. Indeed. Supporting `alist += [x]` but not `x += 1` would have been deeply unsatisfactory. I think the status quo is actually very clever: augmented assignment is broadly equivalent to the expanded assignment: obj += foo => obj = obj + foo except that the target obj has the opportunity to optimize it to an in-place operation, even if that makes the semantics slightly different from the ordinary & operator. > > Except Perl. > > Please don't. I don't think there is any need for that. No harm is done by a little light-hearted banter relating to the rivalry between programming language communities. In the big picture, a bit of friendly rivalry probably does both communities good. Regardless of what virtues Perl may or may not have, in some ways the philosophy behind Perl is dramatically opposed to that of Python. Many of us like Python code because it goes against Perl's ideals. We discourage many of the things that Perl and its culture encourages: terse, long one-liners, heavy use of regexes, heavy use of sigils and symbols, "clever" code, and More Than One Way To Do It. There's a natural rivalry between Perl and Python. We should be permitted to criticise other languages, and even dismiss them as useless. Programming languages do not have the assumption of dignity owed to human beings. Nick's dismissal of Perl was obviously intended as light-hearted and not entirely serious. This is an informal forum, a mailing list, and throw-away comments intended as humour shouldn't be held to the same standard as serious statements intended as part of a logical argument. Had Nick said "Python should not do X, because Perl does it and nothing Perl does is of any value at all" then we'd be justified in asking him to defend his claim, or to simply dismiss it as obviously ridiculous. But he didn't. I doubt Nick meant that Perl was literally lacking in any redeeming features: exaggeration and hyperbole are time-honoured forms of humour. His comments weren't disruptive and they most certainly were not attacking anyone. It would require the most extraordinarily thin skin to take that as an attack against people who use Perl, and even if somebody did, well, this is not a support group for people so lacking in self-esteem that their self-image is equated to their choice of programming language. (I believe we've already had to ban, or at least warn, one regular to this list under the terms of the CoC, because he couldn't distinguish ill-founded criticism of Python the language from criticism of the people designing and using the language, and he responded with personal attacks.) I'm sorry that I've written more words defending Nick's throw-away two word comment than you expended in lightly chastising him. But I didn't want to just say that I disagreed with you -- I wanted to give the reasons why I support our right to light-heartedly disrespect other languages. (And of course you're right, Python has borrowed regular expression syntax and the Cheeseshop from Perl, and possibly more.) -- Steve From roland at catalogix.se Thu Nov 17 07:51:26 2016 From: roland at catalogix.se (Roland Hedberg) Date: Thu, 17 Nov 2016 14:51:26 +0200 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: > On 17 Nov 2016, at 12:58, Cory Benfield wrote: > > >> On 16 Nov 2016, at 14:53, Paul Moore wrote: >> >> I'm not a web developer as such, although I do write code that >> consumes web services on occasion. I don't know what OIDC is, but I do >> know, for example, that some services use OAuth. So I can imagine >> being in a situation of saying "I want to get data from a web API xxx, >> and it needs OAuth identification, how do I do that in Python?" >> Typically, the API docs are in terms of something like Javascript, >> with a browser UI, so don't help much for a command line script which >> is the sort of thing I'd be writing. In that situation, a well-known, >> easy to use module for OAuth in Python would be fantastic. >> >> Agreed that it could as easily be on PyPI as in the stdlib, but >> discoverability isn't as good with PyPI - I can scan the stdlib docs, >> but for PyPI I'd end up scanning Google, and what I found that way was >> oauthlib - I didn't see any mention of pyoidc. I can't comment on what >> that implies, though. In my brief search though I didn't find any sort >> of command line "Hello world" level example. > > I think this is actually another great example of why we should resist attempts to add modules to the standard library without enormous caution. I think that fundamentally in most of these cases the audience on python-dev is not equipped to decide whether an implementation deserves to become a default battery. And this is a surprisingly good example case. > > With all due respect to Roland, pyoidc is not the incumbent in the synchronous OAuth space: requests-oauthlib is. A quick Google search for ?python oauth? turned up the following client libraries in my top 10 results: oauthlib (discarded because it?s a sans-IO implementation that recommends requests-oauthlib as a client), python-oauth2, requests-oauthlib, and Google?s OAuth 2 client library (discarded because it is bundled into the Google API client libraries module and so distorts my download counts below). > > A quick query of the PyPI download database for the three months shows the following download counts for those modules: > > - requests-oauthlib == 1,897,048 > - oauth2 == 349,759 > - pyoidc == 10,520 > > This is not intended to be chastening for Roland: all new modules start with low download counts. No offence taken ! :-) But you should distinguish between OAuth2 and OIDC. OIDC is a profile of OAuth2 for usage in the case where you not only need authorization (and access tokens) but also authentication and/or user info. > As the current lead maintainer of requests-oauthlib, let me say publicly and loudly that I?d love to have pyoidc replace requests-oauthlib. I *hate* requests-oauthlib. I maintain it literally only to prevent it falling into disrepair because it is extremely widely used. I would love a better library to come along so that we can sunset requests-oauthlib. I am entirely prepared to believe that Roland?s module is better than requests-oauthlib: it would be hard for it not to be. :-) > However, *right now*, pyoidc does not have anything like a majority (or even a plurality) of mindshare amongst Python developers writing OAuth clients. So why should pyoidc be added to the standard library over the competing implementations? The only reason I can see to add it is if it is a better implementation than its competitors, and the python-dev community believe that developers using the competitor implementations would be better served using pyoidc instead. Is that the case? Do we have some objective measure of this? The only possible objective measurement I can see would be testing the implementation for standard compliance. > Paul, you mentioned that discovery on PyPI is a problem: I don?t contest that at all. But I don?t think the solution to that problem is to jam modules into the standard library, and I think even less of that idea when there is no formal process available for python-dev to consider the implementations available for the standard library. Instead, I think we need a way to be able to ask the question: ?what does the wider Python development community consider to be the gold standard for solving problem X??. I do not think that adding modules to the standard library is the way to answer that question. > > The TL;DR of this massive argument is: I think the community of people who actually use OAuth on a regular basis are better placed to judge what the best-in-class battery for OAuth is. What we need is a way to surface their collective opinions to people who don?t know what options are available, rather than to make commitments to long-term support of a module in the standard library. > >>> It should be noted that I believe that Python?s standard library is already too big, and has had a tendency in the past to expand into cases that were not warranted. >> >> I should also note that I rely heavily on the stdlib, and for a >> non-trivial amount of the work I do (which is one-off scripts, not >> full-blown applications) having to go outside the stdlib, other than >> for a very few select modules, is a step change in complexity. So I'm >> a fan of the "batteries included" approach. > > I think this is the other problem that needs solving, and because I?m a full-time OSS developer with complete admin rights on my development and production targets I?m badly placed to solve it. What needs to be done to make it easier for people in your position to obtain non-included batteries? Can anything be done at all? > >> I don't know whether OAuth is a sufficiently common requirement to >> warrant going into the stdlib. My instinct is that if you're >> integrating it into a web app, then there's no value in it being in >> the stdlib as you'll already need 3rd party modules. If it's practical >> to use OAuth from a simple Python script (say, to integrate with a web >> API like github) then being in the stdlib could be a benefit. But how >> many people write Python scripts that use/need OAuth? I've no feel for >> that. > > Yeah, OAuth from Python scripts isn?t entirely uncommon. It?s mostly used to interact with third-party APIs, which usually use OAuth to allow for revocable and granular permissions grants to specific scripts. > > Cory > From p.f.moore at gmail.com Thu Nov 17 08:42:31 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 13:42:31 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: <8ADA9A2B-2489-4B71-A411-81D6A089922C@lukasa.co.uk> References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> <8ADA9A2B-2489-4B71-A411-81D6A089922C@lukasa.co.uk> Message-ID: On 17 November 2016 at 12:27, Cory Benfield wrote: > This isn?t me disagreeing with you, just me pointing out that the fuzziness around this makes me nervous. It has been my experience that a large number of protocol implementations in the standard library are already struggling to meet their maintenance goals, and I?d be pretty reluctant about wanting to add to that burden. The fuzziness is a clear and definite issue here. My perspective is that this far, Python (specifically the core dev team[1]) has done a good job of balancing that fuzziness. I don't expect this to ever be a simple decision to make. All I ask is that we avoid countering a complex decision process with an over-simple guideline (specifically, that "things don't need to go into the stdlib because of pip/PyPI[2]). Regardless of how things go in the long term, I think it's good to keep the debates open in this area, and I appreciate your comments. I'll certainly be thinking about my personal answers to the questions you raised, and I expect I'll change at least some of my views as a result. Paul [1] Even though I'm a core dev, I view myself as a "normal user" in this context, and take no personal credit for the scope of the stdlib. [2] Ironically, I'm also a pip developer, so if I seem confused, I claim the right to be :-) From ayush.agg90 at gmail.com Thu Nov 17 08:56:08 2016 From: ayush.agg90 at gmail.com (Ayush Aggarwal) Date: Thu, 17 Nov 2016 19:26:08 +0530 Subject: [Python-ideas] Unable to sniff outgoing traffic using raw sockets in python2.7 Message-ID: Hello, Following is my code : #!/usr/bin/python import socket import struct import binascii rawSocket = socket.socket(socket.PF_PACKET,socket.SOCK_RAW,socket.htons(0x0800)) # use 0x0800 for IPv4 packets , 0x0003 is for sniffing all kinds of packets while True: pkt= rawSocket.recvfrom(2048) ethernetHeader = pkt[0][0:14] pr = unicode(ethernetHeader, errors='replace') print pr eth_hdr = struct.unpack("!6s6s2s",ethernetHeader) print "Source MAC Address :" , binascii.hexlify(eth_hdr[1]) print "Destination MAC Address : " , binascii.hexlify(eth_hdr[0]) print "Protocol : " , binascii.hexlify(eth_hdr[2]) ipHeader = pkt[0][14:34] ip_hdr = struct.unpack("!12s4s4s",ipHeader) print "Source ip ADDRESS : " + socket.inet_ntoa(ip_hdr[1]) print "Destination IP Address: " + socket.inet_ntoa(ip_hdr[2]) # initial part of the tcp header tcpHeader = pkt[0][34:54] tcp_hdr = struct.unpack("!HH16s",tcpHeader) print "Source Port ADDRESS : " ,tcp_hdr[0] print "Destination Port ADDRESS : " , tcp_hdr[1] Issues : 1. Unable to capture any outgoing IPv4 traffic. I ran the sniff() method in Scapy and it does capture the outgoing packets. 2. I am NOT USING PROMISCUOUS MODE , still most of the packes I am receiving neither have my IP or MAC in either of the source or destination fields. 3. Captured data is different from the one observed using Scapy or Wireshark. Request you to kindly clarify these observations. Thanks and Regards, Ayush From rosuav at gmail.com Thu Nov 17 09:45:32 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 Nov 2016 01:45:32 +1100 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: On Thu, Nov 17, 2016 at 11:51 PM, Roland Hedberg wrote: >> A quick query of the PyPI download database for the three months shows the following download counts for those modules: >> >> - requests-oauthlib == 1,897,048 >> - oauth2 == 349,759 >> - pyoidc == 10,520 >> >> This is not intended to be chastening for Roland: all new modules start with low download counts. > > No offence taken ! :-) > But you should distinguish between OAuth2 and OIDC. OIDC is a profile of OAuth2 for usage in the case where you not only need > authorization (and access tokens) but also authentication and/or user info. When you're looking at oauth2, there are myriad uses for it, and thus large numbers of people looking for the module. But OIDC is something I had never heard of until this thread (turns out it's something built on top of OAuth2). Your module may well be best-in-show for OIDC (unproven, but assume it for the nonce), but unless it's also best-in-show for OAuth2, it's not going to have the broad draw/appeal that I would hope for in a new stdlib module. Perhaps the best step forward is to publish blog posts demonstrating how your module compares to other OAuth2 libraries. That would put the module name alongside various keywords that people will search for, and thus improve its visibility. Consider this thought process, which I'd say is fairly typical: 1) I want to use Fred's Wonderful Spamination API. 2) FWSA's docs say that I need to use this thing called OAuth. 3) What's OAuth? How do I use it? Search the web. 4) Oh, there's OAuth1 and OAuth2. Which should I use? Ahh, FWSA's docs say OAuth2. Okay. 5) I need a Python module that does OAuth2. 6) Search the web, or search PyPI? Personally, when I hit step 6, I search the web. PyPI search is exhaustive but not very usefully ranked (for this purpose). Searching for a keyword or protocol will give undue weight to a module whose name is simply that word, even if that module is terrible, unmaintained, etc, etc. Properly-ranked web search results are generally more useful in pointing me to the appropriate package, even if they're telling me to use something with a very different name. (Consider a search for "python http". You'll get httplib/http.client, but shortly after that, you get pointed to 'requests'.) As another bonus, blog posts of that nature will help to explain to more experienced devs "why should this matter to me". People who've already used requests-oauthlib are unlikely to reach for a new and unproven package without a good reason. So give them that reason! :) Also, as I mentioned earlier, the Python Wiki may well have an appropriate spot for this to be mentioned. It's worth a check. ChrisA From p.f.moore at gmail.com Thu Nov 17 10:15:08 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Nov 2016 15:15:08 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: On 17 November 2016 at 14:45, Chris Angelico wrote: > Personally, when I hit step 6, I search the web. PyPI search is > exhaustive but not very usefully ranked (for this purpose). Searching > for a keyword or protocol will give undue weight to a module whose > name is simply that word, even if that module is terrible, > unmaintained, etc, etc. Properly-ranked web search results are > generally more useful in pointing me to the appropriate package, even > if they're telling me to use something with a very different name. > (Consider a search for "python http". You'll get httplib/http.client, > but shortly after that, you get pointed to 'requests'.) > > As another bonus, blog posts of that nature will help to explain to > more experienced devs "why should this matter to me". People who've > already used requests-oauthlib are unlikely to reach for a new and > unproven package without a good reason. So give them that reason! :) Additionally, I look for simple usage examples. When I did search for OAuth, I got lots of hits for libraries, some even included "how to add OAuth to your Flask app" examples. But not one showed me how I should call a web service that uses OAuth from the Python interpreter prompt using that library. Contrast the first page of the requests documentation: >>> import requests >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass')) >>> r.status_code 200 >>> r.headers['content-type'] 'application/json; charset=utf8' >>> r.encoding 'utf-8' >>> r.text u'{"type":"User"...' >>> r.json() {u'private_gists': 419, u'total_private_repos': 77, ...} With that, I immediately see how to use the code. Something similar with an OAuth using service would be what I'm looking for. I'd consider that sort of use case focused documentation as a minimum for any library that was looking to be included in the stdlib. I know I've been arguing earlier in this thread that "OAuth may be a good candidate for the stdlib". Here, what I'm saying is "... but I don't see any library implementing it that's ready for stdlib inclusion". While oic may be highly standards-compliant, IMO it's not ready for stdlib inclusion without user-focused design and documentation. On the other hand, nor was any other OAuth package I found via a quick search. Paul From cory at lukasa.co.uk Thu Nov 17 10:28:31 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 17 Nov 2016 15:28:31 +0000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> <8ADA9A2B-2489-4B71-A411-81D6A089922C@lukasa.co.uk> Message-ID: > On 17 Nov 2016, at 13:42, Paul Moore wrote: > > On 17 November 2016 at 12:27, Cory Benfield wrote: >> This isn?t me disagreeing with you, just me pointing out that the fuzziness around this makes me nervous. It has been my experience that a large number of protocol implementations in the standard library are already struggling to meet their maintenance goals, and I?d be pretty reluctant about wanting to add to that burden. > > Regardless of how things go in the long term, I think it's good to > keep the debates open in this area, and I appreciate your comments. > I'll certainly be thinking about my personal answers to the questions > you raised, and I expect I'll change at least some of my views as a > result. Agreed. For what it?s worth, I?ll almost always find myself on the ?let?s not add it to the stdlib? side of that argument, but I?m entirely willing to lose those arguments. I think we?re best served by having voices on both sides of the debate who believe themselves to be right in the *general* case but are willing to treat each case on the merits. There are certainly lots of requests for addition to the stdlib I have no objection to: for example, data structure and algorithm implementations in the standard library almost always seem like no-brainers to me. So I agree, I?m going to keep an eye on this space as we move forward, and for my part I promise to treat each case on the merits, despite my general belief of ?small is beautiful?. Cory From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 17 12:26:21 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 18 Nov 2016 02:26:21 +0900 Subject: [Python-ideas] Unable to sniff outgoing traffic using raw sockets in python2.7 In-Reply-To: References: Message-ID: <22573.59453.125888.192458@turnbull.sk.tsukuba.ac.jp> Ayush, This list is for new ideas for developing the Python language and standard libraries. You are unlikely to get a useful answer here. You would be better off asking on python-list at python.org or Stack Overflow, which are channels devoted to helping you write code with Python. Regards, Ayush Aggarwal writes: > Following is my code : [code snipped] > Request you to kindly clarify these observations. From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 17 12:30:17 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 18 Nov 2016 02:30:17 +0900 Subject: [Python-ideas] Comfort zones on list [was: Technical possibilities ...] In-Reply-To: <20161117123030.GU3365@ando.pearwood.info> References: <582CCAB6.2030001@canterbury.ac.nz> <22573.7926.125503.317807@turnbull.sk.tsukuba.ac.jp> <20161117123030.GU3365@ando.pearwood.info> Message-ID: <22573.59689.880193.649768@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > > > Except Perl. > > > > Please don't. > > I don't think there is any need for that. No harm is done by a > little light-hearted banter relating to the rivalry between > programming language communities. I don't disagree with that principle, just with the evaluation of the tone, and even that is the difference between rgb.txt's grey49 and grey51. Calling old-style Perl one-liners "line noise" I would count as "light-hearted banter". "All languages have something to teach us -- but let's be really careful about which parts of Perl we study" I would count as lighthearted *and* making the point that we have our own style, which is pretty close to diametrically opposed to the one-liners that Perl is so useful for. "We have nothing to learn from Perl" in context (IIRC, Perl had not been mentioned in this thread to that point, and definitely the emphasis was on C/C++ and Java) was gratuitous and extreme, pure trash talking. Which has its place.[1] I don't think it's unreasonable to request tone-down when it's gratuitous. And I did mean "request" -- I wouldn't follow up to the OP if he chose to disagree with me, even in public. Here I'm discussing a principle that is not absolute, just one I'd like to advocate, and your post gives me a chance. > Regardless of what virtues Perl may or may not have, in some ways the > philosophy behind Perl is dramatically opposed to that of Python. So make that point. Just do it better than I did above! :-) > We should be permitted to criticise other languages, Sure. > and even dismiss them as useless. That's another matter, slightly different, but it's an important difference. There was an excellent blog post a couple years ago on "Contempt Culture"[2]. I deny that Nick's comment was made out of contempt, but I have to suspect that past contemptuous utterances provided context for it. The point of the term "culture" is that the contempt becomes natural in context, and indetectable to the speaker, and many of the listeners. > Programming languages do not have the assumption of dignity owed to > human beings. No, but their users are quite capable of *taking* offense where none was offered to them. And nasty people are quite capable of quoting out of context, just to see you and him fight. After a lifetime of using those facts as an excuse for my own borderline insults[3], I've decided I like the much more friendly style of the Python dev lists. It's not that there's no conflict here, but generally the arguments are delivered in such a way that people stay calm throughout. > this is not a support group for people so lacking in self-esteem > that their self-image is equated to their choice of programming > language. Of course not. But are you aware that many women are of the opinion that the Python dev community is an uncomfortable place to hang out? I don't know why yet, but the women who have told me they feel that way are quite capable of holding their own in a battle of words and wits when they want to, so they could handle python-dev if they wanted to. With as few reviewers as we have, I would rather risk convincing a few participants I'm some sort of "PC fanatic" than risk chasing away the brain power of half the human race. No, I don't *know* that humorous dismissals of whole programming environments contribute to that alleged atmosphere. I don't even have full understanding of what the issue is, just that I respect the women who have mentioned it to me enough to consider their statements to be evidence (not "proof"). I don't claim to have anything more than personal preference, and a very vague hope for increased participation from a group I think is currently underrepresented here, behind my request this time. > I'm sorry that I've written more words defending Nick's throw-away two > word comment than you expended in lightly chastising him. I'm not. These are very difficult and subtle issues, and you've expressed an important point of view eloquently. And I am very aware that current subscribers of the list have the right to express themselves with such humor, at least as I understand the current list policy.[4] I just wish such expression would be avoided, and I like to think my wish is partly motivated by a certain amount of public-spirited-ness, and that said motivation is in response to a real issue. Steve Footnotes: [1] Cf. http://www.jwz.org/gruntle/rbarip.html, though nothing could be like that any more. ;-) [2] http://blog.aurynn.com/contempt-culture. I suspect it's not the origin of the term, but don't feel like digging further. [3] My out-and-out insults were generally committed with malice aforethought, in the expectation of getting as good as I gave.[1] [4] I suspect that if asked, Brett would sympathize with my preference, and if not asked directly, he would say nothing precisely because anything he says might be construed to be policy, and this isn't. But you'd have to ask him. :-) From steve at pearwood.info Thu Nov 17 19:26:06 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Nov 2016 11:26:06 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <20161117000654.GS3365@ando.pearwood.info> Message-ID: <20161118002605.GV3365@ando.pearwood.info> On Thu, Nov 17, 2016 at 07:37:36AM +0100, Mikhail V wrote: > On 17 November 2016 at 01:06, Steven D'Aprano wrote: > > > It doesn't matter what keyword you come up with, you still have the > > problem that this idea introduces a new keyword. New keywords always > > break backwards compatibility, which means there has to be a good reason > > to do it. > > Any meaningful name you pick as a keyword will probably clash with > > somebody's code, which means you will break their working program by > > introducing a new keyword. > > I appreciate how you explain things, you point to the root of problems and > it always makes my logic better. Thank you :-) [...] > > But the first question > > we'd ask is, what do other languages do? Python rarely invents new > > programming ideas itself, but it does copy good ideas from other > > languages. > > I don't know why, but this sounds somehow pessimistic. In some ways it is. But Python is a conservative language (despite the "radical" feature of significant indentation, but even that was a proven concept from at least one other language, ABC, before Guido used the idea). Things may have been different two decades ago when Python was a brand-new language with a handful of users. Python is a mature language now with a huge user-base. It is probably one of the top five most popular languages in the world, and certainly one of the top ten. We have a responsibility to our users: - to minimise the amount of disruption to their code when they upgrade; - that means avoiding breaking backwards compatibility unless there is significant benefit to make up for the pain; - and long deprecation periods before removing obsolete features; - since it typically takes five years, or even ten, to remove a deprecated feature, we should avoid wasting users' time by adding new and exciting experimental features that turn out to be a bad idea. Nick Coghlan has a good blog post that explains the tension in the Python community between those who want the language to evolve faster, and those who want it to evolve slower: http://www.curiousefficiency.org/posts/2011/04/musings-on-culture-of-python-dev.html For people used to the rapid change in the application space (new versions of Chrome virtually daily; Ubuntu brings out new operating system versions every six months) Python seems to move really, really slowly. But for a programming language, Python moves radically fast: most C code written in the 1970s probably works correctly today, and it can easily be a decade between C versions. [...] > First main question then would be: > Are augmented assignment operators needed *much*? They are. They were one of the most often requested features before Python. Most people consider that they improve the quality of code. [...] > In this case the work on thinking out a better > syntax for in-place stuff is not so useless as it > seems to be now. Mikhail, you are missing the point that people have already spent decades thinking about "a better syntax for in-place stuff", and for Python that syntax is augmented assignment. I'm sorry that *you personally* don't like this syntax. That puts you in a very small minority. Not everybody likes every part of Python syntax. But I think it is time for you to give up: you cannot win this battle. > If Python will *someday* be able to optimise > the simple A = A + 1 to in-place without > need of special syntax (by type annotation probably?), I really, really hope not. Type annotations are irrelevant here. Python doesn't need type annotations to know the type of A at runtime. The advantage of augmented assignment is that it allows me, the programmer, to decide whether or not I want in-place array operations. It is *my* choice, not the compiler's, whether I create a new list: A = B = [1, 2, 3] A = A + [4] assert B == [1, 2, 3] or make the change in-place: A = B = [1, 2, 3] A += [4] assert B == [1, 2, 3, 4] [...] > WAS: > Binary_mask += numpy.sum(B, C) > > NEW: > 1). prefix keyword approach examples: > > incal Binary_mask + numpy.sum(B, C) > inc Binary_mask + numpy.sum(B, C) > calc Binary_mask + numpy.sum(B, C) > exec Binary_mask + numpy.sum(B, C) Those suggestions waste perfectly good and useful variable names as keywords (I have code that uses inc and calc as names, and exec is the name of a built-in function). And not one of those examples makes it clear that this is an assignment. Augmented assignment does make it clear: it uses a variation on the = binding operator. (Its not actually an operator, but for lack of a better name, I'll call it one.) It follows the same basic syntax as regular assignment: target = expression except that he augmented operator is inserted before the equals sign: target -= expression target += expression target *= expression etc. At this point, I think it is a waste of time to continue discussing alternatives to augmented assignment syntax. I'm happy to discuss the culture of Python and how we decide on making changes to the language, but I am not interested in discussing augmented assignment itself. -- Steve From ncoghlan at gmail.com Thu Nov 17 22:58:04 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Nov 2016 13:58:04 +1000 Subject: [Python-ideas] Support for OAuth2/OIDC in the standard distribution ? In-Reply-To: References: <9DE9541A-D15F-45DE-A27A-F2AB3E10B58F@catalogix.se> <11F01071-94A8-47FD-A3F5-DD86FC984ECA@lukasa.co.uk> <78514CFA-9421-4B04-AF7A-EE3417714111@lukasa.co.uk> <91180E21-C6B5-4FA2-86BE-5AC97B9B52D3@lukasa.co.uk> Message-ID: On 17 November 2016 at 21:35, Paul Moore wrote: > On 17 November 2016 at 10:58, Cory Benfield wrote: >> Instead, I think we need a way to be able to ask the question: ?what does the wider Python development community consider to be the gold standard for solving problem X??. > > Agreed, that's the key unsolved question for Python packaging. Not just Python packaging - open source publishing in general, and one of the big metrics we have providing evidence of this is hosted package growth rates. Some registries try to present exponential growth in the number of hosted packages as a good thing, but they're often wrong to do so: in most cases, that kind of exponential growth is more likely to represent a failure of software discovery mechanisms (so folks are publishing their own custom solutions to previously solved problems rather than adopting existing tools as "good enough") than it is actual growth in the number of different problem domains with readily available published toolkits for tackling them. That's fine in a software-as-creativity-and-play context, but it's a problem in the software-as-a-means-to-an-end mindset that is applicable to most professional development activities. The one upside I see to the current state of affairs is that this problem isn't *new* - it's existed for as long as we've had software, it was just hidden away behind the walls of the institutions writing custom in-house software. Now that more software publication and consumption activities are instead starting to happen in the open, the problem can be quantified, and various automated techniques brought to bear on tackling it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mikhailwas at gmail.com Fri Nov 18 08:06:42 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 18 Nov 2016 14:06:42 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <20161118002605.GV3365@ando.pearwood.info> References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> Message-ID: On 18 November 2016 at 01:26, Steven D'Aprano wrote: > At this point, I think it is a waste of time to continue discussing > alternatives to augmented assignment syntax. I'm happy to discuss the > culture of Python and how we decide on making changes to the language, > but I am not interested in discussing augmented assignment itself. Fair and honest answer. How one would discuss then about "how we decide on making changes" at all? In this particular case, of course I am not interested in assignment operator *itself* but I am very interested how the judgement on syntax would see, namely I was very interested to see how people would imagine writing something in some other way and how it feels. Thats it. This is kind of too naive, but such questions I think will and should appear. And as I see there are quite a lot of absolutely different opinions on it. Some probably even consider not worth it even looking at examples, since it from beginning will return "syntax error". Yes there were such kind of answers (not in this thread, but still I've seen such approach). I observed the previous thread, where it was asked about changing += to =+, namely change the direction of symbols and there indeed followed a bunch of answers, that a =- 2 will be same as: a = -2 Well, kind of true, but you know, not necesserily the original poster did not know it. And what is much more important, that apart from the compatibility and tradition issue, there could be still something to think about, so one probably should try to "read between the lines" and see how the OP comes to this idea. Yes, it is probably dozen of people (I don't know) who decides for syntax changes, but the amount of potential users is huge and I personally tend to look in the future rather than looking into other languages or worrying *too much* about oldies who got used to some particular syntax. I will become one day an old fart too, but I will never be angry on those who uses better syntax and that my syntax is obsolete now. For me personally switching to new syntax, if it looks better, would be only pleasure and kind of interesting experience. So it depends and I don't want to hurt noboby with that. As for backward compatibility issues: it is not my destiny, but there is an analogy with my earlier discussion about numeric represantations. One wants to keep the amount of Python versions minimal and make minimal changes and thats right. But IMHO one should probably make even less versions of Python and *longer* period before new version, but with *more* radical changes to the syntax, the question is where do you find the people who take responsibilty, that these changes will be good. In similar way, there is no sense to make small corrections to numeric representation or writing, but it is important to keep the work on it and only rarely make radical improvements. A good example is relatively new "format" method for printing strings. I remember reading a thread on SO and I was almost shocked how some people write about it. Even given the fact that nobody prohibited the old % formatter, in some comments I literally feel the hate. And it is really sad, since it is just a change to a better readable syntax. In this moment I really felt sorry for python developers who want to make something good to people and must hear this. I would punish those and take away the % formatter. (that was a joke:). Mikhail From mikhailwas at gmail.com Fri Nov 18 14:19:28 2016 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 18 Nov 2016 20:19:28 +0100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <20161118002605.GV3365@ando.pearwood.info> References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> Message-ID: On 18 November 2016 at 01:26, Steven D'Aprano wrote: Sorry, just for a second one more comment to your comment. BTW, Steve, there is no any "battle", only peace and pleasure. And exchange of knowledge and opinions of adult intelligents. > Mikhail, you are missing the point that people have already spent > decades thinking about "a better syntax for in-place stuff", and for > Python that syntax is augmented assignment. Wikipedia says they are all the same in all languages. Probably you meant something else. But if there are alternatives, I would look into it. Especially if they are a product of "decades" of work of titans of typography, this must be something divinely beautiful in contrast to current ones. > [...] >> WAS: >> Binary_mask += numpy.sum(B, C) >> >> NEW: >> 1). prefix keyword approach examples: >> >> incal Binary_mask + numpy.sum(B, C) >> inc Binary_mask + numpy.sum(B, C) >> calc Binary_mask + numpy.sum(B, C) >> exec Binary_mask + numpy.sum(B, C) > [...] not one of those examples makes it > clear that this is an assignment. But is it not because you see it first time in your life? That is just one point on the problem of judgement of syntax, regardless of what it be. > Augmented assignment does make it clear: it uses a variation on the = > binding operator. (Its not actually an operator, but for lack of a > better name, I'll call it one.) It follows the same basic syntax as > regular assignment: > > target = expression > > except that he augmented operator is inserted before the equals sign: > > target -= expression > target += expression > target *= expression Which makes these two lines: target -= expression target = expression Quite similar visually but with *absolutely* different meaning. Steven, you always give reviews and advices which exceed all expectations in quality and clearness and I learned a lot from you, serious. But I dare me a light nitpick: probably syntaxes is not your element ;) Mikhail From srkunze at mail.de Fri Nov 18 14:27:23 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 18 Nov 2016 20:27:23 +0100 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: <20161106080711.GV3365@ando.pearwood.info> References: <20161106080711.GV3365@ando.pearwood.info> Message-ID: <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> On 06.11.2016 09:07, Steven D'Aprano wrote: > I'm having a lot of difficulty in understanding your use-case here, > and so maybe I've completely misunderstood something. Although, this thread is dead for a week or so, I am still curious to hear the real-world use-case. I am equally puzzled by the fact that somebody really wants to use a context manager not to work like a context manager; without even considering not to use context managers at all and using regular functions instead. Cheers, Sven From ethan at stoneleaf.us Fri Nov 18 14:31:21 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 18 Nov 2016 11:31:21 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> Message-ID: <582F5709.8030105@stoneleaf.us> On 11/18/2016 11:19 AM, Mikhail V wrote: > On 18 November 2016 at 01:26, Steven D'Aprano wrote: I'm not seeing D'Aprano's posts, except in Mikhail's replies -- Mikhail, is he replying directly to you? If so, please stop including Python Ideas in your replies. Thank you. -- ~Ethan~ From ethan at stoneleaf.us Fri Nov 18 14:49:22 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 18 Nov 2016 11:49:22 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> <582F5709.8030105@stoneleaf.us> Message-ID: <582F5B42.5020404@stoneleaf.us> On 11/18/2016 11:34 AM, Chris Angelico wrote: > Hmm. I'm seeing them. Where are you reading -ideas? Is there possibly > a glitch between mailing list and newsgroup? Yeah, I'm using the mailing list. On 11/18/2016 11:39 AM, Mikhail V wrote: > I am really sorry :( Didn't intended it > Something with my mailer it showed me python-ideas in to: in Stevens reply. Looks like it's not an issue with you, Mikhail, my apologies. I'll ask Brett to check it out. -- ~Ethan~ From ram at rachum.com Fri Nov 18 15:42:29 2016 From: ram at rachum.com (Ram Rachum) Date: Fri, 18 Nov 2016 22:42:29 +0200 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> References: <20161106080711.GV3365@ando.pearwood.info> <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> Message-ID: Sure, here are a couple of use cases: 1. I'm making a program that lets people lease machines. They can issue a command to lease 7 machines. When they do, my program leases them one by one and adds them all to an exit stack, so in case there aren't 7 machines available, all the machines we leased get released and the situation is back to normal. If everything goes fine, I do pop_all on the exit stack so it doesn't get exited and the machines stay leased, then the command exits and the user gets his machines. 2. I have a test suite that creates a temporary folder to put files that are used by the test. The temporary folder is created by a context manager that deletes it at the end. But, if the test fails I want to move the temporary folder away into a dedicated folder for the user to be able to examine those files later to figure out why the test fails. So I want to tell the temporary folder context manager to not delete the folder, because it'll fail since it was moved away so it's not at the expected location. (If you're replying please keep me in "to" because the Gmail filter I set up isn't smart enough to let the messages in otherwise.) On Nov 18, 2016 21:27, "Sven R. Kunze" wrote: > On 06.11.2016 09:07, Steven D'Aprano wrote: > >> I'm having a lot of difficulty in understanding your use-case here, and >> so maybe I've completely misunderstood something. >> > > Although, this thread is dead for a week or so, I am still curious to hear > the real-world use-case. I am equally puzzled by the fact that somebody > really wants to use a context manager not to work like a context manager; > without even considering not to use context managers at all and using > regular functions instead. > > Cheers, > Sven > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Nov 18 15:52:31 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 18 Nov 2016 12:52:31 -0800 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: <20161106080711.GV3365@ando.pearwood.info> <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> Message-ID: <582F6A0F.4000905@stoneleaf.us> On 11/18/2016 12:42 PM, Ram Rachum wrote: > Sure, here are a couple of use cases: > > 1. I'm making a program that lets people lease machines. They can issue > a command to lease 7 machines. When they do, my program leases them one > by one and adds them all to an exit stack, so in case there aren't 7 > machines available, all the machines we leased get released and the > situation is back to normal. If everything goes fine, I do pop_all on > the exit stack so it doesn't get exited and the machines stay leased, > then the command exits and the user gets his machines. So you're using the contextlib.ExitStack context manager? > 2. I have a test suite that creates a temporary folder to put files that > are used by the test. The temporary folder is created by a context > manager that deletes it at the end. But, if the test fails I want to > move the temporary folder away into a dedicated folder for the user to > be able to examine those files later to figure out why the test fails. > So I want to tell the temporary folder context manager to not delete > the folder, because it'll fail since it was moved away so it's not at > the expected location. My first thought is to make a custom context manager for this use-case, but if you really want to use a generator instead can't you just put in an existence check for the directory and only delete if it is still there? -- ~Ethan~ From greg.ewing at canterbury.ac.nz Fri Nov 18 16:00:10 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 Nov 2016 10:00:10 +1300 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: References: <20161106080711.GV3365@ando.pearwood.info> <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> Message-ID: <582F6BDA.2060407@canterbury.ac.nz> Ram Rachum wrote: > 1. I'm making a program that lets people lease machines. They can issue > a command to lease 7 machines. ... If everything goes fine, I do pop_all on > the exit stack so it doesn't get exited and the machines stay leased, Seems to me that would be done more easily and clearly without involving a context manager at all: machines = [] try: for i in range(num_machines_required): machines.append(lease_machine()) except AllMachinesInUseError: for machine in machines: release_machine(machine) del machines[:] A context manager is a tool to be used if it helps. If it doesn't help, don't be afraid to not use it! > 2. I have a test suite that creates a temporary folder to put files that > are used by the test. ... But, if the test fails I want to > move the temporary folder away into a dedicated folder Again, don't use a context manager, just write a try-except that does what you want. -- Greg From steve at pearwood.info Fri Nov 18 18:13:00 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 Nov 2016 10:13:00 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <582F5709.8030105@stoneleaf.us> References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> <582F5709.8030105@stoneleaf.us> Message-ID: <20161118231259.GW3365@ando.pearwood.info> On Fri, Nov 18, 2016 at 11:31:21AM -0800, Ethan Furman wrote: > On 11/18/2016 11:19 AM, Mikhail V wrote: > >On 18 November 2016 at 01:26, Steven D'Aprano wrote: > > I'm not seeing D'Aprano's posts, except in Mikhail's replies -- Mikhail, is > he replying directly to you? If so, please stop including Python Ideas in > your replies. I'm not replying off-list, but something has clearly gone wrong with the mailing list. (I'm BCC'ing the list owners.) My earlier post (the one Mikhail has replied to) shows up in the ActiveState archive: http://code.activestate.com/lists/python-ideas/43270/ but not in the official python-list archive: https://mail.python.org/pipermail/python-ideas/2016-November/thread.html Later I see a reply from Ethan to a question posed by Chris Angelico http://code.activestate.com/lists/python-ideas/43276/ but I never received Chris' question. Neither Chris' question nor Ethan's response show up in the pipermail archive. -- Steve From rosuav at gmail.com Fri Nov 18 18:22:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 Nov 2016 10:22:45 +1100 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <20161118231259.GW3365@ando.pearwood.info> References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> <582F5709.8030105@stoneleaf.us> <20161118231259.GW3365@ando.pearwood.info> Message-ID: On Sat, Nov 19, 2016 at 10:13 AM, Steven D'Aprano wrote: > Later I see a reply from Ethan to a question posed by Chris Angelico > > http://code.activestate.com/lists/python-ideas/43276/ > > but I never received Chris' question. Neither Chris' question nor > Ethan's response show up in the pipermail archive. Part of that is because I actually responded off-list (though I don't mind that it went back on-list). However, Ethan's response should have gone to the list and the archive. ChrisA From ncoghlan at gmail.com Fri Nov 18 23:10:37 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Nov 2016 14:10:37 +1000 Subject: [Python-ideas] Generator-based context managers can't skip __exit__ In-Reply-To: <582F6BDA.2060407@canterbury.ac.nz> References: <20161106080711.GV3365@ando.pearwood.info> <4420df90-c4bd-d425-73ec-5161b96cdfa5@mail.de> <582F6BDA.2060407@canterbury.ac.nz> Message-ID: On 19 November 2016 at 07:00, Greg Ewing wrote: > Ram Rachum wrote: >> >> 1. I'm making a program that lets people lease machines. They can issue a >> command to lease 7 machines. ... If everything goes fine, I do pop_all on >> the exit stack so it doesn't get exited and the machines stay leased, > > > Seems to me that would be done more easily and clearly > without involving a context manager at all ExitStack() handles that case by design - doing "pop_all() gives you a fresh ExitStack() instance, while making close() and __exit__() on the original no-ops. Since ExitStack() deliberately never cleans up the stack implicitly, you end up being able to do: machines = [] with contextlib.ExitStack() as cleanup: for i in range(num_machines_required): machines.append(lease_machine()) cleanup.callback(release_machine, machine) cleanup.pop_all() # It worked, so don't clean anything up yet return machines However, ExitStack *itself* can't readily be written using contextlib.contextmanager - it's not impossible, but it's convoluted enough not to be worth the hassle, since you'd need to do something like: class _ExitStack: # Like the current ExitStack, but without __enter__/__exit__ @contextmanager def exit_stack() state = _ExitStack() try: yield state finally: state.close() Externally, the visible differences would be that: - "es_cm = exit_stack()" and "es_state = exit_stack().__enter__()" would give different results - it would be slower and use more memory due to the additional layer of indirection Since the extra layer of indirection doesn't buy you any new capabilities and has a runtime performance cost, there's no real reason to do it. The general case of that pattern is to yield a state variable that just has a single attribute "needs_cleanup": @contextmanager def my_cm() ... # Do setup operations here state = types.SimpleNamespace(needs_cleanup=True) try: yield state finally: if state.needs_cleanup: ... # Do cleanup operations here However, as with ExitStack, at that point, you're usually going to be better off just implementing __enter__ and __exit__ yourself, and not worrying about using the generator format. This kind of difference in flexibility isn't really specific to context managers though - it's a particular instance of the general pattern that custom classes are often more convenient than closures when you actually *want* to expose externally mutable state. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From Nikolaus at rath.org Fri Nov 18 23:30:27 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 18 Nov 2016 20:30:27 -0800 Subject: [Python-ideas] Technical possibilities for a syntax [was: Reverse assignment operators ...] In-Reply-To: <582F5709.8030105@stoneleaf.us> (Ethan Furman's message of "Fri, 18 Nov 2016 11:31:21 -0800") References: <20161117000654.GS3365@ando.pearwood.info> <20161118002605.GV3365@ando.pearwood.info> <582F5709.8030105@stoneleaf.us> Message-ID: <8737io9jmk.fsf@vostro.rath.org> On Nov 18 2016, Ethan Furman wrote: > On 11/18/2016 11:19 AM, Mikhail V wrote: >> On 18 November 2016 at 01:26, Steven D'Aprano wrote: > > I'm not seeing D'Aprano's posts, except in Mikhail's replies FWIW, I can see them on Gmane. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From bookoftrust at gmail.com Mon Nov 21 18:52:06 2016 From: bookoftrust at gmail.com (andreas) Date: Mon, 21 Nov 2016 18:52:06 -0500 Subject: [Python-ideas] Contribution Reward System Message-ID: <20161121235206.GO25747@localhost.localdomain> Hi, I was thinking if a reward system for contributions would add motivation to those who want to contribute. Something in the lines of a reputation system. This has been done before with success. Thanks From steve at pearwood.info Mon Nov 21 12:16:30 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 22 Nov 2016 04:16:30 +1100 Subject: [Python-ideas] Contribution Reward System In-Reply-To: <20161121235206.GO25747@localhost.localdomain> References: <20161121235206.GO25747@localhost.localdomain> Message-ID: <20161121171628.GX3365@ando.pearwood.info> On Mon, Nov 21, 2016 at 06:52:06PM -0500, andreas wrote: > Hi, > > I was thinking if a reward system for contributions would add motivation > to those who want to contribute. Or, it could destroy the motivation of those who want to contribute for the love of the language and a desire to give back to the community, and encourage those who don't care about Python or the community but are motivated by the selfish desire to collect points and rewards. Giving people rewards is very risky. You can just as easily encourage behaviour you didn't intend to. Before coming to your suggested solution ("Rewards! Upvotes! Downvotes! Reputation Points! Badges!") perhaps you can explain what problem you hope to solve. > Something in the lines of a reputation system. If Python had a reputation system from the beginning, I expect that Guido's reputation score would be about -10000 by now because he won't add braces to the language. > This has been done before with success. Oh? Where? Can you give us more detail please? -- Steve From mal at egenix.com Mon Nov 21 12:27:02 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 21 Nov 2016 18:27:02 +0100 Subject: [Python-ideas] Contribution Reward System In-Reply-To: <20161121235206.GO25747@localhost.localdomain> References: <20161121235206.GO25747@localhost.localdomain> Message-ID: <58332E66.1040106@egenix.com> Hi Andreas, we already have such a system. The awards are collectively called Python Community Awards; they come at different levels and are granted by the PSF: https://www.python.org/community/awards/ The CSAs are probably the ones you are thinking about: https://www.python.org/community/awards/psf-awards/ Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 21 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 22.11.2016 00:52, andreas wrote: > Hi, > > I was thinking if a reward system for contributions would add motivation > to those who want to contribute. > > Something in the lines of a reputation system. > > This has been done before with success. > > Thanks > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Nov 21 22:17:41 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 22 Nov 2016 12:17:41 +0900 Subject: [Python-ideas] Contribution Reward System In-Reply-To: <20161121235206.GO25747@localhost.localdomain> References: <20161121235206.GO25747@localhost.localdomain> Message-ID: <22579.47317.209943.38926@turnbull.sk.tsukuba.ac.jp> andreas writes: > I was thinking if a reward system for contributions would add > motivation to those who want to contribute. > > Something in the lines of a reputation system. I think if you look at the number of patches sitting on the tracker waiting to be reviewed, you'll see that we have plenty of patch submissions from less experienced contributors. Probably the best motivation for those contributors would be halving the time to review by committers. But new committers don't grow on trees, and they tend to be very busy people. They're generally not motivated by badges. It's true that there are jobs that less experienced contributors *can* do that are un-fun, like confirming defect reports and even triaging issues. These might be motivated by a reputation system, but they're also easy to do sloppily if done too quickly. So we might need reviewers to check them. As Steven d'A points out, it may be more straightforward to have them done by people who are motivated to put in careful hard work in the first place. Documentation is another area where new contributors can often make contributions, but again these contributions need review or they can do more harm than good (and that's true for experienced contributors as well). One practice that worked pretty well as motivation a few years ago was "review exchange" where a committer would offer one patch review in return for confirmation and triage of five new issues. But I haven't seen that offer made explicitly for years. While I agree with Steven that we need to be cautious about changing the reward structure in principle, and there will also be practical issues of both workflow changes and coding new features since our mailing lists and issue tracker don't support reputations and badges, if you want to push this forward, I'd recommend being more explicit about the "successful examples" you mention, and describing best practices in this area. Steve From alexander.belopolsky at gmail.com Tue Nov 22 12:43:39 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 22 Nov 2016 12:43:39 -0500 Subject: [Python-ideas] Contribution Reward System In-Reply-To: <22579.47317.209943.38926@turnbull.sk.tsukuba.ac.jp> References: <20161121235206.GO25747@localhost.localdomain> <22579.47317.209943.38926@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Nov 21, 2016 at 10:17 PM, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > One practice that worked pretty well as motivation a few years ago was > "review exchange" where a committer would offer one patch review in > return for confirmation and triage of five new issues. But I haven't > seen that offer made explicitly for years. > I am not sure I made this offer publicly before, but with respect to datetime module contributions, I have an outstanding offer to fill in the C implementation for any patch that is otherwise complete (feature/bug fix proposal is accepted, implemented in Python and the patch includes tests and necessary documentation.) I am also happy to offer a "review exchange", but only within my areas of expertise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bookoftrust at gmail.com Tue Nov 22 14:32:10 2016 From: bookoftrust at gmail.com (andreas) Date: Tue, 22 Nov 2016 14:32:10 -0500 Subject: [Python-ideas] Freelance Marketplace Message-ID: <20161122193210.GD8986@localhost.localdomain> Hi, With freelancing and freelancing marketplaces being so popular, why not have a platform especially for Python related tasks and only limit the community to a job board ? Andreas From fleblanc50 at gmail.com Tue Nov 22 16:07:17 2016 From: fleblanc50 at gmail.com (fleblanc50) Date: Tue, 22 Nov 2016 22:07:17 +0100 Subject: [Python-ideas] Decorator to avoid a mistake Message-ID: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Hi there, I like python easy extend class and reusse code, but sometime I overwrite involontary some functions or variables from parents. I think about a decorator to confirm overwrite is intended and put a warning if is not present. class A: def hello(self): print('Hello A') class B: def hello(self): print('Hello B') class C: @overwrite def hello(self): print('Hello C') b=B() c=C() b.hello() Warning overwrite hello method... Hello B c.hello() Hello C Dont know if this should be add to language this way or another, or if this should be in pylint only ... Perhaps someone get better way to achieve this or something already exist? May the python be with you, Regards From adrian.orive.oneca at gmail.com Tue Nov 22 16:24:27 2016 From: adrian.orive.oneca at gmail.com (=?UTF-8?Q?Adri=C3=A1n_Orive_Oneca?=) Date: Tue, 22 Nov 2016 22:24:27 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: This is just a convenience utility that would impact performance. This kind of enhancements, in my opinion, should be taken care by the IDEs, not by the interpreters. -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Nov 22 17:49:37 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 22 Nov 2016 16:49:37 -0600 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: I think you could implement this yourself with metaclasses and it wouldn't have much (if any) performance hit per-call or per-instantiation (just a bit slower when creating the class definition). It's a bit heavy-handed-hand-holding?if you ask me?but if you want to do it, the language gives you the ability. On Tue, Nov 22, 2016 at 3:24 PM, Adri?n Orive Oneca < adrian.orive.oneca at gmail.com> wrote: > This is just a convenience utility that would impact performance. This > kind of enhancements, in my opinion, should be taken care by the IDEs, not > by the interpreters. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 22 19:08:43 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Nov 2016 11:08:43 +1100 Subject: [Python-ideas] Freelance Marketplace In-Reply-To: <20161122193210.GD8986@localhost.localdomain> References: <20161122193210.GD8986@localhost.localdomain> Message-ID: <20161123000842.GY3365@ando.pearwood.info> On Tue, Nov 22, 2016 at 02:32:10PM -0500, andreas wrote: > Hi, > > With freelancing and freelancing marketplaces being so popular, Are they? I'm afraid I'm not really sure what you mean by "freelancing marketplaces". > why not have a > platform especially for Python related tasks and only limit the community to a > job board ? Can you explain what you mean by "a platform especially for Python related tasks"? How is that different from a job board? -- Steve From ncoghlan at gmail.com Wed Nov 23 01:33:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Nov 2016 16:33:22 +1000 Subject: [Python-ideas] Freelance Marketplace In-Reply-To: <20161122193210.GD8986@localhost.localdomain> References: <20161122193210.GD8986@localhost.localdomain> Message-ID: On 23 November 2016 at 05:32, andreas wrote: > Hi, > > With freelancing and freelancing marketplaces being so popular, why not have a > platform especially for Python related tasks and only limit the community to a > job board ? Such services already exist for the wider open source community (e.g. bountysource.com), and folks willing to advocate for and implement Python changes are free to sign up for those platforms if they want to do so. Anyone that chooses to do so is also free to start a task funding market specifically for the Python community and encourage both developers and end users to sign up. However, neither of those approaches requires the review and approval of the CPython core development team as a whole, so they're off-topic for this list (which is intended for discussion of language design ideas, rather than sustaining engineering funding models). The one aspect of contribution funding that's specific to core development is the page at https://docs.python.org/devguide/motivations.html, where core developers that *are* open to freelancing opportunities can state that, and provide links to other sites with the relevant details. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fleblanc50 at gmail.com Wed Nov 23 03:08:57 2016 From: fleblanc50 at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Leblanc?=) Date: Wed, 23 Nov 2016 09:08:57 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: It's why I'd prefer this integrate in language, but if there no way to get it without performance cost I will have a look to a pylint solution... 2016-11-22 23:49 GMT+01:00 Nick Timkovich : > I think you could implement this yourself with metaclasses and it wouldn't > have much (if any) performance hit per-call or per-instantiation (just a > bit slower when creating the class definition). > > It's a bit heavy-handed-hand-holding?if you ask me?but if you want to do > it, the language gives you the ability. > > On Tue, Nov 22, 2016 at 3:24 PM, Adri?n Orive Oneca < > adrian.orive.oneca at gmail.com> wrote: > >> This is just a convenience utility that would impact performance. This >> kind of enhancements, in my opinion, should be taken care by the IDEs, not >> by the interpreters. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 23 04:10:34 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Nov 2016 09:10:34 +0000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: On 23 November 2016 at 08:08, Fran?ois Leblanc wrote: > It's why I'd prefer this integrate in language, but if there no way to get > it without performance cost > I will have a look to a pylint solution... The point here is that if there is a way to get it without a performance cost (I can't imagine there would be, it's got to add a check for whether you're overriding a method at a minimum, but you may be able to come up with something) then you can do that within the language as it stands. It may need a specialised metaclass, but that's not a problem. At least, if it *is* an issue to you, you can argue for having it be the behaviour of the default metaclass once you've demonstrated that it works and doesn't hurt performance. And if there is a performance hit, then having the behaviour as opt-in based on a custom metaclass means that people can choose if they want to pay that cost. Paul From fleblanc50 at gmail.com Wed Nov 23 04:29:45 2016 From: fleblanc50 at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Leblanc?=) Date: Wed, 23 Nov 2016 10:29:45 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: I can imagine using a metaclass specialized witch can be activate or desactivate but the cost of decorator call still be here... I think its will be a good improvement if we can provide a solution for this, and i ask myself if this can be set in interpreter with a flag to activate for exemple and with no performance cost without the flag set, so you will have the choice and a good way to check youre code. To have no performance cost perhaps the best way it's to use docstring and pylint... 2016-11-23 10:10 GMT+01:00 Paul Moore : > On 23 November 2016 at 08:08, Fran?ois Leblanc > wrote: > > It's why I'd prefer this integrate in language, but if there no way to > get > > it without performance cost > > I will have a look to a pylint solution... > > The point here is that if there is a way to get it without a > performance cost (I can't imagine there would be, it's got to add a > check for whether you're overriding a method at a minimum, but you may > be able to come up with something) then you can do that within the > language as it stands. It may need a specialised metaclass, but that's > not a problem. At least, if it *is* an issue to you, you can argue for > having it be the behaviour of the default metaclass once you've > demonstrated that it works and doesn't hurt performance. And if there > is a performance hit, then having the behaviour as opt-in based on a > custom metaclass means that people can choose if they want to pay that > cost. > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skreft at gmail.com Wed Nov 23 16:29:35 2016 From: skreft at gmail.com (Sebastian Kreft) Date: Thu, 24 Nov 2016 08:29:35 +1100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: Related thread https://mail.python.org/pipermail/python-ideas/2016-July/041095.html On Nov 23, 2016 20:30, "Fran?ois Leblanc" wrote: > > I can imagine using a metaclass specialized witch can be activate or > desactivate but the cost of decorator > call still be here... > > I think its will be a good improvement if we can provide a solution for > this, and i ask myself if this can be set > > in interpreter with a flag to activate for exemple and with no performance > cost without the flag set, so you > > will have the choice and a good way to check youre code. > > To have no performance cost perhaps the best way it's to use docstring and > pylint... > > > > 2016-11-23 10:10 GMT+01:00 Paul Moore : > >> On 23 November 2016 at 08:08, Fran?ois Leblanc >> wrote: >> > It's why I'd prefer this integrate in language, but if there no way to >> get >> > it without performance cost >> > I will have a look to a pylint solution... >> >> The point here is that if there is a way to get it without a >> performance cost (I can't imagine there would be, it's got to add a >> check for whether you're overriding a method at a minimum, but you may >> be able to come up with something) then you can do that within the >> language as it stands. It may need a specialised metaclass, but that's >> not a problem. At least, if it *is* an issue to you, you can argue for >> having it be the behaviour of the default metaclass once you've >> demonstrated that it works and doesn't hurt performance. And if there >> is a performance hit, then having the behaviour as opt-in based on a >> custom metaclass means that people can choose if they want to pay that >> cost. >> >> Paul >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Nov 24 17:48:32 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 24 Nov 2016 23:48:32 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: Similar or related issue recently open and quickly closed: http://bugs.python.org/issue28776 "Duplicate method names should be an error" In short, such job should be done by linters. I'm quite sure that many of them already implement such check. Victor From rikudou__sennin at live.com Fri Nov 25 14:13:54 2016 From: rikudou__sennin at live.com (adil gourinda) Date: Fri, 25 Nov 2016 19:13:54 +0000 Subject: [Python-ideas] Python Documentation Message-ID: This is a small example of my idea, If you appreciate my little work I will continue working on it. If no, thank you for your attention and your comprehension :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Python Library Reference's Notes (v0.0).odt Type: application/vnd.oasis.opendocument.text Size: 17805 bytes Desc: Python Library Reference's Notes (v0.0).odt URL: From prometheus235 at gmail.com Fri Nov 25 14:21:28 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Fri, 25 Nov 2016 14:21:28 -0500 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: You can do it at run-time, if you so desire, without a measurable performance hit with a metaclass. Here's a hacky demo: https://gist.github.com/nicktimko/5f08d6adfa1dbe1319c3bfc715ec0aa4#file-override_guard-ipynb (Pedants: Any performance hit will be constant-time and probably less than a stray import that you don't need. If you're worried about optimizing constant-time things, I think you have larger problems.) On Thu, Nov 24, 2016 at 5:48 PM, Victor Stinner wrote: > Similar or related issue recently open and quickly closed: > http://bugs.python.org/issue28776 > "Duplicate method names should be an error" > > In short, such job should be done by linters. I'm quite sure that many > of them already implement such check. > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamespic at gmail.com Fri Nov 25 15:37:21 2016 From: jamespic at gmail.com (James Pic) Date: Fri, 25 Nov 2016 21:37:21 +0100 Subject: [Python-ideas] Distribution agnostic Python project packaging In-Reply-To: References: Message-ID: Hi all, Please let me thank you for sharing some of your insight and passion, and for your tolerance, I'm sorry I thought it would be the best mailing list to go ahead and bluntly propose to have something like jars in python core. It's really great to see such a variety of solutions, and I've been trying to study some of them. The container image based solutions using LXD is definitely somewhere we're going on the long term, and conda seems like a good replacement for virtualenv in production. Best regards, James, from Angoul?me B) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamespic at gmail.com Fri Nov 25 15:39:36 2016 From: jamespic at gmail.com (James Pic) Date: Fri, 25 Nov 2016 21:39:36 +0100 Subject: [Python-ideas] Distribution agnostic Python project packaging In-Reply-To: References: Message-ID: PS: FTR, pyinstaller seems widely use by a part of our community, but wasn't represented in this thread, perhaps this can give some good thinking to our devops community too :) http://www.pyinstaller.org/? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Nov 25 19:32:58 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 26 Nov 2016 11:32:58 +1100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: <20161126003257.GC3365@ando.pearwood.info> On Fri, Nov 25, 2016 at 02:21:28PM -0500, Nick Timkovich wrote: > You can do it at run-time, if you so desire, without a measurable > performance hit with a metaclass. Here's a hacky demo: > https://gist.github.com/nicktimko/5f08d6adfa1dbe1319c3bfc715ec0aa4#file-override_guard-ipynb All I get at that page is "Sorry, something went wrong." Don't you love informative error messages? > (Pedants: Any performance hit will be constant-time and probably less than > a stray import that you don't need. If you're worried about optimizing > constant-time things, I think you have larger problems.) time.sleep(360000) is constant time *wink* Sorry I couldn't resist... But seriously... this sort of check belongs in a linter, not in the core language, because its not necessarily an error or a mistake to override an existing method. In subclasses, the ability to create a method with the same name as one in a parent class is fundamental to how inheritence works, and even within a single class there's a use (admittedly uncommon) for re-using the same name: class Spam: def method(self): ... if condition: method = wrapper(method) The down-side of this flexibility and power is that more responsibility is placed in the hands of the programmer, which is hard on beginners. Sometimes I think Python-as-a-teaching-language and Python-as-a- production-language are strongly opposed. I wonder whether there might be a case to be made for a --with-training-wheels option? But that's better placed in the IDE, not the core language. I think this request is perhaps better suited as a feature request for IDLE rather than a language feature. -- Steve From guido at python.org Fri Nov 25 22:26:51 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 25 Nov 2016 19:26:51 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <20161126003257.GC3365@ando.pearwood.info> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: This idea is being kicked around from forum to forum and nobody wants to have it. Here it's brought up from time to time and the response is usually "let a linter do it". In mypy (which is essentially a powerful linter) it was proposed ( https://github.com/python/mypy/issues/1888) and the response was essentially "better go to python-ideas or bugs.python.org". It's also been proposed as a PEP 484 feature: https://github.com/python/typing/issues/269#issuecomment-243765549 . I think one reason why such proposals are unwelcome to experienced users may be that when done right this is totally legitimate, and the requirement to use an @override decorator is just making code more verbose with very little benefit. (I could see a benefit -- if this is used consistently it could make spelunking a large codebase easier, because you would know which methods are overrides. Something like mypy could then enforce its use. But an IDE could also just highlight method overrides differently, maybe PyCharm or PyDev already do that?) -- --Guido van Rossum (python.org/~guido ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Fri Nov 25 23:44:12 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Fri, 25 Nov 2016 23:44:12 -0500 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: Here's hopefully a working link; GitHub's rendering seems to be iffy: https://nbviewer.jupyter.org/urls/gist.githubusercontent.com/nicktimko/5f08d6adfa1dbe1319c3bfc715ec0aa4/raw/37dd95cd92be7b7e5af01b98ce9e8e00a705b3f7/override_guard.ipynb Knowing how to do some introspection with dunder magic attributes can make tracing methods easier. There's also a bunch of caveats that could probably bite you (and make my demo full of holes) On Fri, Nov 25, 2016 at 10:26 PM, Guido van Rossum wrote: > This idea is being kicked around from forum to forum and nobody wants to > have it. > > Here it's brought up from time to time and the response is usually "let a > linter do it". > > In mypy (which is essentially a powerful linter) it was proposed ( > https://github.com/python/mypy/issues/1888) and the response was > essentially "better go to python-ideas or bugs.python.org". > > It's also been proposed as a PEP 484 feature: https://github.com/python/ > typing/issues/269#issuecomment-243765549 . > > I think one reason why such proposals are unwelcome to experienced users > may be that when done right this is totally legitimate, and the requirement > to use an @override decorator is just making code more verbose with very > little benefit. (I could see a benefit -- if this is used consistently it > could make spelunking a large codebase easier, because you would know which > methods are overrides. Something like mypy could then enforce its use. But > an IDE could also just highlight method overrides differently, maybe > PyCharm or PyDev already do that?) > > -- > --Guido van Rossum (python.org/~guido ) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Fri Nov 25 23:50:41 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 26 Nov 2016 13:50:41 +0900 Subject: [Python-ideas] Python Documentation In-Reply-To: References: Message-ID: <22585.5281.689278.734505@turnbull.sk.tsukuba.ac.jp> I've redirected to core-mentorship where we'll clarify the proposed contribution and explain requirements for contribution to Python. Steve adil gourinda writes: > This is a small example of my idea, From wes.turner at gmail.com Sat Nov 26 01:00:38 2016 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 26 Nov 2016 00:00:38 -0600 Subject: [Python-ideas] Python Documentation In-Reply-To: <22585.5281.689278.734505@turnbull.sk.tsukuba.ac.jp> References: <22585.5281.689278.734505@turnbull.sk.tsukuba.ac.jp> Message-ID: - I can't open this .odt OpenOffice Document on my phone; or with GitHub. The CPython docs are here: - Src: https://hg.python.org/cpython - Src: https://hg.python.org/cpython/file/tip/Doc - Src: https://github.com/python/cpython - Src: https://github.com/python/cpython/tree/master/Doc The Python Devguide docs are here: - https://docs.python.org/devguide/#full-table-of-contents - https://docs.python.org/devguide/docquality.html - https://docs.python.org/devguide/documenting.html ODT -> RST - https://pypi.python.org/pypi/odt2sphinx/ - http://pandoc.org translates between formats like ODT and ReStructuredText On Friday, November 25, 2016, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > I've redirected to core-mentorship where we'll clarify the proposed > contribution and explain requirements for contribution to Python. > > Steve > > > adil gourinda writes: > > This is a small example of my idea, > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fleblanc50 at gmail.com Sat Nov 26 01:12:44 2016 From: fleblanc50 at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Leblanc?=) Date: Sat, 26 Nov 2016 07:12:44 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: Le 26 nov. 2016 4:27 AM, "Guido van Rossum" a ?crit : > > This idea is being kicked around from forum to forum and nobody wants to have it. Sorry, I want to have it but I dont want to pay the price in performance cost.. > Here it's brought up from time to time and the response is usually "let a linter do it". It's true, but in other hand because the subject come back from time to time, perhaps something better can be imagine the need is here. > In mypy (which is essentially a powerful linter) it was proposed ( https://github.com/python/mypy/issues/1888) and the response was essentially "better go to python-ideas or bugs.python.org". > > It's also been proposed as a PEP 484 feature: https://github.com/python/typing/issues/269#issuecomment-243765549 . > > I think one reason why such proposals are unwelcome to experienced users may be that when done right this is totally legitimate, and the requirement to use an @override decorator is just making code more verbose with very little benefit. (I could see a benefit -- if this is used consistently it could make spelunking a large codebase easier, because you would know which methods are overrides. Something like mypy could then enforce its use. But an IDE could also just highlight method overrides differently, maybe PyCharm or PyDev already do that?) The goal is not to make the code more verbose at all, it's to make code more sure in big or/and collaborative projects. The goal it's to give the possibility to secure the use of one of the more powerful feature of python. I think this kind of mistake may slow the time development and consequently afraid manager of using python in business environnement. Last time i have extend a Tkinter Top le vel and get in trouble with some comportement, I've just introduce a configuration function in my extended class. We often give same name for same functionnalities and that is the trouble if its not volontary. In case the function or variable is runtime defined its very hard to detect the error and pylint tools like will not help... > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 26 02:16:21 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 26 Nov 2016 17:16:21 +1000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On 26 November 2016 at 13:26, Guido van Rossum wrote: > I think one reason why such proposals are unwelcome to experienced users may > be that when done right this is totally legitimate, and the requirement to > use an @override decorator is just making code more verbose with very little > benefit. It's also the case that if we're going to provide decorators to help with class definitions, there are more valuable things that we can do beyond merely warning about accidental method and attribute overrides. For example, I've recently started using Hynek Schlawack's attrs project, which pairs up an "attr.attributes" class decorator with an "attr.attrib()" descriptor to make the following a class definition with totally ordered instances and a nice repr() implementation: @attributes class MyClass: attr1 = attrib() attr2 = attrib() ... It's *really* nice to use, and means I'm more inclined to define explicit helper classes for interim results rather than passing convenient-but-hard-to-debug raw tuples, lists and dicts around. The docs summarise the special method implementations that the decorator will inject for you in https://attrs.readthedocs.io/en/stable/why.html#hand-written-classes and go into more details on how it works at https://attrs.readthedocs.io/en/stable/how-does-it-work.html While the focus of attrs itself is very much on defining data structures where composition is the main form of re-use rather than inheritance, subclassing *is* supported, and such a system could also be used to complain about accidental attribute and method name collisions. Integrating a check for accidental overrides with a method-boilerplate reduction class decorator like the one in attrs would help address a few potential objections: - undecorated class definitions would be unaffected, so there'd be no start-up performance hit for existing code - leaving out the decorator when you intended to use it would also mean you wouldn't have an __init__ method defined, so you'd notice pretty fast if you forgot to use it - for the cost of some explicit decorators on the class definition and any method overrides, you get to skip writing a whole set of special methods (__new__/__init__, __repr__/etc, __eq__/__ne__, __lt__/etc) The specific names from the attrs project wouldn't be suitable for the standard library, but something like the following might work: @autoinit class MyClass: attr1 = instanceattr() attr2 = instanceattr() Where "instanceattr" is inspired by "classmethod" and "staticmethod", while "autoinit" refers directly to the main benefit of the class decorator (i.e. it writes your __init__ method for you based on the "instanceattr" definitions). The standardised variant of that could be kept very simple, while more advanced features like data validation and conversion were left to third party libraries like attrs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Nov 26 06:15:08 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Nov 2016 11:15:08 +0000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On 26 November 2016 at 07:16, Nick Coghlan wrote: > On 26 November 2016 at 13:26, Guido van Rossum wrote: >> I think one reason why such proposals are unwelcome to experienced users may >> be that when done right this is totally legitimate, and the requirement to >> use an @override decorator is just making code more verbose with very little >> benefit. > > It's also the case that if we're going to provide decorators to help > with class definitions, there are more valuable things that we can do > beyond merely warning about accidental method and attribute overrides. This comment made me think about another potential issue with the @override proposal. If overriding a method without using the decorator were to produce a warning, how would that interact with custom class decorators or metaclasses that inject methods into a class? Would they have to take special care to detect and mark overrides? While it's true that if a custom metaclass injects an unintended override, that's just as much a problem as if the user explicitly writes one, the problem arises for the end user of such a custom metaclass, who will get an obscure warning about internals that they probably don't know how to fix. Nothing insurmountable, sure, but it does act as a reminder that a change like this could have pretty wide reaching implications... Paul From ncoghlan at gmail.com Sat Nov 26 06:26:40 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 26 Nov 2016 21:26:40 +1000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On 26 November 2016 at 21:15, Paul Moore wrote: > On 26 November 2016 at 07:16, Nick Coghlan wrote: >> On 26 November 2016 at 13:26, Guido van Rossum wrote: >>> I think one reason why such proposals are unwelcome to experienced users may >>> be that when done right this is totally legitimate, and the requirement to >>> use an @override decorator is just making code more verbose with very little >>> benefit. >> >> It's also the case that if we're going to provide decorators to help >> with class definitions, there are more valuable things that we can do >> beyond merely warning about accidental method and attribute overrides. > > This comment made me think about another potential issue with the > @override proposal. If overriding a method without using the decorator > were to produce a warning, how would that interact with custom class > decorators or metaclasses that inject methods into a class? Would they > have to take special care to detect and mark overrides? It would potentially be even more annoying than that - if we did something like this without being sufficiently careful about it, you'd need to use the decorator any time you overwrote a special method that's actually implemented on "object". Ditto for method overrides when inheriting from an abstract base class, or any other base class that defines an API where the base class method implementation raises NotImplementedError. By contrast, if it's opt-in via a class decorator, then folks that want additional definition time assistance from the interpreter can request that explicitly, while existing code remains unaffected. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fleblanc50 at gmail.com Sat Nov 26 07:24:38 2016 From: fleblanc50 at gmail.com (France3) Date: Sat, 26 Nov 2016 13:24:38 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: <78fa6341-7e7e-b59a-848f-4fb6cb53b1a4@gmail.com> If we can sawp base class object with a flag It would be possible to use a special base object that could warn for this kind of problem? This can let write specifics metaclass witch replace base object by default, and when all is checked for end user don't use the flag... Does it is possible? What do you think about? Le 26/11/2016 ? 12:26, Nick Coghlan a ?crit : > On 26 November 2016 at 21:15, Paul Moore wrote: >> On 26 November 2016 at 07:16, Nick Coghlan wrote: >>> On 26 November 2016 at 13:26, Guido van Rossum wrote: >>>> I think one reason why such proposals are unwelcome to experienced users may >>>> be that when done right this is totally legitimate, and the requirement to >>>> use an @override decorator is just making code more verbose with very little >>>> benefit. >>> It's also the case that if we're going to provide decorators to help >>> with class definitions, there are more valuable things that we can do >>> beyond merely warning about accidental method and attribute overrides. >> This comment made me think about another potential issue with the >> @override proposal. If overriding a method without using the decorator >> were to produce a warning, how would that interact with custom class >> decorators or metaclasses that inject methods into a class? Would they >> have to take special care to detect and mark overrides? > It would potentially be even more annoying than that - if we did > something like this without being sufficiently careful about it, you'd > need to use the decorator any time you overwrote a special method > that's actually implemented on "object". Ditto for method overrides > when inheriting from an abstract base class, or any other base class > that defines an API where the base class method implementation raises > NotImplementedError. > > By contrast, if it's opt-in via a class decorator, then folks that > want additional definition time assistance from the interpreter can > request that explicitly, while existing code remains unaffected. > > Cheers, > Nick. > From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Nov 26 09:24:53 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 26 Nov 2016 23:24:53 +0900 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <20161126003257.GC3365@ando.pearwood.info> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: <22585.39733.715551.13439@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Sometimes I think Python-as-a-teaching-language and Python-as-a- > production-language are strongly opposed. I haven't found that to be the case. Linters are quite effective, as long as you discipline the students to use them. I don't think this check is in the linters I use, and if not I'd like to see it added, though. > I wonder whether there might be a case to be made for a > --with-training-wheels option? You could make it, but I can't imagine any argument that doesn't immediately fall prey to > But that's better placed in the IDE, not the core language. Maybe my imagination is just poor, but I think the burden of proof is on those who would add --with-training-wheels to the implementation. (N.B. This is an implementation issue, not something that would be in the language spec! :-) From ncoghlan at gmail.com Sat Nov 26 09:23:13 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 27 Nov 2016 00:23:13 +1000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <78fa6341-7e7e-b59a-848f-4fb6cb53b1a4@gmail.com> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> <78fa6341-7e7e-b59a-848f-4fb6cb53b1a4@gmail.com> Message-ID: On 26 November 2016 at 22:24, France3 wrote: > replace base object by default, and when all is checked for end user > don't use the flag... > > Does it is possible? What do you think about? There's no need to do this in a base class, since it can be done via external introspection. That introspection could live in at least a couple of different places: - a class decorator, which checks for unmarked overrides at class definition time - a test utility, that given a module name, imports the module, finds any defined classes, and checks them for unmarked overrides All an override marker would have to do to enable that introspection is to set a particular attribute on the method definition. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fleblanc50 at gmail.com Sat Nov 26 11:42:08 2016 From: fleblanc50 at gmail.com (=?UTF-8?Q?Fran=C3=A7ois_Leblanc?=) Date: Sat, 26 Nov 2016 17:42:08 +0100 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> <78fa6341-7e7e-b59a-848f-4fb6cb53b1a4@gmail.com> Message-ID: Le 26 nov. 2016 3:23 PM, "Nick Coghlan" a ?crit : > > On 26 November 2016 at 22:24, France3 wrote: > > replace base object by default, and when all is checked for end user > > don't use the flag... > > > > Does it is possible? What do you think about? > > There's no need to do this in a base class, since it can be done via > external introspection. That introspection could live in at least a > couple of different places: > > - a class decorator, which checks for unmarked overrides at class > definition time > - a test utility, that given a module name, imports the module, finds > any defined classes, and checks them for unmarked overrides > > All an override marker would have to do to enable that introspection > is to set a particular attribute on the method definition. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia I think that it's will be hard to take care of all situations this way... Moreover having a way to rewrite object base class open more perspectives from my point of view. -------------- next part -------------- An HTML attachment was scrubbed... URL: From celelibi at gmail.com Sat Nov 26 20:00:07 2016 From: celelibi at gmail.com (Celelibi) Date: Sun, 27 Nov 2016 02:00:07 +0100 Subject: [Python-ideas] Make threading.Event support test-and-set Message-ID: <20161127010002.GN6207@celelaptop.no-ip.org> Hello, I have a case of multithreaded program where under some condition, I would want to test the flag of the Event object and clear it atomically. This would allow me to know which thread has actually cleared the flag and thus should perform some more work before setting it again. The isSet method is of course not good enough because it's subject to race conditions. (Not even sure the GIL would make it safe.) The changes to threading.py would be pretty trivial: @@ -515,8 +515,10 @@ """ with self._cond: + prevflag = self._flag self._flag = True self._cond.notify_all() + return prevflag def clear(self): """Reset the internal flag to false. @@ -526,7 +528,9 @@ """ with self._cond: + prevflag = self._flag self._flag = False + return prevflag def wait(self, timeout=None): """Block until the internal flag is true. What do you think about it? Best regards, Celelibi From torsava at redhat.com Mon Nov 28 08:28:40 2016 From: torsava at redhat.com (Tomas Orsava) Date: Mon, 28 Nov 2016 14:28:40 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library Message-ID: Hi! We have written a draft PEP entitled "Distributing a Subset of the Standard Library" that aims to standardize and improve how Python handles omissions from its standard library. This is relevant both to Python itself as well as to Linux and other distributions that are packaging it, as they already separate out parts of the standard library into optionally installable packages. Ideas leading up to this PEP were discussed on the python-dev mailing list: https://mail.python.org/pipermail/python-dev/2016-July/145534.html Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html ------------------------------------------------------------------------ Abstract ======== Python is sometimes being distributed without its full standard library. However, there is as of yet no standardized way of dealing with importing a missing standard library module. This PEP proposes a mechanism for identifying which standard library modules are missing and puts forth a method of how attempts to import a missing standard library module should be handled. Motivation ========== There are several use cases for including only a subset of Python's standard library. However, there is so far no formal specification of how to properly implement distribution of a subset of the standard library. Namely, how to safely handle attempts to import a missing *stdlib* module, and display an informative error message. CPython ------- When one of Python standard library modules (such as ``_sqlite3``) cannot be compiled during a Python build because of missing dependencies (e.g. SQLite header files), the module is simply skipped. If you then install this compiled Python and use it to try to import one of the missing modules, Python will go through the ``sys.path`` entries looking for it. It won't find it among the *stdlib* modules and thus it will continue onto ``site-packages`` and fail with a ModuleNotFoundError_ if it doesn't find it. .. _ModuleNotFoundError: https://docs.python.org/3.7/library/exceptions.html#ModuleNotFoundError This can confuse users who may not understand why a cleanly built Python is missing standard library modules. Linux and other distributions ----------------------------- Many Linux and other distributions are already separating out parts of the standard library to standalone packages. Among the most commonly excluded modules are the ``tkinter`` module, since it draws in a dependency on the graphical environment, and the ``test`` package, as it only serves to test Python internally and is about as big as the rest of the standard library put together. The methods of omission of these modules differ. For example, Debian patches the file ``Lib/tkinter/__init__.py`` to envelop the line ``import _tkinter`` in a *try-except* block and upon encountering an ``ImportError`` it simply adds the following to the error message: ``please install the python3-tk package`` [#debian-patch]_. Fedora and other distributions simply don't include the omitted modules, potentially leaving users baffled as to where to find them. Specification ============= When, for any reason, a standard library module is not to be included with the rest, a file with its name and the extension ``.missing.py`` shall be created and placed in the directory the module itself would have occupied. This file can contain any Python code, however, it *should* raise a ModuleNotFoundError_ with a helpful error message. Currently, when Python tries to import a module ``XYZ``, the ``FileFinder`` path hook goes through the entries in ``sys.path``, and in each location looks for a file whose name is ``XYZ`` with one of the valid suffixes (e.g. ``.so``, ..., ``.py``, ..., ``.pyc``). The suffixes are tried in order. If none of them are found, Python goes on to try the next directory in ``sys.path``. The ``.missing.py`` extension will be added to the end of the list, and configured to be handled by ``SourceFileLoader``. Thus, if a module is not found in its proper location, the ``XYZ.missing.py`` file is found and executed, and further locations are not searched. The CPython build system will be modified to generate ``.missing.py`` files for optional modules that were not built. Rationale ========= The mechanism of handling missing standard library modules through the use of the ``.missing.py`` files was chosen due to its advantages both for CPython itself and for Linux and other distributions that are packaging it. The missing pieces of the standard library can be subsequently installed simply by putting the module files in their appropriate location. They will then take precedence over the corresponding ``.missing.py`` files. This makes installation simple for Linux package managers. This mechanism also solves the minor issue of importing a module from ``site-packages`` with the same name as a missing standard library module. Now, Python will import the ``.missing.py`` file and won't ever look for a *stdlib* module in ``site-packages``. In addition, this method of handling missing *stdlib* modules can be implemented in a succinct, non-intrusive way in CPython, and thus won't add to the complexity of the existing code base. The ``.missing.py`` file can be customized by the packager to provide any desirable behaviour. While we strongly recommend that these files only raise a ModuleNotFoundError_ with an appropriate message, there is no reason to limit customization options. Ideas leading up to this PEP were discussed on the `python-dev mailing list`_. .. _`python-dev mailing list`: https://mail.python.org/pipermail/python-dev/2016-July/145534.html Backwards Compatibility ======================= No problems with backwards compatibility are expected. Distributions that are already patching Python modules to provide custom handling of missing dependencies can continue to do so unhindered. Reference Implementation ======================== Reference implementation can be found on `GitHub`_ and is also accessible in the form of a `patch`_. .. _`GitHub`: https://github.com/torsava/cpython/pull/1 .. _`patch`: https://github.com/torsava/cpython/pull/1.patch References ========== .. [#debian-patch] http://bazaar.launchpad.net/~doko/python/pkg3.5-debian/view/head:/patches/tkinter-import.diff Copyright ========= This document has been placed in the public domain. ------------------------------------------------------------------------ Regards, Tomas Orsava -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 28 08:39:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Nov 2016 00:39:29 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On Tue, Nov 29, 2016 at 12:28 AM, Tomas Orsava wrote: > We have written a draft PEP entitled "Distributing a Subset of the Standard > Library" that aims to standardize and improve how Python handles omissions > from its standard library. This is relevant both to Python itself as well as > to Linux and other distributions that are packaging it, as they already > separate out parts of the standard library into optionally installable > packages. Thanks for writing this up! Since you're already working on GitHub, it's probably most straight-forward for you to create a PR against the PEPs repository: https://github.com/python/peps Looks like the next available PEP number is 534. ChrisA PEP editor From p.f.moore at gmail.com Mon Nov 28 09:32:16 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Nov 2016 14:32:16 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On 28 November 2016 at 13:28, Tomas Orsava wrote: > The ``.missing.py`` extension will be added to the end of the list, and > configured to be handled by ``SourceFileLoader``. Thus, if a module is not > found in its proper location, the ``XYZ.missing.py`` file is found and > executed, and further locations are not searched. Am I right to think that if a user had a file tkinter.missing.py in the current directory, then they'd get that in preference to the stdlib tkinter? Obviously this is no different from having a tkinter.py file in that directory, so it's not like this is a major problem, but it might be worth pointing out this minor incompatibility. Also, and possibly more of an issue, use of the ".missing.py" file will mean that a user can't provide their own implementation of the module later on sys.path. I don'rt know if this is a significant issue on Unix platforms. On Windows, there is a 3rd party implementation of the curses module which (as I understand it) can be user installed. If Python included a curses.missing.py, that would no longer work. Certainly these are only minor points, but worth considering. Paul From torsava at redhat.com Mon Nov 28 10:51:14 2016 From: torsava at redhat.com (Tomas Orsava) Date: Mon, 28 Nov 2016 16:51:14 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> On 11/28/2016 03:32 PM, Paul Moore wrote: > On 28 November 2016 at 13:28, Tomas Orsava wrote: >> The ``.missing.py`` extension will be added to the end of the list, and >> configured to be handled by ``SourceFileLoader``. Thus, if a module is not >> found in its proper location, the ``XYZ.missing.py`` file is found and >> executed, and further locations are not searched. > Am I right to think that if a user had a file tkinter.missing.py in > the current directory, then they'd get that in preference to the > stdlib tkinter? Obviously this is no different from having a > tkinter.py file in that directory, so it's not like this is a major > problem, but it might be worth pointing out this minor > incompatibility. Correct, both tkinter.py and tkinter.missing.py in the current directory will take precedence. I will note this in the backwards compatibility section. > Also, and possibly more of an issue, use of the ".missing.py" file > will mean that a user can't provide their own implementation of the > module later on sys.path. I don'rt know if this is a significant issue > on Unix platforms. On Windows, there is a 3rd party implementation of > the curses module which (as I understand it) can be user installed. If > Python included a curses.missing.py, that would no longer work. > > Certainly these are only minor points, but worth considering. I believe I may have found the Windows curses implementation, it's called PDCurses [0], and this website [1] appears to be distributing it under the name `curses`. Could some Windows user please check if compiling Python with the current reference implementation [2] of this PEP indeed generates a `curses.missing.py` file among the stdlib files? If so, we might consider skipping the generation of the .missing.py file for the curses module on Windows. [0] http://pdcurses.sourceforge.net/ [1] http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses [2] https://www.python.org/dev/peps/pep-0534/#reference-implementation Thank you for the feedback! From p.f.moore at gmail.com Mon Nov 28 11:13:04 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Nov 2016 16:13:04 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 28 November 2016 at 15:51, Tomas Orsava wrote: > I believe I may have found the Windows curses implementation, it's called > PDCurses [0], and this website [1] appears to be distributing it under the > name `curses`. My apologies, I should have included a pointer. That is indeed the distribution I was thinking of. > Could some Windows user please check if compiling Python with the current > reference implementation [2] of this PEP indeed generates a > `curses.missing.py` file among the stdlib files? If so, we might consider > skipping the generation of the .missing.py file for the curses module on > Windows. I'll see if I can make some time to do the test. But as the change is to setup.py, and the Windows build uses Visual Studio project files to do the build, I expect that it won't generate missing.py files on Windows. In actual fact, that may be the simplest solution, to note that the build part of this change is restricted to Unix (non-Windows) platforms specifically. As there's no real concept of a "distribution version" of Python on Windows, it's probably not something that will be that important on that platform (and support for .missing.py files is there, it would just be necessary for distributors to manually create those files as needed). Paul From random832 at fastmail.com Mon Nov 28 11:35:03 2016 From: random832 at fastmail.com (Random832) Date: Mon, 28 Nov 2016 11:35:03 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> On Mon, Nov 28, 2016, at 10:51, Tomas Orsava wrote: > Could some Windows user please check if compiling Python with the > current reference implementation [2] of this PEP indeed generates a > `curses.missing.py` file among the stdlib files? If so, we might > consider skipping the generation of the .missing.py file for the curses > module on Windows. "Skip it for curses on Windows" doesn't seem like an acceptable solution, because tomorrow there could be another module, on another platform, that needs a similar fix. I think it'd be better to fix the logic. Searching the whole path for whatever.py before searching for whatever.missing.py makes sense to me and I'm not sure why this isn't the proposal. Honestly, though, I'm not sure of the need for the PEP in general. "However, there is as of yet no standardized way of dealing with importing a missing standard library module." is simply not true. The standardized way of dealing with it is that the import statement will raise an ImportError exception. Why exactly is that not good enough? A distribution could, for example, include an excepthook in site.py that prints an informative error message when an ImportError is unhandled for a list of modules that it knows about. Or they could modify the *default* excepthook in the interpreter itself. From guido at python.org Mon Nov 28 11:38:05 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 08:38:05 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: Overall I think this is a good idea. I have one hit: It seems that there are two possible strategies for searching the .missing.py file: 1. (Currently in the PEP) search it at the same time as the .py file when walking along sys.path. - Pro: prevents confusion when the user accidentally has their own matching file later in sys.path. - Con: prevents the user from installing a matching file intentionally (e.g. a 3rd party version). 2. After exhausting sys.path, search it again just for .missing.py files (or perhaps remember the location of the .missing.py file during the first search but don't act immediately on it -- this has the same effect). - Pro: allows user to install their own version. - Con: if the user has a matching file by accident, that file will be imported, causing more confusion. I personally would weigh these so as to prefer (2). The option of installing your own version when the standard version doesn't exist seems reasonable; there may be reasons that you can't or don't want to install the distribution's version. I don't worry much about the danger of accidental name conflicts (have you ever seen this?). --Guido On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore wrote: > On 28 November 2016 at 15:51, Tomas Orsava wrote: > > I believe I may have found the Windows curses implementation, it's called > > PDCurses [0], and this website [1] appears to be distributing it under > the > > name `curses`. > > My apologies, I should have included a pointer. That is indeed the > distribution I was thinking of. > > > Could some Windows user please check if compiling Python with the current > > reference implementation [2] of this PEP indeed generates a > > `curses.missing.py` file among the stdlib files? If so, we might > consider > > skipping the generation of the .missing.py file for the curses module on > > Windows. > > I'll see if I can make some time to do the test. But as the change is > to setup.py, and the Windows build uses Visual Studio project files to > do the build, I expect that it won't generate missing.py files on > Windows. In actual fact, that may be the simplest solution, to note > that the build part of this change is restricted to Unix (non-Windows) > platforms specifically. As there's no real concept of a "distribution > version" of Python on Windows, it's probably not something that will > be that important on that platform (and support for .missing.py files > is there, it would just be necessary for distributors to manually > create those files as needed). > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abrault at mapgears.com Mon Nov 28 12:04:27 2016 From: abrault at mapgears.com (Alexandre Brault) Date: Mon, 28 Nov 2016 12:04:27 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: I would also prefer (2) for exactly the example given in this thread. The Windows version of curses.missing.py could raise a ModuleNotFoundError saying that curses is not available on Windows, but a developer who wants to can install PDCurses to implement the stdlib module. I don't think the few cases where an stdlib package is both missing and has a name collision with an incompatible module are enough to outweigh the benefits of being able to install a third-party package to implement a missing part of the stdlib Alex On 2016-11-28 11:38 AM, Guido van Rossum wrote: > Overall I think this is a good idea. I have one hit: > > It seems that there are two possible strategies for searching the > .missing.py file: > > 1. (Currently in the PEP) search it at the same time as the .py file > when walking along sys.path. > - Pro: prevents confusion when the user accidentally has their own > matching file later in sys.path. > - Con: prevents the user from installing a matching file > intentionally (e.g. a 3rd party version). > > 2. After exhausting sys.path, search it again just for .missing.py > files (or perhaps remember the location of the .missing.py file during > the first search but don't act immediately on it -- this has the same > effect). > - Pro: allows user to install their own version. > - Con: if the user has a matching file by accident, that file will > be imported, causing more confusion. > > I personally would weigh these so as to prefer (2). The option of > installing your own version when the standard version doesn't exist > seems reasonable; there may be reasons that you can't or don't want to > install the distribution's version. I don't worry much about the > danger of accidental name conflicts (have you ever seen this?). > > > --Guido > > On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore > wrote: > > On 28 November 2016 at 15:51, Tomas Orsava > wrote: > > I believe I may have found the Windows curses implementation, > it's called > > PDCurses [0], and this website [1] appears to be distributing it > under the > > name `curses`. > > My apologies, I should have included a pointer. That is indeed the > distribution I was thinking of. > > > Could some Windows user please check if compiling Python with > the current > > reference implementation [2] of this PEP indeed generates a > > `curses.missing.py ` file among the > stdlib files? If so, we might consider > > skipping the generation of the .missing.py file for the curses > module on > > Windows. > > I'll see if I can make some time to do the test. But as the change is > to setup.py, and the Windows build uses Visual Studio project files to > do the build, I expect that it won't generate missing.py files on > Windows. In actual fact, that may be the simplest solution, to note > that the build part of this change is restricted to Unix (non-Windows) > platforms specifically. As there's no real concept of a "distribution > version" of Python on Windows, it's probably not something that will > be that important on that platform (and support for .missing.py files > is there, it would just be necessary for distributors to manually > create those files as needed). > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Nov 28 12:14:43 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 28 Nov 2016 09:14:43 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On Nov 28, 2016 8:38 AM, "Guido van Rossum" wrote: > > Overall I think this is a good idea. I have one hit: > > It seems that there are two possible strategies for searching the .missing.py file: > > 1. (Currently in the PEP) search it at the same time as the .py file when walking along sys.path. > - Pro: prevents confusion when the user accidentally has their own matching file later in sys.path. > - Con: prevents the user from installing a matching file intentionally (e.g. a 3rd party version). > > 2. After exhausting sys.path, search it again just for .missing.py files (or perhaps remember the location of the .missing.py file during the first search but don't act immediately on it -- this has the same effect). > - Pro: allows user to install their own version. > - Con: if the user has a matching file by accident, that file will be imported, causing more confusion. > > I personally would weigh these so as to prefer (2). The option of installing your own version when the standard version doesn't exist seems reasonable; there may be reasons that you can't or don't want to install the distribution's version. I don't worry much about the danger of accidental name conflicts (have you ever seen this?). I was going to make a similar comment, because it seems to me that it could make sense for a redistributor to want to move some bits of the stdlib into wheels, and this is most conveniently handled by letting bits of the stdlib live in site-packages. Also note that in Guido's option 2, we only incur the extra fstat calls if the import would otherwise fail. In option 1, there are extra fstat calls (and thus disk seeks etc.) adding some overhead to every import. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 28 12:28:55 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 09:28:55 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On Mon, Nov 28, 2016 at 9:14 AM, Nathaniel Smith wrote: > Also note that in Guido's option 2, we only incur the extra fstat calls if > the import would otherwise fail. In option 1, there are extra fstat calls > (and thus disk seeks etc.) adding some overhead to every import. Oh, that's an important consideration! Yes, adding .missing.py to the list of extensions would cause extra stat() calls and potentially slow down every import. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 28 13:11:12 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Nov 2016 10:11:12 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: Am I missing something? Given Python's dynamic nature, there is simply no way to know if a method is overriding a superclass' method until it is called -- and, now that I think about it even then you don't know. At compile time, none of the superclasses may have the given method. At run time, a method could be monkey patched into the superclass before the subclass' instance is created. In fact, at run time, the superclass could get the method added after instances are created. At calling time, the subclass' method will be found, and used, and the search stops there -- no way to know if there is one with the same name further up the MRO. This is simply incompatable with a language this dynamic. On Fri, Nov 25, 2016 at 7:26 PM, Guido van Rossum wrote: > Here it's brought up from time to time and the response is usually "let a > linter do it". > Exactly -- it can't be enforced, but maybe it's nice for a linter to give some information at develop time. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 28 13:22:43 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 10:22:43 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 10:11 AM, Chris Barker wrote: > Am I missing something? > > Given Python's dynamic nature, there is simply no way to know if a method > is overriding a superclass' method until it is called -- and, now that I > think about it even then you don't know. > > At compile time, none of the superclasses may have the given method. > > At run time, a method could be monkey patched into the superclass before > the subclass' instance is created. > > In fact, at run time, the superclass could get the method added after > instances are created. > > At calling time, the subclass' method will be found, and used, and the > search stops there -- no way to know if there is one with the same name > further up the MRO. > > This is simply incompatable with a language this dynamic. > Not so fast! Python is also so dynamic that you can easily create a metaclass (or a class decorator) that does the checking (assuming reasonable behavior of all classes involved) at class definition time. Given that the use of such a metaclass will be voluntary, for classes with unreasonable behavior it should simply not be applied. I'm not sure yet if it is desirable to have such a metaclass or class decorator in the stdlib, nor have I thought through its design. Such a design should first be deployed via PyPI and then we can judge its effectiveness based on how popular it becomes. (There may already be one, I've not done that research either.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Nov 28 14:54:48 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 28 Nov 2016 14:54:48 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 11/28/2016 11:38 AM, Guido van Rossum wrote: > Overall I think this is a good idea. I have one hit: > > It seems that there are two possible strategies for searching the > .missing.py file: > > 1. (Currently in the PEP) search it at the same time as the .py file > when walking along sys.path. > - Pro: prevents confusion when the user accidentally has their own > matching file later in sys.path. > - Con: prevents the user from installing a matching file intentionally > (e.g. a 3rd party version). > > 2. After exhausting sys.path, search it again just for .missing.py files > (or perhaps remember the location of the .missing.py file during the > first search but don't act immediately on it -- this has the same effect). > - Pro: allows user to install their own version. The Windows distribution. for instance, could have a mod.missing.py file for every non-Windows, unix-only module. This would cut down on 'Why did this import fail?' questions on SO and python-list. And without breaking anything. Even for non-beginners, it would save having to look up whether the an import failure is inherent on the platform or due to a typo. > - Con: if the user has a matching file by accident, that file will be > imported, causing more confusion. > > I personally would weigh these so as to prefer (2). The option of > installing your own version when the standard version doesn't exist > seems reasonable; there may be reasons that you can't or don't want to > install the distribution's version. I don't worry much about the danger > of accidental name conflicts (have you ever seen this?). The accidental conflict reports I have seen were due to scripts in as in the same directory as the program, rather than modules in site-packages. -- Terry Jan Reedy From guido at python.org Mon Nov 28 15:00:55 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 12:00:55 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On Mon, Nov 28, 2016 at 11:54 AM, Terry Reedy wrote: > On 11/28/2016 11:38 AM, Guido van Rossum wrote: > >> Overall I think this is a good idea. I have one hit: >> >> It seems that there are two possible strategies for searching the >> .missing.py file: >> >> 1. (Currently in the PEP) search it at the same time as the .py file >> when walking along sys.path. >> - Pro: prevents confusion when the user accidentally has their own >> matching file later in sys.path. >> - Con: prevents the user from installing a matching file intentionally >> (e.g. a 3rd party version). >> >> 2. After exhausting sys.path, search it again just for .missing.py files >> (or perhaps remember the location of the .missing.py file during the >> first search but don't act immediately on it -- this has the same effect). >> - Pro: allows user to install their own version. >> > > The Windows distribution. for instance, could have a mod.missing.py file > for every non-Windows, unix-only module. This would cut down on 'Why did > this import fail?' questions on SO and python-list. And without breaking > anything. Even for non-beginners, it would save having to look up whether > the an import failure is inherent on the platform or due to a typo. > Yes, this is why I like the proposal (but it would still have this benefit with option (2)). > - Con: if the user has a matching file by accident, that file will be >> imported, causing more confusion. >> >> I personally would weigh these so as to prefer (2). The option of >> installing your own version when the standard version doesn't exist >> seems reasonable; there may be reasons that you can't or don't want to >> install the distribution's version. I don't worry much about the danger >> of accidental name conflicts (have you ever seen this?). >> > > The accidental conflict reports I have seen were due to scripts in as in > the same directory as the program, rather than modules in site-packages. Yes we get those occasionally -- but do those ever happen specifically for stdlib module names that aren't installed because the platform doesn't support them or requires them to be installed separately? I.e. would the .missing.py feature actually have benefitted those reports? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Nov 28 15:02:20 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 12:02:20 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: <583C8D4C.6020904@stoneleaf.us> On 11/28/2016 08:38 AM, Guido van Rossum wrote: > 2. After exhausting sys.path, search it again just for .missing.py files > (or perhaps remember the location of the .missing.py file during the > first search but don't act immediately on it -- this has the same effect). > - Pro: allows user to install their own version. > - Con: if the user has a matching file by accident, that file will be > imported, causing more confusion. > > I personally would weigh these so as to prefer (2). The option of installing > your own version when the standard version doesn't exist seems reasonable; > there may be reasons that you can't or don't want to install the distribution's > version. I also agree that (2) is the better option. General Python philosophy seems to be to not prohibit actions unless there is a *really* good reason (i.e. sum()ing strings), and option (1) would require installing third-party modules into the stdlib to workaround the prohibition. -- ~Ethan~ From ethan at stoneleaf.us Mon Nov 28 15:05:01 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 12:05:01 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> Message-ID: <583C8DED.9020709@stoneleaf.us> On 11/28/2016 08:35 AM, Random832 wrote: > On Mon, Nov 28, 2016, at 10:51, Tomas Orsava wrote: >> Could some Windows user please check if compiling Python with the >> current reference implementation [2] of this PEP indeed generates a >> `curses.missing.py` file among the stdlib files? If so, we might >> consider skipping the generation of the .missing.py file for the curses >> module on Windows. > > "Skip it for curses on Windows" doesn't seem like an acceptable > solution, because tomorrow there could be another module, on another > platform, that needs a similar fix. I think it'd be better to fix the > logic. Searching the whole path for whatever.py before searching for > whatever.missing.py makes sense to me [...] Agreed. > Honestly, though, I'm not sure of the need for the PEP in general. > "However, there is as of yet no standardized way of dealing with > importing a missing standard library module." is simply not true. The > standardized way of dealing with it is that the import statement will > raise an ImportError exception. Why exactly is that not good enough? Because it is unfriendly. Helpful error messages are a great tool to both beginner and seasoned programmers. > A distribution could, for example, include an excepthook in site.py that > prints an informative error message when an ImportError is unhandled for > a list of modules that it knows about. [...] As you say above, that list will fall out of date. Better to have a standard method that is easily implemented. -- ~Ethan~ From ethan at stoneleaf.us Mon Nov 28 15:51:29 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 12:51:29 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <583C98D1.6080600@stoneleaf.us> On 11/28/2016 05:28 AM, Tomas Orsava wrote: > Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html Overall +1, but using Guido's #2 option instead for handling *.missing.py files (searching all possible locations for the module before falling back to the stdlib xxx.missing.py default). -- ~Ethan~ From guido at python.org Mon Nov 28 16:01:00 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 13:01:00 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <583C98D1.6080600@stoneleaf.us> References: <583C98D1.6080600@stoneleaf.us> Message-ID: On Mon, Nov 28, 2016 at 12:51 PM, Ethan Furman wrote: > On 11/28/2016 05:28 AM, Tomas Orsava wrote: > > Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html >> > > Overall +1, but using Guido's #2 option instead for handling *.missing.py > files (searching all possible locations for the module before falling back > to the stdlib xxx.missing.py default). > Actually the .missing.py feature would be useful for other use cases, so it shouldn't be limited to the stdlib part of sys.path. (Also I'm withdrawing my idea of searching for it while searching for the original .py since that would burden successful imports with extra stat() calls.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Nov 28 16:11:01 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 13:11:01 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <583C98D1.6080600@stoneleaf.us> Message-ID: <583C9D65.5070600@stoneleaf.us> On 11/28/2016 01:01 PM, Guido van Rossum wrote: > On Mon, Nov 28, 2016 at 12:51 PM, Ethan Furman wrote: >> On 11/28/2016 05:28 AM, Tomas Orsava wrote: >>> Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html >> >> >> Overall +1, but using Guido's #2 option instead for handling >> *.missing.py files (searching all possible locations for the >> module before falling back to the stdlib xxx.missing.py default). > > Actually the .missing.py feature would be useful for other use > cases, so it shouldn't be limited to the stdlib part of sys.path. > (Also I'm withdrawing my idea of searching for it while searching > for the original .py since that would burden successful imports > with extra stat() calls.) Absolutely. The key point in your counter proposal is not failing at the first .missing.py file possible, but rather searching all possible locations first. If we do the full search for the import first, then a full search for the .missing.py, and that ends up not hurting performance at all for successful imports -- well, that's just icing on the cake. :) One "successful" use-case that would be impacted is the fallback import idiom: try: # this would do two full searches before getting the error import BlahBlah except ImportError: import blahblah -- ~Ethan~ From p.f.moore at gmail.com Mon Nov 28 16:26:58 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Nov 2016 21:26:58 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <583C9D65.5070600@stoneleaf.us> References: <583C98D1.6080600@stoneleaf.us> <583C9D65.5070600@stoneleaf.us> Message-ID: On 28 November 2016 at 21:11, Ethan Furman wrote: > One "successful" use-case that would be impacted is the fallback import > idiom: > > try: > # this would do two full searches before getting the error > import BlahBlah > except ImportError: > import blahblah Under this proposal, the above idiom could potentially now fail. If there's a BlahBlah.missing.py, then that will get executed rather than an ImportError being raised, so the fallback wouldn't be executed. This could actually be a serious issue for code that currently protects against optional stdlib modules not being available like this. There's no guarantee that I can see that a .missing.py file would raise ImportError (even if we said that was the intended behaviour, there's nothing to enforce it). Could the proposal execute the .missing.py file and then raise ImportError? I could imagine that having problems of its own, though... Paul From wolfgang.maier at biologie.uni-freiburg.de Mon Nov 28 16:33:08 2016 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 28 Nov 2016 22:33:08 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <583C98D1.6080600@stoneleaf.us> <583C9D65.5070600@stoneleaf.us> Message-ID: On 28.11.2016 22:26, Paul Moore wrote: > On 28 November 2016 at 21:11, Ethan Furman wrote: >> One "successful" use-case that would be impacted is the fallback import >> idiom: >> >> try: >> # this would do two full searches before getting the error >> import BlahBlah >> except ImportError: >> import blahblah > > Under this proposal, the above idiom could potentially now fail. If > there's a BlahBlah.missing.py, then that will get executed rather than > an ImportError being raised, so the fallback wouldn't be executed. > This could actually be a serious issue for code that currently > protects against optional stdlib modules not being available like > this. There's no guarantee that I can see that a .missing.py file > would raise ImportError (even if we said that was the intended > behaviour, there's nothing to enforce it). > > Could the proposal execute the .missing.py file and then raise > ImportError? I could imagine that having problems of its own, > though... > How about addressing both concerns by triggering the search for .missing.py only if an ImportError bubbles up uncaught (a bit similar to StopIteration nowadays)? Wolfgang From chris.barker at noaa.gov Mon Nov 28 16:32:52 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Nov 2016 13:32:52 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 10:22 AM, Guido van Rossum wrote: > At calling time, the subclass' method will be found, and used, and the >> search stops there -- no way to know if there is one with the same name >> further up the MRO. >> >> This is simply incompatable with a language this dynamic. >> > > Not so fast! Python is also so dynamic that you can easily create a > metaclass (or a class decorator) that does the checking (assuming > reasonable behavior of all classes involved) at class definition time. > but that's class definition time -- can't classes be dynamically mangled later on -- after subclasses have been defined? If I have this right the mro is set at class definition time, so it is knows which classes are in the tree -- but class objects themselves are mutable -- methods can be added later on. so a subclass could have a method that isn't overriding anything when it's defined, but ends up overriding something later on, 'cause the base class changed under it. Isn't this why the mro is searched at method calling time? rather than a static table being used? (if not -- wouldnt that be a nice optimization?) Granted -- this kind of late class mangling has got to be pretty unusual (and maybe only useful for mocking, or??) but it is there. Which is why it could be useful to have some syntax or convention for specifying "I'm intending to override a method", that linters or type checkers, or whatever external tool, can use. But it can't be enforced at runtime in the language. much like type annotations... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 28 16:37:09 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 13:37:09 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 1:32 PM, Chris Barker wrote: > On Mon, Nov 28, 2016 at 10:22 AM, Guido van Rossum > wrote: > > >> At calling time, the subclass' method will be found, and used, and the >>> search stops there -- no way to know if there is one with the same name >>> further up the MRO. >>> >>> This is simply incompatable with a language this dynamic. >>> >> >> Not so fast! Python is also so dynamic that you can easily create a >> metaclass (or a class decorator) that does the checking (assuming >> reasonable behavior of all classes involved) at class definition time. >> > > but that's class definition time -- can't classes be dynamically mangled > later on -- after subclasses have been defined? > They can, and they @override can be bypassed. I don't see that as a condemnation of @overload -- it just means that it's not perfect, which is fine with me (given that we're talking about monkey-patching here). > If I have this right the mro is set at class definition time, so it is > knows which classes are in the tree -- but class objects themselves are > mutable -- methods can be added later on. so a subclass could have a > method that isn't overriding anything when it's defined, but ends up > overriding something later on, 'cause the base class changed under it. > > Isn't this why the mro is searched at method calling time? rather than a > static table being used? (if not -- wouldnt that be a nice optimization?) > > Granted -- this kind of late class mangling has got to be pretty unusual > (and maybe only useful for mocking, or??) but it is there. > > Which is why it could be useful to have some syntax or convention for > specifying "I'm intending to override a method", that linters or type > checkers, or whatever external tool, can use. But it can't be enforced at > runtime in the language. > Depends on how much enforcement you need. Bike locks also don't typically enforce that your bike doesn't get stolen... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 28 16:39:09 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Nov 2016 13:39:09 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <583C98D1.6080600@stoneleaf.us> <583C9D65.5070600@stoneleaf.us> Message-ID: On Mon, Nov 28, 2016 at 1:26 PM, Paul Moore wrote: > > One "successful" use-case that would be impacted is the fallback import > > idiom: > > > > try: > > # this would do two full searches before getting the error > > import BlahBlah > > except ImportError: > > import blahblah > > Under this proposal, the above idiom could potentially now fail. higher on the thread, someone said that IMportError was not robust enough, because it didn't give near,y as meaninful an error message as it might. > There's no guarantee that I can see that a .missing.py file > would raise ImportError (even if we said that was the intended > behaviour, there's nothing to enforce it). > there is nothing to enforce all sorts of things -- I dont hinkt it's so wrong to have it in the spec that .missing.py fles NEED to raise an ImportError and they could give nice meaningful error messages that way without breaking old code. Could the proposal execute the .missing.py file and then raise > ImportError? I could imagine that having problems of its own, > though... if the ImportError is raised by the surrounding code, then it would need a protocol to get the nice error message -- raising an Exception is already a protocol -- let's use that one. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 28 16:44:39 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Nov 2016 13:44:39 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 1:37 PM, Guido van Rossum wrote: > They can, and they @override can be bypassed. I don't see that as a > condemnation of @overload -- it just means that it's not perfect, which is > fine with me (given that we're talking about monkey-patching here). > sure -- but this all strikes me as kind of like type checking -- there is a lot of use for robust code, but we don't want it at run-time in the language. Also -- the ship has kinda sailed on this - maybe a @not_override would make more sense. Isn't the goal to make sure you don't accidentally override a method? saying "I know I'm overriding this" is less useful than "I'm not intending to override anything here" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Nov 28 16:48:48 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 13:48:48 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <583C98D1.6080600@stoneleaf.us> <583C9D65.5070600@stoneleaf.us> Message-ID: <583CA640.4090202@stoneleaf.us> On 11/28/2016 01:26 PM, Paul Moore wrote: > On 28 November 2016 at 21:11, Ethan Furman wrote: >> One "successful" use-case that would be impacted is the fallback import >> idiom: >> >> try: >> # this would do two full searches before getting the error >> import BlahBlah >> except ImportError: >> import blahblah > > Under this proposal, the above idiom could potentially now fail. If > there's a BlahBlah.missing.py, then that will get executed rather than > an ImportError being raised, so the fallback wouldn't be executed. Which is why the strong recommendation is for the .missing.py file to raise an ImportError exception, but with a useful error message, such as "Tkinter is not currently installed. Install python-tkinter to get it." > This could actually be a serious issue for code that currently > protects against optional stdlib modules not being available like > this. There's no guarantee that I can see that a .missing.py file > would raise ImportError (even if we said that was the intended > behaviour, there's nothing to enforce it). Presumably the folks doing the splitting know what they are doing. Any cPython auto-generated .missing.py files would be correct: raise ImportError("tkinter was not compiled due to ...") . > Could the proposal execute the .missing.py file and then raise > ImportError? I could imagine that having problems of its own, > though... Yeah, I don't think that's a good idea. -- ~Ethan~ From guido at python.org Mon Nov 28 16:50:19 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 13:50:19 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 1:44 PM, Chris Barker wrote: > > On Mon, Nov 28, 2016 at 1:37 PM, Guido van Rossum > wrote: > > >> They can, and they @override can be bypassed. I don't see that as a >> condemnation of @overload -- it just means that it's not perfect, which is >> fine with me (given that we're talking about monkey-patching here). >> > > sure -- but this all strikes me as kind of like type checking -- there is > a lot of use for robust code, but we don't want it at run-time in the > language. > > Also -- the ship has kinda sailed on this - maybe a @not_override would > make more sense. > > Isn't the goal to make sure you don't accidentally override a method? > saying "I know I'm overriding this" is less useful than "I'm not intending > to override anything here" > I think you're fighting a straw man. I never said @override should be added to the language. I said that it would be useful to have a 3rd party metaclass or a class decorator that implements it which packages may voluntarily use to constrain their subclasses (or their own uses -- different designs are possible). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 28 17:01:05 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Nov 2016 14:01:05 -0800 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On Mon, Nov 28, 2016 at 1:50 PM, Guido van Rossum wrote: > Also -- the ship has kinda sailed on this - maybe a @not_override would >> make more sense. >> >> Isn't the goal to make sure you don't accidentally override a method? >> saying "I know I'm overriding this" is less useful than "I'm not intending >> to override anything here" >> > > I think you're fighting a straw man. I never said @override should be > added to the language. I said that it would be useful to have a 3rd party > metaclass or a class decorator that implements it which packages may > voluntarily use to constrain their subclasses (or their own uses -- > different designs are possible). > I know -- I just happened to add that to a reply to you... That was for the OP, or anyone else thinking of writing such a thing. And I still think it would better be added to a linting tool than at run-time -- but let whoever writes it figure that out. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Nov 28 17:19:50 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 28 Nov 2016 14:19:50 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On Mon, Nov 28, 2016 at 5:28 AM, Tomas Orsava wrote: [...] > Specification > ============= > > When, for any reason, a standard library module is not to be included with > the > rest, a file with its name and the extension ``.missing.py`` shall be > created > and placed in the directory the module itself would have occupied. This > file > can contain any Python code, however, it *should* raise a > ModuleNotFoundError_ > with a helpful error message. > > Currently, when Python tries to import a module ``XYZ``, the ``FileFinder`` > path hook goes through the entries in ``sys.path``, and in each location > looks > for a file whose name is ``XYZ`` with one of the valid suffixes (e.g. > ``.so``, > ..., ``.py``, ..., ``.pyc``). The suffixes are tried in order. If none of > them are found, Python goes on to try the next directory in ``sys.path``. > > The ``.missing.py`` extension will be added to the end of the list, and > configured to be handled by ``SourceFileLoader``. Thus, if a module is not > found in its proper location, the ``XYZ.missing.py`` file is found and > executed, and further locations are not searched. I'd suggest that we additional specify that if we find a foo.missing.py, then the code is executed but -- unlike a regular module load -- it's not automatically inserted into sys.modules["foo"]. That seems like it could only create confusion. And it doesn't restrict functionality, because if someone really wants to implement some clever shenanigans, they can always modify sys.modules["foo"] by hand. This also suggests that the overall error-handling flow for 'import foo' should look like: 1) run foo.missing.py 2) if it raises an exception: propagate that 3) otherwise, if sys.modules["foo"] is missing: raise some variety of ImportError. 4) otherwise, use sys.modules["foo"] as the object that should be bound to 'foo' in the original invoker's namespace I think this might make everyone who was worried about exception handling downthread happy -- it allows a .missing.py file to successfully import if it really wants to, but only if it explicitly fulfills 'import' requirement that the module should somehow be made available. -n -- Nathaniel J. Smith -- https://vorpus.org From steve.dower at python.org Mon Nov 28 17:33:31 2016 From: steve.dower at python.org (Steve Dower) Date: Mon, 28 Nov 2016 14:33:31 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> On 28Nov2016 1419, Nathaniel Smith wrote: > I'd suggest that we additional specify that if we find a > foo.missing.py, then the code is executed but -- unlike a regular > module load -- it's not automatically inserted into > sys.modules["foo"]. That seems like it could only create confusion. > And it doesn't restrict functionality, because if someone really wants > to implement some clever shenanigans, they can always modify > sys.modules["foo"] by hand. In before Brett says "you can do this with an import hook", because, well, we can do this with an import hook :) Given that, this wouldn't necessarily need to be an executable file. The finder could locate a "foo.missing" file and raise ModuleNotFoundError with the contents of the file as the message. No need to allow/require any Python code at all, and no risk of polluting sys.modules. Cheers, Steve From wolfgang.maier at biologie.uni-freiburg.de Mon Nov 28 17:42:41 2016 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 28 Nov 2016 23:42:41 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On 28.11.2016 23:19, Nathaniel Smith wrote: > > I'd suggest that we additional specify that if we find a > foo.missing.py, then the code is executed but -- unlike a regular > module load -- it's not automatically inserted into > sys.modules["foo"]. That seems like it could only create confusion. > And it doesn't restrict functionality, because if someone really wants > to implement some clever shenanigans, they can always modify > sys.modules["foo"] by hand. > > This also suggests that the overall error-handling flow for 'import > foo' should look like: > > 1) run foo.missing.py > 2) if it raises an exception: propagate that > 3) otherwise, if sys.modules["foo"] is missing: raise some variety of > ImportError. > 4) otherwise, use sys.modules["foo"] as the object that should be > bound to 'foo' in the original invoker's namespace > > I think this might make everyone who was worried about exception > handling downthread happy -- it allows a .missing.py file to > successfully import if it really wants to, but only if it explicitly > fulfills 'import' requirement that the module should somehow be made > available. > A refined (from my previous post which may have ended up too nested) alternative: instead of triggering an immediate search for a .missing.py file, why not have the interpreter intercept any ModuleNotFoundError that bubbles up to the top without being caught, then uses the name attribute of the exception to look for the .missing.py file. Agreed, this is more complicated to implement, but it would avoid any performance loss in situations where running code knows how to deal with the missing module anyway. Wolfgang From ethan at stoneleaf.us Mon Nov 28 17:47:57 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 14:47:57 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <583CB41D.3060303@stoneleaf.us> On 11/28/2016 02:42 PM, Wolfgang Maier wrote: > A refined (from my previous post which may have ended up too nested) > alternative: instead of triggering an immediate search for a .missing.py > file, why not have the interpreter intercept any ModuleNotFoundError > that bubbles up to the top without being caught, then uses the name > attribute of the exception to look for the .missing.py file. Agreed, > this is more complicated to implement, but it would avoid any perfor- >mance loss in situations where running code knows how to deal with the > missing module anyway. So we only have the hit when the exception is going to kill the interpreter? +1 -- ~Ethan~ From steve.dower at python.org Mon Nov 28 17:48:47 2016 From: steve.dower at python.org (Steve Dower) Date: Mon, 28 Nov 2016 14:48:47 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> Message-ID: <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> On 28Nov2016 1433, Steve Dower wrote: > On 28Nov2016 1419, Nathaniel Smith wrote: >> I'd suggest that we additional specify that if we find a >> foo.missing.py, then the code is executed but -- unlike a regular >> module load -- it's not automatically inserted into >> sys.modules["foo"]. That seems like it could only create confusion. >> And it doesn't restrict functionality, because if someone really wants >> to implement some clever shenanigans, they can always modify >> sys.modules["foo"] by hand. > > In before Brett says "you can do this with an import hook", because, > well, we can do this with an import hook :) And since I suggested it, here's a rough proof-of-concept: import importlib.abc import os import sys class MissingPathFinder(importlib.abc.MetaPathFinder): def find_spec(self, fullname, path, target=None): for p in (path or sys.path): file = os.path.join(p, fullname + ".missing") if os.path.isfile(file): with open(file, 'r', encoding='utf-8') as f: raise ModuleNotFoundError(f.read()) sys.meta_path.append(MissingPathFinder()) import foo Add a "foo.missing" file to your working directory and you'll get the message from that instead of the usual one. Cheers, Steve From rosuav at gmail.com Mon Nov 28 17:52:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Nov 2016 09:52:21 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On Tue, Nov 29, 2016 at 9:19 AM, Nathaniel Smith wrote: > This also suggests that the overall error-handling flow for 'import > foo' should look like: > > 1) run foo.missing.py > 2) if it raises an exception: propagate that > 3) otherwise, if sys.modules["foo"] is missing: raise some variety of > ImportError. > 4) otherwise, use sys.modules["foo"] as the object that should be > bound to 'foo' in the original invoker's namespace +1, because this also provides a coherent way to reword the try/except import idiom: # Current idiom # somefile.py try: import foo except ImportError: import subst_foo as foo # New idiom: # foo.missing.py import subst_foo as foo import sys; sys.modules["foo"] = foo #somefile.py import foo ChrisA From wolfgang.maier at biologie.uni-freiburg.de Mon Nov 28 18:00:28 2016 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 29 Nov 2016 00:00:28 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On 28.11.2016 23:52, Chris Angelico wrote: > > +1, because this also provides a coherent way to reword the try/except > import idiom: > > # Current idiom > # somefile.py > try: > import foo > except ImportError: > import subst_foo as foo > > # New idiom: > # foo.missing.py > import subst_foo as foo > import sys; sys.modules["foo"] = foo > #somefile.py > import foo > Hmm. I would rather take this example as an argument against the proposed behavior. It invites too many clever hacks. I thought that the idea was that .missing.py does *not* act as a replacement module, but, more or less, just as a message generator. From toddrjen at gmail.com Mon Nov 28 18:07:12 2016 From: toddrjen at gmail.com (Todd) Date: Mon, 28 Nov 2016 18:07:12 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: On Mon, Nov 28, 2016 at 6:00 PM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > On 28.11.2016 23:52, Chris Angelico wrote: > >> >> +1, because this also provides a coherent way to reword the try/except >> import idiom: >> >> # Current idiom >> # somefile.py >> try: >> import foo >> except ImportError: >> import subst_foo as foo >> >> # New idiom: >> # foo.missing.py >> import subst_foo as foo >> import sys; sys.modules["foo"] = foo >> #somefile.py >> import foo >> >> > Hmm. I would rather take this example as an argument against the proposed > behavior. It invites too many clever hacks. I thought that the idea was > that .missing.py does *not* act as a replacement module, but, more or less, > just as a message generator. > > Is there a reason we need a full-blown message generator? Why couldn't there just be a text file, and the contents of that text file are used as the error message for an ImportError? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Nov 28 18:50:57 2016 From: brett at python.org (Brett Cannon) Date: Mon, 28 Nov 2016 23:50:57 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> Message-ID: On Mon, 28 Nov 2016 at 14:49 Steve Dower wrote: > On 28Nov2016 1433, Steve Dower wrote: > > On 28Nov2016 1419, Nathaniel Smith wrote: > >> I'd suggest that we additional specify that if we find a > >> foo.missing.py, then the code is executed but -- unlike a regular > >> module load -- it's not automatically inserted into > >> sys.modules["foo"]. That seems like it could only create confusion. > >> And it doesn't restrict functionality, because if someone really wants > >> to implement some clever shenanigans, they can always modify > >> sys.modules["foo"] by hand. > > > > In before Brett says "you can do this with an import hook", because, > > well, we can do this with an import hook :) > > And since I suggested it, here's a rough proof-of-concept: > > import importlib.abc > import os > import sys > > class MissingPathFinder(importlib.abc.MetaPathFinder): > def find_spec(self, fullname, path, target=None): > for p in (path or sys.path): > file = os.path.join(p, fullname + ".missing") > if os.path.isfile(file): > with open(file, 'r', encoding='utf-8') as f: > raise ModuleNotFoundError(f.read()) > > sys.meta_path.append(MissingPathFinder()) > import foo > > > Add a "foo.missing" file to your working directory and you'll get the > message from that instead of the usual one. > Since this PEP directly affects import I'm going to weigh in. First, this won't necessarily create more stat calls depending on how it's implemented. importlib.machinery.FileFinder which does the searching for files on sys.path caches directory contents for as long as the granularity of the file system's mtime is (e.g. a 1 second mtime granularity means directory contents are cached for 1 second). This means that if the check occurs within that granularity (whether immediately after looking for *.py or in some way through a second pass) then there's no file system overhead. Second, as proposed the PEP probably shouldn't change importlib.machinery.SourceFileLoader and instead should return some new loader that only handles these *.missing.py files (just like a different loader is returned for extension modules). This allows for the loader to be simpler and avoids making any custom loader from no longer implementing current semantics (although I have tried to structure things in importlib to make it so subclassing is an attractive option for people so this isn't vital, just something to at least consider). Third, Steve channeled me properly and this actually doesn't require any changes to any pre-existing code and can instead be implemented as an importlib.abc.MetaPathFinder that is at the end of sys.meta_path which means there wouldn't be any local shadowing of modules available farther down sys.path (although this would lead to more stat calls). Fourth, if you make a meta-path finder and use static data you do away with any performance issue with the file system. And since it would be installed at the end of the sys.meta_path -- and thus after importlib.machinery.PathFinder -- you effectively shadow it with a successful import and so there's no need to worry about the information leaking out unless someone mucks with sys.meta_path. Fifth, people have asked for some way to catch/log/manipulate the response of import when a module isn't found, e.g. renaming modules in 2/3 migrations was the first major instance of this, but I've heard others wanting to log this to detect what modules they should install for users in a cloud environment. This is a specific solution to a general problem that some have asked for so it might warrant thinking about whether a more general solution could work (but never enough to warrant me trying to solve it above other issues). Sixth, this would be easier to deal with if import got refactored into its own object and out of the sys.module for easier manipulation of the import process. ;) Seventh, these *.missing.py files if they are directly executed are totally going to be abused like *.pth files, I can just feel it in my bones. We need to be okay with this if we accept this PEP as-is. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 28 19:24:44 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 Nov 2016 11:24:44 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <20161129002444.GI3365@ando.pearwood.info> On Mon, Nov 28, 2016 at 02:32:16PM +0000, Paul Moore wrote: > Also, and possibly more of an issue, use of the ".missing.py" file > will mean that a user can't provide their own implementation of the > module later on sys.path. I don'rt know if this is a significant issue > on Unix platforms. On Windows, there is a 3rd party implementation of > the curses module which (as I understand it) can be user installed. If > Python included a curses.missing.py, that would no longer work. > > Certainly these are only minor points, but worth considering. I don't think that's a minor point, I think its a very important one. -- Steve From steve at pearwood.info Mon Nov 28 19:55:34 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 Nov 2016 11:55:34 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <583C8DED.9020709@stoneleaf.us> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> <583C8DED.9020709@stoneleaf.us> Message-ID: <20161129005533.GJ3365@ando.pearwood.info> On Mon, Nov 28, 2016 at 12:05:01PM -0800, Ethan Furman wrote: > >Honestly, though, I'm not sure of the need for the PEP in general. > >"However, there is as of yet no standardized way of dealing with > >importing a missing standard library module." is simply not true. The > >standardized way of dealing with it is that the import statement will > >raise an ImportError exception. Why exactly is that not good enough? > > Because it is unfriendly. Helpful error messages are a great tool to both > beginner and seasoned programmers. Random already covers that. There's no reason why packagers can't fix that. > >A distribution could, for example, include an excepthook in site.py that > >prints an informative error message when an ImportError is unhandled for > >a list of modules that it knows about. [...] > > As you say above, that list will fall out of date. Better to have a > standard method that is easily implemented. I think you have misunderstood Random's observation. Random notes correctly that treating "curses on Windows" as a special case will get out of date. Today its curses, tomorrow it might be curses and foo, then curses foo and bar, then just foo. Who knows? And what about Linux and Mac users? Might we start deploying third-party replacements for Windows-only std lib modules? (If any.) This is effectively a black-list: - don't add a .missing file for these modules where the list depends on guessing what *other* people do. But Random's observation doesn't apply to the packager. They cannot fall out of date, since they're generating a *white-list* of modules they have split out of the std lib into a separate package. Instead of the packager doing this: - remove foo, bar, baz from the standard python package; - add foo.missing, bar.missing, baz.missing to the python package; - add foo, bar, baz to the python-extras package Random suggest that they do this: - remove foo, bar, baz from the standard python package; - add foo, bar, baz to the list of modules that ImportError knows about; - add foo, bar, baz to the python-extras package. It can no more get out of date than can the .missing files. Instead of adding a complex mechanism for searching the PYTHONPATH twice, the second time looking for .missing files, here's a counter proposal: - Add a single config file in a known, standard place, call it "missing.ini" for the sake of the argument; - If present, that file should be a list of module names as keys and custom error messages as values; foo: try running "yum install spam-python" bar: try running "yum install spam-python" baz: try running "yum install eggs-python" - When ImportError is raised, Python looks at that file, and if the module name is found, it gives the custom error message in addition to the regular error message: import foo ImportError: No module named 'foo' try running "yum install spam-python" -- Steve From steve at pearwood.info Mon Nov 28 20:14:55 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 Nov 2016 12:14:55 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: Message-ID: <20161129011455.GK3365@ando.pearwood.info> On Tue, Nov 29, 2016 at 09:52:21AM +1100, Chris Angelico wrote: > +1, because this also provides a coherent way to reword the try/except > import idiom: > > # Current idiom > # somefile.py > try: > import foo > except ImportError: > import subst_foo as foo Nice, clean and self-explanatory: import a module, if it fails, import its replacement. Obviously that idiom must die. *wink* > # New idiom: > # foo.missing.py > import subst_foo as foo > import sys; sys.modules["foo"] = foo > #somefile.py > import foo Black magic where the replacement happens out of sight. What if I have two files? # a.py try: import spam except ImportError: import ham as spam # b.py try: import spam except ImportError: import cornedbeef as spam The current idiom makes it the responsibility of the importer to decide what happens when the import fails. Perhaps it is as simple as importing a substitute module, but it might log a warning, monkey-patch some classes, who knows? The important thing is that the importer decides whether to fail or try something else. Your proposal means that the importer no longer has any say in the matter. The administrator of the site chooses what .missing files get added, they decide whether or not to log a warning, they decide whether to substitute ham.py or cornedbeef.py for spam. I do not like this suggestion, and the mere possibility that it could happen makes this entire .missing suggestion a very strong -1 from me. To solve the simple problem of providing a more useful error message when an ImportError occurs, we do not need to execute arbitrary code in a .missing.py file. That's using a nuclear-powered bulldozer to crack a peanut. -- Steve From random832 at fastmail.com Mon Nov 28 20:46:59 2016 From: random832 at fastmail.com (Random832) Date: Mon, 28 Nov 2016 20:46:59 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <583C8DED.9020709@stoneleaf.us> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> <583C8DED.9020709@stoneleaf.us> Message-ID: <1480384019.3800418.801906337.761BE5F1@webmail.messagingengine.com> On Mon, Nov 28, 2016, at 15:05, Ethan Furman wrote: > Because it is unfriendly. Helpful error messages are a great tool to > both beginner and seasoned programmers. There won't be a helpful error message unless the distributor writes one. > > A distribution could, for example, include an excepthook in site.py that > > prints an informative error message when an ImportError is unhandled for > > a list of modules that it knows about. [...] > > As you say above, that list will fall out of date. Better to have a > standard method that is easily implemented. Whatever the standard method is, it has to be something we can direct distributors to modify, it's simply not something Python can do on its own (which means maybe distributors should be part of the conversation here). The default exception hook is as good a place as any. Maybe write most of the logic and get the distributors to just populate an empty-by-default array of structs with the module name and error message (and what about localization?) And the idea that building a ".missing.py" for every optional module that's disabled is going to adequate is a bit naive. For one thing, they're not going to *be* disabled, the distributors are going to build the whole thing and break up the installed files into packages. And you've still got to get the distributors to actually put their friendly error message in those files, and the missing.py files are build artifacts instead of a source file that they can patch. From ethan at stoneleaf.us Mon Nov 28 21:21:30 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Nov 2016 18:21:30 -0800 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <1480384019.3800418.801906337.761BE5F1@webmail.messagingengine.com> References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> <1480350903.3034972.801341057.3DC0CC12@webmail.messagingengine.com> <583C8DED.9020709@stoneleaf.us> <1480384019.3800418.801906337.761BE5F1@webmail.messagingengine.com> Message-ID: <583CE62A.1050309@stoneleaf.us> On 11/28/2016 05:46 PM, Random832 wrote: > On Mon, Nov 28, 2016, at 15:05, Ethan Furman wrote: >> Because it is unfriendly. Helpful error messages are a great tool to >> both beginner and seasoned programmers. > > There won't be a helpful error message unless the distributor writes > one. The purpose of this PEP, if I understand correctly, is to settle on a standard for the location of that distributor written helpful error message. As a bonus, cpython itself can use the same mechanism for modules that are possible to build in the stdlib, but weren't. This would be useful for folks that build their own version. > Whatever the standard method is, it has to be something we can direct > distributors to modify, it's simply not something Python can do on its > own. --- Yo, Distributor! If you move tkinter to a separate module, please add a tkinter.missing file in your main python package where tkinter is supposed to be; this file should contain a helpful message on how to install the tkinter package! --- There. Done. > The default exception hook is as good a place as any. Maybe write > most of the logic and get the distributors to just populate an > empty-by-default array of structs with the module name and error message > (and what about localization?) This might handle the stdlib portion, but the PEP's solution could be easily extended to handle any Python application that is installable in pieces. As far as localization -- it's a small text file, surely there are mechanisms already to deal with that? (I don't know, it's not a problem I have to deal with.) > And the idea that building a ".missing.py" for every optional module > that's disabled is going to adequate is a bit naive. Who said disabled? The PEP says missing, as in not there -- not disabled, as in there but ... what? not able to be used? > For one thing, they're not going to *be* disabled, Ah, whew -- we agree on something! ;) > the distributors are going to build > the whole thing and break up the installed files into packages. And > you've still got to get the distributors to actually put their friendly > error message in those files, No, we don't. We provide a mechanism for them to use, and they use it or don't at their whim. It's a quality-of-implementation issue. > and the missing.py files are build > artifacts instead of a source file that they can patch. They are the ones that decide how to segment the stdlib, so they get to do the work. I would imagine, or at least hope, that they have the build and segmentation code under version control -- they can patch that. -- ~Ethan~ From rosuav at gmail.com Mon Nov 28 21:48:03 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Nov 2016 13:48:03 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <20161129011455.GK3365@ando.pearwood.info> References: <20161129011455.GK3365@ando.pearwood.info> Message-ID: On Tue, Nov 29, 2016 at 12:14 PM, Steven D'Aprano wrote: > What if I have two files? > > # a.py > try: > import spam > except ImportError: > import ham as spam > > # b.py > try: > import spam > except ImportError: > import cornedbeef as spam > In the same project? Then you already have a maintenance nightmare, because 'spam' will sometimes mean the same module (with state shared between the files), but might mean two distinct modules (and thus unrelated module objects). In different projects? They won't conflict. ChrisA From askvictor at gmail.com Mon Nov 28 23:58:05 2016 From: askvictor at gmail.com (victor rajewski) Date: Tue, 29 Nov 2016 04:58:05 +0000 Subject: [Python-ideas] (no subject) Message-ID: I teach a computing subject to high school students using Python as our primary language. One thing that often causes confusion at the start is error messages/exceptions. I think it would lower entry point quite a bit by making error messages more readable to novices. Recent research found reduced frequency of overall errors, errors per student, and repeated errors when error messages were enhanced [1]. This could be implemented by a command-line switch when running python, or a different python executable (along the lines of pythonw.exe vs python.exe) such as python-easy. A simpler error message might get rid of excess information which can confuse the novice (such info might be dumped to a file to allow more advanced users to debug), provide a brief description of what the error might mean, and/or how you might fix it. So a SyntaxError might tell the user that they've probably forgotten a :, and a NameError might say that the item has been defined yet, or they've made a typo. A couple of examples follow: Traceback (most recent call last): File "foo.py", line 2, in l[10]=14 IndexError: list assignment index out of range A better message might be: You tried to use l[10] when l is only 4 elements long. You can add items to l using l.append(value), or check your index value to make sure that's really the position you wanted to access. Traceback (most recent call last): File "foo.py", line 2, in while name != "quit" and reponse != "quit": NameError: name 'reponse' is not defined A better message might be: You're trying to use the value of 'reponse', but that variable hasn't got a value yet. You can give it a value earlier in the code, or it could be a typo. You have a variable called 'response' - is that the one you meant? Traceback (most recent call last): File "foo.py", line 2, in print(length(l)) NameError: name 'length' is not defined A better message might be: Python doesn't recognise the function "length". Did you mean len?' File "foo.py", line 2 for i in l ^ SyntaxError: invalid syntax A better message might be: You have a for loop without a : (colon). Try adding a colon at the end of the line. Any thoughts? [1]: http://researchrepository.ucd.ie/handle/10197/7583 -- Victor Rajewski -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Tue Nov 29 00:45:04 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Tue, 29 Nov 2016 03:45:04 -0200 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On 2016-11-29 02:58, victor rajewski wrote: > NameError: name 'reponse' is not defined > > A better message might be: > > You're trying to use the value of 'reponse', but that variable hasn't > got a value yet. You can give it a value earlier in the code, or it > could be a typo. You have a variable called 'response' - is that the one > you meant? > This is a really nice feature. GHC and Clang do it for several things and it usually helps (when a live highlighter isn't available). Given that these are HS students, presumably seeing Python for the first time, shouldn't they be using PyCharm to get used to PEP 8 (no l[10]=14) and other best practices (e.g., such as using 'with' instead of 'try/finally' when possible)? This would provide live syntactic (and some semantic) highlighting with short human-readable messages. Overall, I already find Python's exceptions quite readable, and your suggestions propose some VERY long lines which would certainly wrap at a terminal. Maybe this should be an optional feature. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From toddrjen at gmail.com Tue Nov 29 00:56:56 2016 From: toddrjen at gmail.com (Todd) Date: Tue, 29 Nov 2016 00:56:56 -0500 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: Although there is certainly room good improvement in the error messages, making error messages that work in all cases is hard. For example, functions are just another kind of variable, and it is possible for arbitrary variables to act like functions. So error messages that try to differentiate between the two can't with reliably. Also, for list index errors, the error is created by the list class, and it doesn't know the variable name, and there may not even be a variable. Can you give an error message that will work in your example and also work in this example: sorted(x%i for i, x in enumerate(range(3:999)))[::5].__getitem__(1000) This example seems very different to us but is identical as far as the list class is concerned, so you would need a single error message that works equally well in both cases. Syntax errors are also often hard. Consider your example, but with more lines around it: a = [step %s' % i for i in l ] Now your error message will give the wrong information. It often is very hard to even tell what line a syntax error occurs on, not to mention figure out exactly what went wrong. So yes, better error messages would be great. But the sorts of error messages you describe are not generally possible in practice. There are just too many corner- On Nov 28, 2016 11:58 PM, "victor rajewski" wrote: > I teach a computing subject to high school students using Python as our > primary language. One thing that often causes confusion at the start is > error messages/exceptions. I think it would lower entry point quite a bit > by making error messages more readable to novices. Recent research found > reduced frequency of overall errors, errors per student, and repeated > errors when error messages were enhanced [1]. > > This could be implemented by a command-line switch when running python, or > a different python executable (along the lines of pythonw.exe vs > python.exe) such as python-easy. A simpler error message might get rid of > excess information which can confuse the novice (such info might be dumped > to a file to allow more advanced users to debug), provide a brief > description of what the error might mean, and/or how you might fix it. So a > SyntaxError might tell the user that they've probably forgotten a :, and a > NameError might say that the item has been defined yet, or they've made a > typo. A couple of examples follow: > > Traceback (most recent call last): > > File "foo.py", line 2, in > > l[10]=14 > > IndexError: list assignment index out of range > > A better message might be: > > You tried to use l[10] when l is only 4 elements long. You can add items > to l using l.append(value), or check your index value to make sure that's > really the position you wanted to access. > > Traceback (most recent call last): > > File "foo.py", line 2, in > > while name != "quit" and reponse != "quit": > > NameError: name 'reponse' is not defined > > A better message might be: > > You're trying to use the value of 'reponse', but that variable hasn't got > a value yet. You can give it a value earlier in the code, or it could be a > typo. You have a variable called 'response' - is that the one you meant? > > Traceback (most recent call last): > > File "foo.py", line 2, in > > print(length(l)) > > NameError: name 'length' is not defined > > A better message might be: > > Python doesn't recognise the function "length". Did you mean len?' > > File "foo.py", line 2 > > for i in l > > ^ > > SyntaxError: invalid syntax > > A better message might be: > > You have a for loop without a : (colon). Try adding a colon at the end of > the line. > > > Any thoughts? > > [1]: http://researchrepository.ucd.ie/handle/10197/7583 > -- > > Victor Rajewski > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From askvictor at gmail.com Tue Nov 29 01:20:18 2016 From: askvictor at gmail.com (victor rajewski) Date: Tue, 29 Nov 2016 06:20:18 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: Sorry, I forgot the subject line! On Tue., 29 Nov. 2016, 3:58 pm victor rajewski, wrote: > I teach a computing subject to high school students using Python as our > primary language. One thing that often causes confusion at the start is > error messages/exceptions. I think it would lower entry point quite a bit > by making error messages more readable to novices. Recent research found > reduced frequency of overall errors, errors per student, and repeated > errors when error messages were enhanced [1]. > > This could be implemented by a command-line switch when running python, or > a different python executable (along the lines of pythonw.exe vs > python.exe) such as python-easy. A simpler error message might get rid of > excess information which can confuse the novice (such info might be dumped > to a file to allow more advanced users to debug), provide a brief > description of what the error might mean, and/or how you might fix it. So a > SyntaxError might tell the user that they've probably forgotten a :, and a > NameError might say that the item has been defined yet, or they've made a > typo. A couple of examples follow: > > Traceback (most recent call last): > > File "foo.py", line 2, in > > l[10]=14 > > IndexError: list assignment index out of range > > A better message might be: > > You tried to use l[10] when l is only 4 elements long. You can add items > to l using l.append(value), or check your index value to make sure that's > really the position you wanted to access. > > Traceback (most recent call last): > > File "foo.py", line 2, in > > while name != "quit" and reponse != "quit": > > NameError: name 'reponse' is not defined > > A better message might be: > > You're trying to use the value of 'reponse', but that variable hasn't got > a value yet. You can give it a value earlier in the code, or it could be a > typo. You have a variable called 'response' - is that the one you meant? > > Traceback (most recent call last): > > File "foo.py", line 2, in > > print(length(l)) > > NameError: name 'length' is not defined > > A better message might be: > > Python doesn't recognise the function "length". Did you mean len?' > > File "foo.py", line 2 > > for i in l > > ^ > > SyntaxError: invalid syntax > > A better message might be: > > You have a for loop without a : (colon). Try adding a colon at the end of > the line. > > > Any thoughts? > > [1]: http://researchrepository.ucd.ie/handle/10197/7583 > -- > > Victor Rajewski > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Victor Rajewski -------------- next part -------------- An HTML attachment was scrubbed... URL: From mariatta.wijaya at gmail.com Tue Nov 29 02:41:21 2016 From: mariatta.wijaya at gmail.com (Mariatta Wijaya) Date: Mon, 28 Nov 2016 23:41:21 -0800 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: I'm +1 to the idea of improving error messages :) (but maybe not to the exact new error messages proposed) Raymond Hettinger touched on this topic during his Pycon Canada keynote, as one of the positive contributions that you can do to cpython. > > Traceback (most recent call last): > > File "foo.py", line 2, in > > print(length(l)) > > NameError: name 'length' is not defined > > A better message might be: > > Python doesn't recognise the function "length". Did you mean len?' > I recall he gave a similar example to this, where Python could suggest an alternative in case of typo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Nov 29 04:05:12 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Nov 2016 10:05:12 +0100 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: Hi, Python is optimized for performance. Formatting an error message has a cost on performances. I suggest you to teach your student to use the REPL and use a custom exception handler: sys.excepthook: https://docs.python.org/2/library/sys.html#sys.excepthook Using a custom exception handler, you can run expensive functions, like the feature: "suggest len when length is used". The problem is then when students have to use a Python without the custom exception handler. Victor From mal at egenix.com Tue Nov 29 04:13:12 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 29 Nov 2016 10:13:12 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> Message-ID: <583D46A8.2050109@egenix.com> On 29.11.2016 00:50, Brett Cannon wrote: > Seventh, these *.missing.py files if they are directly executed are totally > going to be abused like *.pth files, I can just feel it in my bones. We > need to be okay with this if we accept this PEP as-is. Since the purpose of the PEP was to allow distributors to guide users through the installation process of extra packages in order to get access to parts of the stdlib which are not installed, I think the PEP is overly broad in concept to address this one use case. Just as with .pth files, the possibility to hook arbitrary code execution into the module search path will get abused for all kinds of weird things, esp. if the whole sys.path is scanned for the .missing.py module and not only the part where the stdlib lives (as was suggested in the thread). So why not limit the PEP to just the intended use case ? I.e. define a static list of modules which do make up the Python stdlib and then have the importer turn a ModuleNotFoundError error into a nice distribution specific error message, if and only if the imported module is from the set of stdlib modules. The change of the error message could be done by having the distributor patch the importer or we could have the importer call a function defined via sitecustomize.py by the distributor to return a message. Thinking about this some more... We don't even need a list of stdlib modules. Simply define a general purpose import error formatting function, e.g. sys.formatimporterror(), pass in the name of the module and let it determine the error message based on the available information. A distributor could then provide a custom function that knows about the installed Python packages and then guides the user to install any missing ones. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 29 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From p.f.moore at gmail.com Tue Nov 29 04:39:45 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Nov 2016 09:39:45 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> Message-ID: On 28 November 2016 at 22:33, Steve Dower wrote: > Given that, this wouldn't necessarily need to be an executable file. The > finder could locate a "foo.missing" file and raise ModuleNotFoundError with > the contents of the file as the message. No need to allow/require any Python > code at all, and no risk of polluting sys.modules. I like this idea. Would it completely satisfy the original use case for the proposal? (Or, to put it another way, is there any specific need for arbitrary code execution in the missing.py file?) Paul From wes.turner at gmail.com Tue Nov 29 04:54:32 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 29 Nov 2016 03:54:32 -0600 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: The existing docs for errors and exceptions: - https://docs.python.org/2/library/exceptions.html - https://docs.python.org/3/library/exceptions.html - https://hg.python.org/cpython/file/tip/Doc/library/exceptions.rst - https://github.com/python/cpython/blob/master/Doc/library/exceptions.rst - https://docs.python.org/2/tutorial/errors.html - https://docs.python.org/3/tutorial/errors.html - https://hg.python.org/cpython/file/tip/Doc/tutorial/errors.rst - https://github.com/python/cpython/blob/master/Doc/tutorial/errors.rst - https://www.tutorialspoint.com/python/python_exceptions.htm - If the docs don't answer the question (and match to the search terms), they probably should. - [ ] DOC: something about why "except Exception: pass" is usually bad - [ ] DOC: something about SystemExit and atexit: https://docs.python.org/2/library/atexit.html You can get alot more traceback from pytest (w/ pytest-sugar) and/or nose (with nose-progressive). There is extra information in the stack at exception time; but, IIUC, it would take a number of subclasses with class-specific docs and/or class introspection to be as detailed as "you probably wanted .append there because this is a List and the length is n but the key was". Maybe a "learning mode" which automatically calls inspect.getdoc() on Exception would be useful (sys.excepthook)? Practically, I usually just open an extra IPython shell and run `list.append?` for docs or `list.append??` for (Python but not C!) source (inspect.getsource). IPython also prints the function signature with `?` The pdb++ debugger requires funcsigs in order to print function signatures. If pdb++ is installed, it preempts the standard pdb module; so `nosetests --pdb` and `pytest --pdb` launch pdb++ when an error or exception is raised. https://pypi.python.org/pypi/pdbpp/ http://nose.readthedocs.io/en/latest/plugins/debug.html http://doc.pytest.org/en/latest/usage.html https://docs.python.org/2/library/inspect.html Exceptions could be better someday. Testing (and debugging) skills are always good to learn; coincidentally, there are many great tools for it. ... https://westurner.org/wiki/awesome-python-testing#debugging On Tuesday, November 29, 2016, Victor Stinner wrote: > Hi, > > Python is optimized for performance. Formatting an error message has a > cost on performances. > > I suggest you to teach your student to use the REPL and use a custom > exception handler: sys.excepthook: > https://docs.python.org/2/library/sys.html#sys.excepthook > > Using a custom exception handler, you can run expensive functions, > like the feature: "suggest len when length is used". > > The problem is then when students have to use a Python without the > custom exception handler. > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 29 05:38:23 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Nov 2016 02:38:23 -0800 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner wrote: > Hi, > > Python is optimized for performance. Formatting an error message has a > cost on performances. Sure, but we have to look at this on a case-by-case basis. Is there really important code out there that's generating NameErrors or SyntaxErrors in an inner loop? That seems unlikely to me. Even IndexError I'm a bit skeptical about. I can believe that there's code that intentionally generates and then catches IndexError, but AttributeError in my experience is much more performance-sensitive than IndexError, because every failed hasattr call allocates an AttributeError and hasattr is commonly used for feature checks. Yet AttributeError has a much more informative (= expensive) message than IndexError: In [1]: object().a AttributeError: 'object' object has no attribute 'a' In [2]: list()[0] IndexError: list index out of range -n -- Nathaniel J. Smith -- https://vorpus.org From cory at lukasa.co.uk Tue Nov 29 05:46:23 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 29 Nov 2016 10:46:23 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <20161129011455.GK3365@ando.pearwood.info> Message-ID: <211ABEE7-41BD-4DDC-8093-31BBDA4EDCDD@lukasa.co.uk> > On 29 Nov 2016, at 02:48, Chris Angelico wrote: > > On Tue, Nov 29, 2016 at 12:14 PM, Steven D'Aprano wrote: >> What if I have two files? >> >> # a.py >> try: >> import spam >> except ImportError: >> import ham as spam >> >> # b.py >> try: >> import spam >> except ImportError: >> import cornedbeef as spam >> > > In the same project? Then you already have a maintenance nightmare, > because 'spam' will sometimes mean the same module (with state shared > between the files), but might mean two distinct modules (and thus > unrelated module objects). In different projects? They won't conflict. Well, you *might* have a maintenance nightmare, but you might not. In particular, I should point out that ?spam? is just a name (more correctly referred to as a.spam and b.spam.) If the ?spam? module is intended to have global state that the ?a? and ?b? modules use to communicate then obviously this is a problem. But if it isn?t, then there is exactly no problem with each module choosing its own fallback. As a really silly example, consider sqlite3 again. If there were third-party modules that both implement the sqlite3 API, then there is no reason for each module to agree on what sqlite3 module they use unless types are being passed between them. If we consider ?a? and ?b? as truly separate non-communicating modules, then there?s no issue at all. Cory From wolfgang.maier at biologie.uni-freiburg.de Tue Nov 29 05:51:27 2016 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 29 Nov 2016 11:51:27 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> Message-ID: On 29.11.2016 10:39, Paul Moore wrote: > On 28 November 2016 at 22:33, Steve Dower wrote: >> Given that, this wouldn't necessarily need to be an executable file. The >> finder could locate a "foo.missing" file and raise ModuleNotFoundError with >> the contents of the file as the message. No need to allow/require any Python >> code at all, and no risk of polluting sys.modules. > > I like this idea. Would it completely satisfy the original use case > for the proposal? (Or, to put it another way, is there any specific > need for arbitrary code execution in the missing.py file?) > The only thing that I could think of so far would be cross-platform .missing.py files that query the system (e.g. using the platform module) to generate adequate messages for the specific platform or distro. E.g., correctly recommend to use dnf install or yum install or apt install, etc. From torsava at redhat.com Tue Nov 29 05:54:47 2016 From: torsava at redhat.com (Tomas Orsava) Date: Tue, 29 Nov 2016 11:54:47 +0100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 11/28/2016 05:38 PM, Guido van Rossum wrote: > Overall I think this is a good idea. I have one hit: > > It seems that there are two possible strategies for searching the > .missing.py file: > > 1. (Currently in the PEP) search it at the same time as the .py file > when walking along sys.path. > - Pro: prevents confusion when the user accidentally has their own > matching file later in sys.path. > - Con: prevents the user from installing a matching file > intentionally (e.g. a 3rd party version). > > 2. After exhausting sys.path, search it again just for .missing.py > files (or perhaps remember the location of the .missing.py file during > the first search but don't act immediately on it -- this has the same > effect). > - Pro: allows user to install their own version. > - Con: if the user has a matching file by accident, that file will > be imported, causing more confusion. > > I personally would weigh these so as to prefer (2). The option of > installing your own version when the standard version doesn't exist > seems reasonable; there may be reasons that you can't or don't want to > install the distribution's version. I don't worry much about the > danger of accidental name conflicts (have you ever seen this?). > > --Guido Solution (2) is a very good alternative and can be implemented using a metapath hook as Steve proposed elsewhere in this thread [0]. We considered a similar metapath hook when designing the PEP, but decided against it, to better match the current behavior of third-party modules not being able to replace parts of stdlib. Note that as Brett says elsewhere in the thread, due to caching there would be no extra stat() calls in the usual case. On the other hand, we aren't familiar with Windows, where replacing missing stdlib modules seems to be standard practice. Thanks for letting us know. With a metapath hook, .missing.py files are probably overkill, and the hook can just look at one file (or a static compiled-in list) of ModuleNotFound/ImportError messages for all missing modules, as M.-A. Lemburg and others are suggesting. We'll just need to think about coordinating how the list is generated/updated: the current PEP implicitly allows other parties, besides Python and the distributors, to step in cleanly if they need to?needing to update a single list could lead to messy hacks. We'll update the PEP to go with solution (2). [0] https://mail.python.org/pipermail/python-ideas/2016-November/043837.html Tomas Orsava > > On Mon, Nov 28, 2016 at 8:13 AM, Paul Moore > wrote: > > On 28 November 2016 at 15:51, Tomas Orsava > wrote: > > I believe I may have found the Windows curses implementation, > it's called > > PDCurses [0], and this website [1] appears to be distributing it > under the > > name `curses`. > > My apologies, I should have included a pointer. That is indeed the > distribution I was thinking of. > > > Could some Windows user please check if compiling Python with > the current > > reference implementation [2] of this PEP indeed generates a > > `curses.missing.py ` file among the > stdlib files? If so, we might consider > > skipping the generation of the .missing.py file for the curses > module on > > Windows. > > I'll see if I can make some time to do the test. But as the change is > to setup.py, and the Windows build uses Visual Studio project files to > do the build, I expect that it won't generate missing.py files on > Windows. In actual fact, that may be the simplest solution, to note > that the build part of this change is restricted to Unix (non-Windows) > platforms specifically. As there's no real concept of a "distribution > version" of Python on Windows, it's probably not something that will > be that important on that platform (and support for .missing.py files > is there, it would just be necessary for distributors to manually > create those files as needed). > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Nov 29 06:20:03 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 29 Nov 2016 05:20:03 -0600 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tuesday, November 29, 2016, Nathaniel Smith wrote: > On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner > > wrote: > > Hi, > > > > Python is optimized for performance. Formatting an error message has a > > cost on performances. > > Sure, but we have to look at this on a case-by-case basis. Is there > really important code out there that's generating NameErrors or > SyntaxErrors in an inner loop? That seems unlikely to me. > > Even IndexError I'm a bit skeptical about. I can believe that there's > code that intentionally generates and then catches IndexError, but > AttributeError in my experience is much more performance-sensitive > than IndexError, because every failed hasattr call allocates an > AttributeError and hasattr is commonly used for feature checks. Yet > AttributeError has a much more informative (= expensive) message than > IndexError: > > In [1]: object().a > AttributeError: 'object' object has no attribute 'a' > > In [2]: list()[0] > IndexError: list index out of range https://docs.python.org/2/tutorial/datastructures.html#more-on-lists https://docs.python.org/2/c-api/list.html#c.PyList_SetItem https://github.com/python/cpython/search?utf8=?&q=PyList_SetItem - https://github.com/python/cpython/blob/master/Include/listobject.h - https://github.com/python/cpython/blob/master/Objects/listobject.c#L208 - https://hg.python.org/cpython/file/tip/Objects/listobject.c#l208 https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes - list (typeshed: List) - Sequence - MutableSequence - It would be great if these continue to match: https://www.google.com/search?q=IndexError%3A+list+index+out+of+range > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Nov 29 06:33:07 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 29 Nov 2016 05:33:07 -0600 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tuesday, November 29, 2016, Wes Turner wrote: > The existing docs for errors and exceptions: > > - https://docs.python.org/2/library/exceptions.html > - https://docs.python.org/3/library/exceptions.html > - https://hg.python.org/cpython/file/tip/Doc/library/exceptions.rst > - https://github.com/python/cpython/blob/master/Doc/library/exceptions.rst > > - https://docs.python.org/2/tutorial/errors.html > - https://docs.python.org/3/tutorial/errors.html > - https://hg.python.org/cpython/file/tip/Doc/tutorial/errors.rst > - https://github.com/python/cpython/blob/master/Doc/tutorial/errors.rst > > - https://www.tutorialspoint.com/python/python_exceptions.htm > > - If the docs don't answer the question (and match to the search terms), > they probably should. > > - [ ] DOC: something about why "except Exception: pass" is usually bad > - [ ] DOC: something about SystemExit and atexit: https://docs.python. > org/2/library/atexit.html > > > You can get alot more traceback from pytest (w/ pytest-sugar) and/or nose > (with nose-progressive). > > There is extra information in the stack at exception time; but, IIUC, it > would take a number of subclasses with class-specific docs and/or class > introspection to be as detailed as "you probably wanted .append there > because this is a List and the length is n but the key was". > > Maybe a "learning mode" which automatically calls inspect.getdoc() on > Exception would be useful (sys.excepthook)? > Practically, I usually just open an extra IPython shell and run > `list.append?` for docs or `list.append??` for (Python but not C!) source > (inspect.getsource). > IPython also prints the function signature with `?` > - [ ] How could I also print out type annotations w/ function signatures and docstrings?/??/??? https://github.com/python/typeshed/blob/master/stdlib/2/typing.pyi > > The pdb++ debugger requires funcsigs in order to print function > signatures. If pdb++ is installed, it preempts the standard pdb module; so > `nosetests --pdb` and `pytest --pdb` launch pdb++ when an error or > exception is raised. > > https://pypi.python.org/pypi/pdbpp/ > > http://nose.readthedocs.io/en/latest/plugins/debug.html > > http://doc.pytest.org/en/latest/usage.html > > https://docs.python.org/2/library/inspect.html > > Exceptions could be better someday. Testing (and debugging) skills are > always good to learn; coincidentally, there are many great tools for it. > > ... https://westurner.org/wiki/awesome-python-testing#debugging > > On Tuesday, November 29, 2016, Victor Stinner > wrote: > >> Hi, >> >> Python is optimized for performance. Formatting an error message has a >> cost on performances. >> >> I suggest you to teach your student to use the REPL and use a custom >> exception handler: sys.excepthook: >> https://docs.python.org/2/library/sys.html#sys.excepthook >> >> Using a custom exception handler, you can run expensive functions, >> like the feature: "suggest len when length is used". >> >> The problem is then when students have to use a Python without the >> custom exception handler. >> >> Victor >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Nov 29 07:11:47 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Nov 2016 12:11:47 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> Message-ID: On 29 November 2016 at 10:51, Wolfgang Maier wrote: > On 29.11.2016 10:39, Paul Moore wrote: >> >> On 28 November 2016 at 22:33, Steve Dower wrote: >>> >>> Given that, this wouldn't necessarily need to be an executable file. The >>> finder could locate a "foo.missing" file and raise ModuleNotFoundError >>> with >>> the contents of the file as the message. No need to allow/require any >>> Python >>> code at all, and no risk of polluting sys.modules. >> >> >> I like this idea. Would it completely satisfy the original use case >> for the proposal? (Or, to put it another way, is there any specific >> need for arbitrary code execution in the missing.py file?) >> > > The only thing that I could think of so far would be cross-platform > .missing.py files that query the system (e.g. using the platform module) to > generate adequate messages for the specific platform or distro. E.g., > correctly recommend to use dnf install or yum install or apt install, etc. Yeah. I'd like to see a genuine example of how that would be used in practice, otherwise I'd be inclined to suggest YAGNI. (Particularly given that this PEP is simply a standardised means of vendor customisation - for special cases, vendors obviously still have the capability to patch or override standard behaviour in any way they like). Paul From ncoghlan at gmail.com Tue Nov 29 08:39:38 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Nov 2016 23:39:38 +1000 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> <20161126003257.GC3365@ando.pearwood.info> Message-ID: On 29 November 2016 at 08:01, Chris Barker wrote: > On Mon, Nov 28, 2016 at 1:50 PM, Guido van Rossum wrote: >>> >>> Also -- the ship has kinda sailed on this - maybe a @not_override would >>> make more sense. >>> >>> Isn't the goal to make sure you don't accidentally override a method? >>> saying "I know I'm overriding this" is less useful than "I'm not intending >>> to override anything here" >> >> I think you're fighting a straw man. I never said @override should be >> added to the language. I said that it would be useful to have a 3rd party >> metaclass or a class decorator that implements it which packages may >> voluntarily use to constrain their subclasses (or their own uses -- >> different designs are possible). > > > I know -- I just happened to add that to a reply to you... > > That was for the OP, or anyone else thinking of writing such a thing. > > And I still think it would better be added to a linting tool than at > run-time -- but let whoever writes it figure that out. Writing linting tools for Python class hierarchies is pretty hard in the general case, since you have to account for the fact that the module level control flow logic is Turing complete (and many linters simply don't try, instead saying "don't do that if you want the linter to work properly"). By contrast, a class decorator or metaclass can do the checks at runtime by walking the MRO, just as ABCMeta can look for @abstractmethod declarations that haven't been overridden in a subclass, and SQL Alchemy can map subclass inheritance to table linkages. If you look at the way attrs writes its automatic __init__ methods for example, it creates a single flat __init__ based on the definition time MRO: https://attrs.readthedocs.io/en/stable/how-does-it-work.html That's the nice part about this kind of thing being opt-in: the definition time tooling *doesn't* have to cope with the full extent of Python's dynamic nature, as it can expect the user to be cooperating with the tool to some degree, and "you're not cooperating" can be a valid error to throw. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 29 08:48:58 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Nov 2016 23:48:58 +1000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On 29 November 2016 at 20:38, Nathaniel Smith wrote: > On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner > wrote: >> Hi, >> >> Python is optimized for performance. Formatting an error message has a >> cost on performances. > > Sure, but we have to look at this on a case-by-case basis. Is there > really important code out there that's generating NameErrors or > SyntaxErrors in an inner loop? That seems unlikely to me. Right, we generally treat error message formatting code as being off the critical performance path. In many (most?) cases, the instances of uninformative error message are just a symptom of the code in question being really *old*, such that it predates the great many improvements made to the low level error reporting machinery over the years. That's not always true (e.g. parser errors are uninformative because the parser doesn't keep track of the state needed to generate nicer messages), but it seems to typically be true for runtime errors where we don't even report the type or representation of a misbehaving value. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 29 09:09:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Nov 2016 00:09:22 +1000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 29 November 2016 at 03:28, Guido van Rossum wrote: > On Mon, Nov 28, 2016 at 9:14 AM, Nathaniel Smith wrote: >> >> Also note that in Guido's option 2, we only incur the extra fstat calls if >> the import would otherwise fail. In option 1, there are extra fstat calls >> (and thus disk seeks etc.) adding some overhead to every import. > > Oh, that's an important consideration! Yes, adding .missing.py to the list > of extensions would cause extra stat() calls and potentially slow down every > import. This is the second time I've seen "but stat calls!" concern in relation to import today, so I'll echo what Brett pointed out in his reply: the import system in recent 3.x releases, along with the importlib2 backport to Python 2.7, builds a cache of the directory contents for path entries rather than making multiple stat calls. The current 3.x source code for that is at https://hg.python.org/cpython/file/tip/Lib/importlib/_bootstrap_external.py#l1280 The significant reduction in the number of stat calls through better caching is the main way the 3.3 import reimplementation in Python managed to be competitive performance-wise with the previous C implementation, and faster when importing from a network filesystem :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 29 09:48:11 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Nov 2016 00:48:11 +1000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 29 November 2016 at 20:54, Tomas Orsava wrote: > With a metapath hook, .missing.py files are probably overkill, and the hook > can just look at one file (or a static compiled-in list) of > ModuleNotFound/ImportError messages for all missing modules, as M.-A. > Lemburg and others are suggesting. We'll just need to think about > coordinating how the list is generated/updated: the current PEP implicitly > allows other parties, besides Python and the distributors, to step in > cleanly if they need to?needing to update a single list could lead to messy > hacks. What if, rather than using an explicitly file-based solution, this was instead defined as a new protocol module, where the new metapath hook imported a "__missing__" module and called a particular function in it (e.g. "__missing__.module_not_found(modname)")? The default missing module implementation hook would just handle CPython's optional modules, but redistributors could patch it to use a mechanism that made sense for them. For example, if we ever get to the point where the Fedora RPM database includes "Provides: pythonXYimport(module.of.interest)" data in addition to "Provides: pythonXYdist(pypi-package-name)" , the right system package to import could be reported for any module, not just standard library ones that have been split out (with the trade-off being that any such checks would make optional imports a bit slower to fail, but that could be mitigated in various ways). Specific applications could also implement their own missing module handling by providing a __missing__.py file alongside their __main__.py, and relying on directory and/or zipfile execution, or else by monkeypatching the __missing__ module at runtime. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From toddrjen at gmail.com Tue Nov 29 10:24:03 2016 From: toddrjen at gmail.com (Todd) Date: Tue, 29 Nov 2016 10:24:03 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> Message-ID: On Nov 29, 2016 5:51 AM, "Wolfgang Maier" < wolfgang.maier at biologie.uni-freiburg.de> wrote: > > On 29.11.2016 10:39, Paul Moore wrote: >> >> On 28 November 2016 at 22:33, Steve Dower wrote: >>> >>> Given that, this wouldn't necessarily need to be an executable file. The >>> finder could locate a "foo.missing" file and raise ModuleNotFoundError with >>> the contents of the file as the message. No need to allow/require any Python >>> code at all, and no risk of polluting sys.modules. >> >> >> I like this idea. Would it completely satisfy the original use case >> for the proposal? (Or, to put it another way, is there any specific >> need for arbitrary code execution in the missing.py file?) >> > > The only thing that I could think of so far would be cross-platform .missing.py files that query the system (e.g. using the platform module) to generate adequate messages for the specific platform or distro. E.g., correctly recommend to use dnf install or yum install or apt install, etc. In those cases it would probably be as easy, if not easier, to do that at build-time, which would get us back to simple text files. Making a standard script is hard, if not impossible, in many cases because the package name often does not match the module name. So you are going to need manual intervention in many cases, and modifying a one-line text file is going to be easier than modifying a script. -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Tue Nov 29 10:55:14 2016 From: toddrjen at gmail.com (Todd) Date: Tue, 29 Nov 2016 10:55:14 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: <583D46A8.2050109@egenix.com> References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> <583D46A8.2050109@egenix.com> Message-ID: On Tue, Nov 29, 2016 at 4:13 AM, M.-A. Lemburg wrote: > On 29.11.2016 00:50, Brett Cannon wrote: > > Seventh, these *.missing.py files if they are directly executed are > totally > > going to be abused like *.pth files, I can just feel it in my bones. We > > need to be okay with this if we accept this PEP as-is. > > Since the purpose of the PEP was to allow distributors to guide > users through the installation process of extra packages in order > to get access to parts of the stdlib which are not installed, > I think the PEP is overly broad in concept to address this one > use case. > > Just as with .pth files, the possibility to hook arbitrary code > execution into the module search path will get abused for > all kinds of weird things, esp. if the whole sys.path is > scanned for the .missing.py module and not only the part > where the stdlib lives (as was suggested in the thread). > > So why not limit the PEP to just the intended use case ? > > I think a better question is why should we artificially limit the PEP? These is something that could be useful outside of the stdlib. At least for Linux packages it is common to split out optional components of a python package into separate linux packages to limit the size and dependencies of the main package. This could help a lot in that situation. > I.e. define a static list of modules which do make up the Python > stdlib and then have the importer turn a ModuleNotFoundError error > into a nice distribution specific error message, if and only > if the imported module is from the set of stdlib modules. > This is hard to do in a general sense. The point is to be able to tell the user what package they should install to get that functionality, but there is no general rule as to what the package should be named, and platform-specific modules would not be installable at all. So every module would need its own error message defined. > > Thinking about this some more... > > We don't even need a list of stdlib modules. Simply define > a general purpose import error formatting function, e.g. > sys.formatimporterror(), pass in the name of the module and > let it determine the error message based on the available > information. > > A distributor could then provide a custom function that > knows about the installed Python packages and then guides > the user to install any missing ones. > > This is getting pretty complicated compared to simply defining a one-line text file containing the error message with the module name somewhere in the file name, as others have proposed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Tue Nov 29 11:32:05 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 29 Nov 2016 16:32:05 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On 29/11/2016 04:58, victor rajewski wrote: > > Traceback (most recent call last): > > File "foo.py", line 2, in > > l[10]=14 > > IndexError: list assignment index out of range > > > A better message might be: > > You tried to use l[10] when l is only 4 elements long. You can add > items to l using l.append(value), or check your index value to make > sure that's really the position you wanted to access. > > > It would make sense to me to upgrade this particular error message to IndexError: list assignment index 10 out of range 0 to 3 if it can be done without too much difficulty or overhead. (An empty list, and perhaps one with only 1 element, would be special cases.) Come to think of it, is the word "assignment" needed? Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 29 12:30:16 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 30 Nov 2016 04:30:16 +1100 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <1e398c9e-1b67-9f6d-0645-a9d19061d110@python.org> <2e0d8c6e-aeaa-2a93-0f8f-a78fe74d6ebf@python.org> <583D46A8.2050109@egenix.com> Message-ID: <20161129173016.GM3365@ando.pearwood.info> On Tue, Nov 29, 2016 at 10:55:14AM -0500, Todd wrote: > On Tue, Nov 29, 2016 at 4:13 AM, M.-A. Lemburg wrote: > > Just as with .pth files, the possibility to hook arbitrary code > > execution into the module search path will get abused for > > all kinds of weird things, esp. if the whole sys.path is > > scanned for the .missing.py module and not only the part > > where the stdlib lives (as was suggested in the thread). > > > > So why not limit the PEP to just the intended use case ? > I think a better question is why should we artificially limit the PEP? Because YAGNI. Overly complex, complicated systems which do more than is needed "because it might be useful one day" is an anti-pattern. The intended use-case is to allow Linux distributions to customize the error message on ImportError. From there, it is a small step to allow *other* people to do the same thing. But it is a BIG step to go from that to a solution that executes arbitrary code. Before we take that big step, we ought to have a good reason. > These is something that could be useful outside of the stdlib. Sure. I don't think there is any proposal to prevent people outside of Linux package distributors from using this mechanism. I'm not sure how this would even be possible: if Red Hat or Debian can create a .missing file, so can anyone else. > At least > for Linux packages it is common to split out optional components of a > python package into separate linux packages to limit the size and > dependencies of the main package. This could help a lot in that situation. Well... I'm not sure how common that it. But it doesn't really matter. This is a good argument for having a separate .missing file for each module, rather than a single flat registry of custom error messages. Separate .missing files will allow any Python package to easily install their own message. But either way, whether there's a single registry or an import hook that searches for .missing files if and only if the import failed, I haven't seen a strong argument for allowing arbitrary Python code. (Earlier I suggested such a flat registry -- I now withdraw that suggestion. I'm satisfied that a separate spam.missing file containing the custom error message when spam.py cannot be found is a better way to handle this.) -- Steve From brett at python.org Tue Nov 29 12:43:20 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Nov 2016 17:43:20 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith wrote: > On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner > wrote: > > Hi, > > > > Python is optimized for performance. Formatting an error message has a > > cost on performances. > > Sure, but we have to look at this on a case-by-case basis. Is there > really important code out there that's generating NameErrors or > SyntaxErrors in an inner loop? That seems unlikely to me. > > Even IndexError I'm a bit skeptical about. I can believe that there's > code that intentionally generates and then catches IndexError, but > AttributeError in my experience is much more performance-sensitive > than IndexError, because every failed hasattr call allocates an > AttributeError and hasattr is commonly used for feature checks. Yet > AttributeError has a much more informative (= expensive) message than > IndexError: > > In [1]: object().a > AttributeError: 'object' object has no attribute 'a' > > In [2]: list()[0] > IndexError: list index out of range > One way to make this cheap is to have a reasonable default message and use attributes on the exceptions trigger the use of the default message. Nearly a year ago I filed a bunch of issues for ideas on providing attributes on exceptions where it made sense, e.g. an index attribute on IndexError ( http://bugs.python.org/issue18162). If we did this then for classes like IndexError there constructor could be `IndexError(index=10, start=0, end=3)` and then __str__() can lazily construct the string representation using a default message, e.g. `"index {} is out of range of{} to {}".format(index, start, end)`. Make the arguments keyword-only and they become backwards-compatible and so the only overhead you pay for these richer messages are keyword-based construction if you simply never access the repr for the exception. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 29 12:42:57 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 29 Nov 2016 09:42:57 -0800 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan wrote: > > SyntaxErrors in an inner loop? That seems unlikely to me. > Syntax Errors are a special case, as by definition the code isn't being run yet (yes, there could be an eval in there...) So we could at least make those more informative without worrying about performance. Also -- would it be possible to tack on the more informative message at a higher level? Once the Exception bubbles up to the REPL, is there enough information available to make a more informative message? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Nov 29 13:26:50 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 29 Nov 2016 12:26:50 -0600 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: I would consider the speed of the "ultimate error handler" (i.e. whatever prints the traceback and kills the program) in the interpreter to be moot, so long as it takes a small fraction of a second. Optimizing Python's speed it crashes super-fast due to an *unhandled* NameError in your program seems folly. Regarding more informative messages for (e.g.) IndexError, would those just apply to built-in types as they're the most universal, or should some additional introspection be done for similar ducks? If it's useful/there's interest, I could try to do some analysis of Python questions on SO and see what the most common errors are. I'd guess things like "'NoneType' object has no attribute ..." would probably be up there, but that's a whole can of worms as to why someone's trying to call a method on, index, etc. something they accidentally whacked (a = a.sort()). On Tue, Nov 29, 2016 at 11:42 AM, Chris Barker wrote: > On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan wrote: > >> > SyntaxErrors in an inner loop? That seems unlikely to me. >> > > Syntax Errors are a special case, as by definition the code isn't being > run yet (yes, there could be an eval in there...) > > So we could at least make those more informative without worrying about > performance. > > Also -- would it be possible to tack on the more informative message at a > higher level? Once the Exception bubbles up to the REPL, is there enough > information available to make a more informative message? > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Nov 29 13:33:16 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Nov 2016 18:33:16 +0000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On Tue, 29 Nov 2016 at 06:49 Nick Coghlan wrote: > On 29 November 2016 at 20:54, Tomas Orsava wrote: > > With a metapath hook, .missing.py files are probably overkill, and the > hook > > can just look at one file (or a static compiled-in list) of > > ModuleNotFound/ImportError messages for all missing modules, as M.-A. > > Lemburg and others are suggesting. We'll just need to think about > > coordinating how the list is generated/updated: the current PEP > implicitly > > allows other parties, besides Python and the distributors, to step in > > cleanly if they need to?needing to update a single list could lead to > messy > > hacks. > > What if, rather than using an explicitly file-based solution, this was > instead defined as a new protocol module, where the new metapath hook > imported a "__missing__" module and called a particular function in it > (e.g. "__missing__.module_not_found(modname)")? > You can answer this question the best, Nick, but would it be worth defining a _stdlib.py that acts as both a marker for where the stdlib is installed -- instead of os.py which is the current marker -- and which also stores metadata like an attribute called `missing` which is a dict that maps modules to ModuleNotFoundError messages? Although maybe this is too specific of a solution (or still too general and we use an e.g. missing.json off of sys.path which contains the same mapping). Otherwise MAL touched on the solution I always had in the back of my head where we let people register a callback that gets passed the name of any module that wasn't found through sys.meta_path. We could either have the return value mean nothing and by default raise ModuleNotFoundError, have the return value be what to set the module to and raise an exception as expected, or have it be more error-specific and return an exception to raise (all of these options also ask whether a default callback doing what is normal is provided or if it's None by default and import continues to provide the default semantics). The perk of the callback is it removes the order sensitivity of any sys.meta_path or sys.path_hooks solution where people might be doing sys.meta_path.append(custom_finder) and thus won't necessarily trigger if a new hook for missing modules is put at the end by default. My personal vote is a callback called at https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948 with a default implementation that raises ModuleNotFoundError just like the current line does. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Nov 29 13:37:50 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Nov 2016 18:37:50 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Tue, 29 Nov 2016 at 10:28 Nick Timkovich wrote: > I would consider the speed of the "ultimate error handler" (i.e. whatever > prints the traceback and kills the program) in the interpreter to be moot, > so long as it takes a small fraction of a second. Optimizing Python's speed > it crashes super-fast due to an *unhandled* NameError in your program seems > folly. > So the performance worry isn't the traceback printer once the call stack fully unwinds but the construction of the exception instance itself. E.g. having to construct an expensive string for every instance of AttributeError is more the worry than printing a traceback. > > Regarding more informative messages for (e.g.) IndexError, would those > just apply to built-in types as they're the most universal, or should some > additional introspection be done for similar ducks? > > If it's useful/there's interest, I could try to do some analysis of Python > questions on SO and see what the most common errors are. I'd guess things > like "'NoneType' object has no attribute ..." would probably be up there, > but that's a whole can of worms as to why someone's trying to call a method > on, index, etc. something they accidentally whacked (a = a.sort()). > I suspect if we decide to try and go with more informative messages with a solution people are happy with then having an idea of where people get tripped up regularly will be good to help focus the work. And as for the NoneType issue specifically, there's an issue for that: http://bugs.python.org/issue28702 . > > On Tue, Nov 29, 2016 at 11:42 AM, Chris Barker > wrote: > > On Tue, Nov 29, 2016 at 5:48 AM, Nick Coghlan wrote: > > > SyntaxErrors in an inner loop? That seems unlikely to me. > > > Syntax Errors are a special case, as by definition the code isn't being > run yet (yes, there could be an eval in there...) > > So we could at least make those more informative without worrying about > performance. > > Also -- would it be possible to tack on the more informative message at a > higher level? Once the Exception bubbles up to the REPL, is there enough > information available to make a more informative message? > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Tue Nov 29 14:45:58 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 29 Nov 2016 11:45:58 -0800 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: <583DDAF6.9040104@brenbarn.net> On 2016-11-29 09:43, Brett Cannon wrote: > One way to make this cheap is to have a reasonable default message and > use attributes on the exceptions trigger the use of the default message. > Nearly a year ago I filed a bunch of issues for ideas on providing > attributes on exceptions where it made sense, e.g. an index attribute on > IndexError (http://bugs.python.org/issue18162). If we did this then for > classes like IndexError there constructor could be `IndexError(index=10, > start=0, end=3)` and then __str__() can lazily construct the string > representation using a default message, e.g. `"index {} is out of range > of{} to {}".format(index, start, end)`. Make the arguments keyword-only > and they become backwards-compatible and so the only overhead you pay > for these richer messages are keyword-based construction if you simply > never access the repr for the exception. I absolutely think this is the way to go. Having the relevant information (the list that was too short, the index that was too big, the key that wasn't there, etc.) is useful in many situations, and it's much better to have that information in a programmatic form than just squashed into an error message. This then makes it relatively easy to write wrappers that take bubbling-up exceptions and try to construct more detailed messages for a less experienced audience. Right now this is difficult or impossible because the exception objects don't record the information that would be needed for these expanded messages. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From tjreedy at udel.edu Tue Nov 29 15:09:19 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 29 Nov 2016 15:09:19 -0500 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On 11/29/2016 11:32 AM, Rob Cliffe wrote: > > > On 29/11/2016 04:58, victor rajewski wrote: >> >> Traceback (most recent call last): >> >> File "foo.py", line 2, in >> >> l[10]=14 >> >> IndexError: list assignment index out of range >> >> >> A better message might be: >> >> You tried to use l[10] when l is only 4 elements long. You can add >> items to l using l.append(value), or check your index value to make >> sure that's really the position you wanted to access. >> >> >> > It would make sense to me to upgrade this particular error message to > IndexError: list assignment index 10 out of range 0 to 3 if it can > be done without too much difficulty or overhead. (An empty list, and > perhaps one with only 1 element, would be special cases.) Come to think > of it, is the word "assignment" needed? It would help if the line were "l1[10] = 2 * l2[13] + 3". -- Terry Jan Reedy From njs at pobox.com Tue Nov 29 15:23:11 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Nov 2016 12:23:11 -0800 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On Nov 29, 2016 9:43 AM, "Brett Cannon" wrote: > > > > On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith wrote: >> >> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner >> wrote: >> > Hi, >> > >> > Python is optimized for performance. Formatting an error message has a >> > cost on performances. >> >> Sure, but we have to look at this on a case-by-case basis. Is there >> really important code out there that's generating NameErrors or >> SyntaxErrors in an inner loop? That seems unlikely to me. >> >> Even IndexError I'm a bit skeptical about. I can believe that there's >> code that intentionally generates and then catches IndexError, but >> AttributeError in my experience is much more performance-sensitive >> than IndexError, because every failed hasattr call allocates an >> AttributeError and hasattr is commonly used for feature checks. Yet >> AttributeError has a much more informative (= expensive) message than >> IndexError: >> >> In [1]: object().a >> AttributeError: 'object' object has no attribute 'a' >> >> In [2]: list()[0] >> IndexError: list index out of range > > > One way to make this cheap is to have a reasonable default message and use attributes on the exceptions trigger the use of the default message. Nearly a year ago I filed a bunch of issues for ideas on providing attributes on exceptions where it made sense, e.g. an index attribute on IndexError (http://bugs.python.org/issue18162). If we did this then for classes like IndexError there constructor could be `IndexError(index=10, start=0, end=3)` and then __str__() can lazily construct the string representation using a default message, e.g. `"index {} is out of range of{} to {}".format(index, start, end)`. Make the arguments keyword-only and they become backwards-compatible and so the only overhead you pay for these richer messages are keyword-based construction if you simply never access the repr for the exception. It seems like this might need some care, though, to make sure that these extra attributes don't end up pinning objects in memory that shouldn't be? Actually I always assumed that was why AttributeError's message was constructed eagerly... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Nov 29 15:28:30 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 29 Nov 2016 20:28:30 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: <583DDAF6.9040104@brenbarn.net> References: <583DDAF6.9040104@brenbarn.net> Message-ID: On 2016-11-29 19:45, Brendan Barnwell wrote: > On 2016-11-29 09:43, Brett Cannon wrote: >> One way to make this cheap is to have a reasonable default message and >> use attributes on the exceptions trigger the use of the default message. >> Nearly a year ago I filed a bunch of issues for ideas on providing >> attributes on exceptions where it made sense, e.g. an index attribute on >> IndexError (http://bugs.python.org/issue18162). If we did this then for >> classes like IndexError there constructor could be `IndexError(index=10, >> start=0, end=3)` and then __str__() can lazily construct the string >> representation using a default message, e.g. `"index {} is out of range >> of{} to {}".format(index, start, end)`. Make the arguments keyword-only >> and they become backwards-compatible and so the only overhead you pay >> for these richer messages are keyword-based construction if you simply >> never access the repr for the exception. > > I absolutely think this is the way to go. Having the relevant > information (the list that was too short, the index that was too big, > the key that wasn't there, etc.) is useful in many situations, and it's > much better to have that information in a programmatic form than just > squashed into an error message. This then makes it relatively easy to > write wrappers that take bubbling-up exceptions and try to construct > more detailed messages for a less experienced audience. Right now this > is difficult or impossible because the exception objects don't record > the information that would be needed for these expanded messages. > Couldn't that result in objects being held for longer, taking up memory, not being collected as promptly, and not releasing resources as quickly? From yselivanov.ml at gmail.com Tue Nov 29 17:36:33 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 29 Nov 2016 17:36:33 -0500 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: <060f046d-c1f7-b253-0004-2c43f85cdbde@gmail.com> I actually thought about adding 'Exception.__hint__' attribute which would have a longer message explaining what happened and how to fix it. displayhooks can be easily modified to print __hint__ when it's set. We can then add C API to set hints. To address any possible performance concerns, we can disable hints in -O mode (essentially we can make them almost zero-cost). Yury On 2016-11-29 12:43 PM, Brett Cannon wrote: > On Tue, 29 Nov 2016 at 02:39 Nathaniel Smith wrote: > >> On Tue, Nov 29, 2016 at 1:05 AM, Victor Stinner >> wrote: >>> Hi, >>> >>> Python is optimized for performance. Formatting an error message has a >>> cost on performances. >> Sure, but we have to look at this on a case-by-case basis. Is there >> really important code out there that's generating NameErrors or >> SyntaxErrors in an inner loop? That seems unlikely to me. >> >> Even IndexError I'm a bit skeptical about. I can believe that there's >> code that intentionally generates and then catches IndexError, but >> AttributeError in my experience is much more performance-sensitive >> than IndexError, because every failed hasattr call allocates an >> AttributeError and hasattr is commonly used for feature checks. Yet >> AttributeError has a much more informative (= expensive) message than >> IndexError: >> >> In [1]: object().a >> AttributeError: 'object' object has no attribute 'a' >> >> In [2]: list()[0] >> IndexError: list index out of range >> > One way to make this cheap is to have a reasonable default message and use > attributes on the exceptions trigger the use of the default message. Nearly > a year ago I filed a bunch of issues for ideas on providing attributes on > exceptions where it made sense, e.g. an index attribute on IndexError ( > http://bugs.python.org/issue18162). If we did this then for classes like > IndexError there constructor could be `IndexError(index=10, start=0, > end=3)` and then __str__() can lazily construct the string representation > using a default message, e.g. `"index {} is out of range of{} to > {}".format(index, start, end)`. Make the arguments keyword-only and they > become backwards-compatible and so the only overhead you pay for these > richer messages are keyword-based construction if you simply never access > the repr for the exception. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From eric at trueblade.com Tue Nov 29 19:14:39 2016 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 29 Nov 2016 19:14:39 -0500 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: <26ce5724-ac48-4ee6-161b-b2395247ffc3@trueblade.com> On 11/29/2016 1:33 PM, Brett Cannon wrote: > > > On Tue, 29 Nov 2016 at 06:49 Nick Coghlan > wrote: > > On 29 November 2016 at 20:54, Tomas Orsava > wrote: > > With a metapath hook, .missing.py files are probably overkill, and > the hook > > can just look at one file (or a static compiled-in list) of > > ModuleNotFound/ImportError messages for all missing modules, as M.-A. > > Lemburg and others are suggesting. We'll just need to think about > > coordinating how the list is generated/updated: the current PEP > implicitly > > allows other parties, besides Python and the distributors, to step in > > cleanly if they need to?needing to update a single list could lead > to messy > > hacks. > > What if, rather than using an explicitly file-based solution, this was > instead defined as a new protocol module, where the new metapath hook > imported a "__missing__" module and called a particular function in it > (e.g. "__missing__.module_not_found(modname)")? > > > You can answer this question the best, Nick, but would it be worth > defining a _stdlib.py that acts as both a marker for where the stdlib is > installed -- instead of os.py which is the current marker -- and which > also stores metadata like an attribute called `missing` which is a dict > that maps modules to ModuleNotFoundError messages? Although maybe this > is too specific of a solution (or still too general and we use an e.g. > missing.json off of sys.path which contains the same mapping). > > Otherwise MAL touched on the solution I always had in the back of my > head where we let people register a callback that gets passed the name > of any module that wasn't found through sys.meta_path. We could either > have the return value mean nothing and by default raise > ModuleNotFoundError, have the return value be what to set the module to > and raise an exception as expected, or have it be more error-specific > and return an exception to raise (all of these options also ask whether > a default callback doing what is normal is provided or if it's None by > default and import continues to provide the default semantics). The perk > of the callback is it removes the order sensitivity of any sys.meta_path > or sys.path_hooks solution where people might be doing > sys.meta_path.append(custom_finder) and thus won't necessarily trigger > if a new hook for missing modules is put at the end by default. How about having a sys.meta_path_last_chance (or whatever), which is identical to anything else on sys.meta_path, but is always guaranteed to be called last. Then instead of: meta_path = sys.meta_path it would be: meta_path = sys.meta_path + ([sys.meta_path_last_chance] if sys.meta_path_last_chance else []) There's no need to invent a new callback signature, or any new logic anywhere: it's literally just another metapath importer, but always guaranteed to be last. Eric. From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 29 21:14:56 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 30 Nov 2016 11:14:56 +0900 Subject: [Python-ideas] Better error messages [was: (no subject)] In-Reply-To: References: Message-ID: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> Mariatta Wijaya writes: > > NameError: name 'length' is not defined > > > A better message might be: > > > Python doesn't recognise the function "length". Did you mean > > len?' This particular change would be useful to a beginning Python programmer. I've made that error often enough myself (maybe I'm not a good example, though, in Lisp the generic function with the same role *is* called "length", so I make that error even today). But I wonder if it's a good example for the generic case. An error message of the form 'Python doesn't recognize the function "". Did you mean ""?' could easily be misleading. Python has functions, but they're a type of object. "length" is a name, which is not an object. The expected type of the object is not function, it's callable. So consider: class Blog: pass blog = log() NameError: Python doesn't recognize the function "log". Did you mean "Blog"? I suspect that might cause cognitive dissonance in a beginner who thinks of a class as a type or data structure and not as a function. And "doesn't recognize the callable" might be awkward (or beginners might get used to it very quickly, I don't know). Also, how do you propose deciding on the alternative to suggest? In this particular case, I expect most developers would agree intuitively. But the Hamming distance is pretty large: 3 of a length of 6. Would you have a dictionary of common errors, and then scan the namespace for minimum Hamming distance among defined names with the right type of value? How about: class Blog: pass blog = get_blog_for_date(someday) logn = log(blog.size) NameError: Python doesn't recognize the function "log". Did you mean "Blog"? Wouldn't NameError: Python doesn't recognize the name "log". Perhaps you need to import the "math" module? be a better message here? On second thought, that might imply that calling with the unqualified name is generally the best style, and teach the beginner to insert from math import * at the top of the module, thus fixing all such errors. We probably don't want that, so maybe NameError: Python doesn't recognize the name "log". There are functions named "log" in the "math" module and the "cmath" module. would be better yet. I definitely agree that there are times when Python's error messages are quite impenetrable for the beginner, and improvement is desirable. I think that I would probably attack this by looking at the builtin namespace and a few stdlib namespaces (math, string, and maybe cmath come immediately to mind), and create a dictionary of "intuitive beginner errors". Then trap NameErrors, and preferentially emit the message from the dictionary on exact matches. Actually, come to think of it, two dictionaries, one for the builtin namespace, one for the selected stdlib, and the following heuristic: if name in common_builtin_typos: emit(common_builtin_typos[name]) else: errors = small_hamming_distance(name, current_namespace, syntax(name)) if errors: emit(errors) else: errors = exact_matches_in_imported_modules(name) if errors: emit(errors) elif name in common_unimported_stdlib_names: emit(common_unimported_stdlib_names[name]) else: emit(error_you_would_have_emitted_anyway) In other words, I don't see a good systematic way to go about this, just pile up heuristics (and maybe remove some as they prove unuseful!) Steve From ncoghlan at gmail.com Tue Nov 29 21:56:19 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Nov 2016 12:56:19 +1000 Subject: [Python-ideas] PEP: Distributing a Subset of the Standard Library In-Reply-To: References: <6e27a05d-6a02-44f0-fa3f-4c14b9e1befc@redhat.com> Message-ID: On 30 November 2016 at 04:33, Brett Cannon wrote: > On Tue, 29 Nov 2016 at 06:49 Nick Coghlan wrote: >> >> On 29 November 2016 at 20:54, Tomas Orsava wrote: >> > With a metapath hook, .missing.py files are probably overkill, and the >> > hook >> > can just look at one file (or a static compiled-in list) of >> > ModuleNotFound/ImportError messages for all missing modules, as M.-A. >> > Lemburg and others are suggesting. We'll just need to think about >> > coordinating how the list is generated/updated: the current PEP >> > implicitly >> > allows other parties, besides Python and the distributors, to step in >> > cleanly if they need to?needing to update a single list could lead to >> > messy >> > hacks. >> >> What if, rather than using an explicitly file-based solution, this was >> instead defined as a new protocol module, where the new metapath hook >> imported a "__missing__" module and called a particular function in it >> (e.g. "__missing__.module_not_found(modname)")? > > > You can answer this question the best, Nick, but would it be worth defining > a _stdlib.py that acts as both a marker for where the stdlib is installed -- > instead of os.py which is the current marker -- and which also stores > metadata like an attribute called `missing` which is a dict that maps > modules to ModuleNotFoundError messages? Although maybe this is too specific > of a solution (or still too general and we use an e.g. missing.json off of > sys.path which contains the same mapping). Really, I think the ideal solution from a distro perspective would be to enable something closer to what bash and other shells support for failed CLI calls: $ blender bash: blender: command not found... Install package 'blender' to provide command 'blender'? [N/y] n This would allow redistributors to point folks towards platform packages (via apt/yum/dnf/PyPM/conda/Canopy/etc) for the components they provide, and towards pip/PyPI for everything else (and while we don't have a dist-lookup-by-module-name service for PyPI *today*, it's something I hope we'll find a way to provide sometime in the next few years). I didn't suggest that during the Fedora-level discussions of this PEP because it didn't occur to me - the elegant simplicity of the new import suffix as a tactical solution to the immediate "splitting the standard library" problem [1] meant I missed that it was really a special case of the general "provide guidance on obtaining missing modules from the system package manager" concept. The problem with that idea however is that while it provides the best possible interactive user experience, it's potentially really slow, and hence too expensive to do for every import error - we would instead need to find a way to run with Wolfgang Maier's suggestion of only doing this for *unhandled* import errors. Fortunately, we do have the appropriate mechanisms in place to support that approach: 1. For interactive use, we have sys.excepthook 2. For non-interactive use, we have the atexit module As a simple example of the former: >>> def module_missing(modname): ... return f"Module not found: {modname}" >>> def my_except_hook(exc_type, exc_value, exc_tb): ... if isinstance(exc_value, ModuleNotFoundError): ... print(module_missing(exc_value.name)) ... >>> sys.excepthook = my_except_hook >>> import foo Module not found: foo >>> import foo.bar Module not found: foo >>> import sys.bar Module not found: sys.bar For the atexit handler, that could be installed by the `site` module, so the existing mechanisms for disabling site module processing would also disable any default exception reporting hooks. Folks could also register their own handlers via either `sitecustomize.py` or `usercustomize.py`. And at that point the problem starts looking less like "Customise the handling of missing modules" and more like "Customise the rendering and reporting of particular types of unhandled exceptions". For example, a custom handler for subprocess.CalledProcessError could introspect the original command and use `shutil.which` to see if the requested command was even visible from the current process (and, in a redistributor provided Python, indicate which system packages to install to obtain the requested command). > My personal vote is a callback called at > https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L948 > with a default implementation that raises ModuleNotFoundError just like the > current line does. Ethan's observation about try/except import chains has got me think that limiting this to handling errors within the context of single import statement will be problematic, especially given that folks can already write their own metapath hook for that case if they really want to. Cheers, Nick. [1] For folks wondering "This problem has existed for years, why suddenly worry about it now?", Fedora's in the process of splitting out an even more restricted subset of the standard library for system tools to use: https://fedoraproject.org/wiki/Changes/System_Python That means "You're relying on a missing stdlib module" is going to come up more often for system tools developers trying to stick within that restricted subset. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bussonniermatthias at gmail.com Tue Nov 29 23:43:15 2016 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Tue, 29 Nov 2016 20:43:15 -0800 Subject: [Python-ideas] Better error messages [was: (no subject)] In-Reply-To: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> References: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> Message-ID: There are a couple of project that tried to improved heuristic on some error messages. Two I can think off are: https://github.com/SylvainDe/DidYouMean-Python and https://github.com/dutc/didyoumean I think that better error messages could be implemented only in the repl, and/or by alternative repl. Having it as an opt-in library would be useful to tweak the messages in various context, for example when used for teaching, or if there are concern about overhead. Some alternative even already implement heuristic (Recursions error in IPython are elided with "... last frames repeated, from the frame below ..." which is way less scarier for beginners. Some other attempts have been made with attributes error[1] (similar to what Stephen is proposing), though the traceback info does not have a ref (or even weakref) on the object on which the attributes access is made, so even more heuristic have to be made. Maybe just adding some more information on exception themselves and giving the recipe of custom except hook would be enough. Then just allow users to pip-install packages with heuristics ? -- M [1] https://github.com/ipython/ipython/pull/9073 On Tue, Nov 29, 2016 at 6:14 PM, Stephen J. Turnbull wrote: > Mariatta Wijaya writes: > > > > NameError: name 'length' is not defined > > > > > A better message might be: > > > > > Python doesn't recognise the function "length". Did you mean > > > len?' > > This particular change would be useful to a beginning Python > programmer. I've made that error often enough myself (maybe I'm not a > good example, though, in Lisp the generic function with the same role > *is* called "length", so I make that error even today). > > But I wonder if it's a good example for the generic case. An error > message of the form 'Python doesn't recognize the function "". > Did you mean ""?' could easily be misleading. Python has > functions, but they're a type of object. "length" is a name, which is > not an object. The expected type of the object is not function, it's > callable. So consider: > > class Blog: > pass > > blog = log() > > NameError: Python doesn't recognize the function "log". Did you > mean "Blog"? > > I suspect that might cause cognitive dissonance in a beginner who > thinks of a class as a type or data structure and not as a function. > And "doesn't recognize the callable" might be awkward (or beginners > might get used to it very quickly, I don't know). > > Also, how do you propose deciding on the alternative to suggest? In > this particular case, I expect most developers would agree > intuitively. But the Hamming distance is pretty large: 3 of a length > of 6. Would you have a dictionary of common errors, and then scan the > namespace for minimum Hamming distance among defined names with the > right type of value? > > How about: > > class Blog: > pass > > blog = get_blog_for_date(someday) > > logn = log(blog.size) > > NameError: Python doesn't recognize the function "log". Did you > mean "Blog"? > > Wouldn't > > NameError: Python doesn't recognize the name "log". Perhaps > you need to import the "math" module? > > be a better message here? On second thought, that might imply that > calling with the unqualified name is generally the best style, and > teach the beginner to insert > > from math import * > > at the top of the module, thus fixing all such errors. We probably > don't want that, so maybe > > NameError: Python doesn't recognize the name "log". There are > functions named "log" in the "math" module and the "cmath" module. > > would be better yet. > > I definitely agree that there are times when Python's error messages > are quite impenetrable for the beginner, and improvement is > desirable. I think that I would probably attack this by looking at > the builtin namespace and a few stdlib namespaces (math, string, and > maybe cmath come immediately to mind), and create a dictionary of > "intuitive beginner errors". Then trap NameErrors, and preferentially > emit the message from the dictionary on exact matches. Actually, come > to think of it, two dictionaries, one for the builtin namespace, one > for the selected stdlib, and the following heuristic: > > if name in common_builtin_typos: > emit(common_builtin_typos[name]) > else: > errors = small_hamming_distance(name, current_namespace, syntax(name)) > if errors: > emit(errors) > else: > errors = exact_matches_in_imported_modules(name) > if errors: > emit(errors) > elif name in common_unimported_stdlib_names: > emit(common_unimported_stdlib_names[name]) > else: > emit(error_you_would_have_emitted_anyway) > > In other words, I don't see a good systematic way to go about this, > just pile up heuristics (and maybe remove some as they prove > unuseful!) > > Steve > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 30 00:27:18 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 30 Nov 2016 14:27:18 +0900 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: <22590.25398.890365.276054@turnbull.sk.tsukuba.ac.jp> Victor Stinner writes: > Using a custom exception handler, you can run expensive functions, > like the feature: "suggest len when length is used". LGTM. > The problem is then when students have to use a Python without the > custom exception handler. Put the exception handler in an importable module in the stdlib. Usual caveats about high bar, etc, but this should solve the "have Python, no handler" issue going forward. Steve From p.f.moore at gmail.com Wed Nov 30 04:05:37 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Nov 2016 09:05:37 +0000 Subject: [Python-ideas] Better error messages [was: (no subject)] In-Reply-To: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> References: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> Message-ID: On 30 November 2016 at 02:14, Stephen J. Turnbull wrote: > How about: > > class Blog: > pass > > blog = get_blog_for_date(someday) > > logn = log(blog.size) > > NameError: Python doesn't recognize the function "log". Did you > mean "Blog"? > > Wouldn't > > NameError: Python doesn't recognize the name "log". Perhaps > you need to import the "math" module? ... and of course up until this example, I'd assumed you were talking about the log function from the logging module :-) I'm a strong +1 on better error messages, but there's always a risk with heuristics that the resulting messages end up worse, not better. Maybe keep it simpler: NameError: Python doesn't recognize the name "log". Maybe you misspelled the name, or did you mean to import the function from a module? and don't try to guess the user's intent. Paul From me at jeltef.nl Wed Nov 30 05:32:48 2016 From: me at jeltef.nl (Jelte Fennema) Date: Wed, 30 Nov 2016 02:32:48 -0800 (PST) Subject: [Python-ideas] Add optional defaults to namedtuple Message-ID: It would be nice to have a supported way to add defaults to namedtuple, so the slightly hacky solution here does not have to be used: http://stackoverflow.com/a/18348004/2570866 Jelte -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed Nov 30 06:44:11 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 30 Nov 2016 11:44:11 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: Message-ID: On 29/11/2016 20:09, Terry Reedy wrote: > On 11/29/2016 11:32 AM, Rob Cliffe wrote: >> >> >> On 29/11/2016 04:58, victor rajewski wrote: >>> >>> Traceback (most recent call last): >>> >>> File "foo.py", line 2, in >>> >>> l[10]=14 >>> >>> IndexError: list assignment index out of range >>> >>> >>> A better message might be: >>> >>> You tried to use l[10] when l is only 4 elements long. You can add >>> items to l using l.append(value), or check your index value to make >>> sure that's really the position you wanted to access. >>> >>> >>> >> It would make sense to me to upgrade this particular error message to >> IndexError: list assignment index 10 out of range 0 to 3 if it can >> be done without too much difficulty or overhead. (An empty list, and >> perhaps one with only 1 element, would be special cases.) Come to think >> of it, is the word "assignment" needed? > > It would help if the line were "l1[10] = 2 * l2[13] + 3". > You're right of course; "assignment" IS meaningful; I missed it. >>> [][0] Traceback (most recent call last): File "", line 1, in IndexError: list index out of range From ncoghlan at gmail.com Wed Nov 30 07:43:39 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 Nov 2016 22:43:39 +1000 Subject: [Python-ideas] Better error messages [was: (no subject)] In-Reply-To: References: <22590.13856.162202.818428@turnbull.sk.tsukuba.ac.jp> Message-ID: On 30 November 2016 at 19:05, Paul Moore wrote: > On 30 November 2016 at 02:14, Stephen J. Turnbull > wrote: >> Wouldn't >> >> NameError: Python doesn't recognize the name "log". Perhaps >> you need to import the "math" module? > > ... and of course up until this example, I'd assumed you were talking > about the log function from the logging module :-) > > I'm a strong +1 on better error messages, but there's always a risk > with heuristics that the resulting messages end up worse, not better. > > Maybe keep it simpler: > > NameError: Python doesn't recognize the name "log". Maybe you > misspelled the name, or did you mean to import the function from a > module? > > and don't try to guess the user's intent. This brings up a point that I was going to mention earlier: when it comes to specialised learning environments, the folks developing the curriculum also know *what problem the student is working on*, and can tailor their error messages accordingly. The reference interpreter is never going to be able to guess intent like that due to the sheer scope of Python's use cases - by covering everything from students writing "Guess a number" games to mechanical engineers modeling and tuning race car performance to sysadmins automating service deployments to web developers responding to user requests to data analysts trying to make sense of noisy data, we end up being *really* limited in the assumptions we can make about what a user was really trying to do when they accidentally ask for something nonsensical. Tweaking some of the default representations to mention common problems should be OK, though. While some veteran programmers may find such prompts a bit condescending, they'd be better equipped than beginners to opt in to alternative exception display options that omit the hints. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vince.vinet at gmail.com Wed Nov 30 08:38:53 2016 From: vince.vinet at gmail.com (Vince Vinet) Date: Wed, 30 Nov 2016 08:38:53 -0500 Subject: [Python-ideas] Decorator to avoid a mistake In-Reply-To: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> References: <5834b386.45f6c20a.f25f.f1ad@mx.google.com> Message-ID: Hello, While I think this should not be "on by default", I don't see the harm in being able to opt-in to this behavior. I also figured spending a few minutes attempting to write this would be fun: https://gist.github.com/veloutin/2ec3e5246651f5de78442516d8e24fc1 Fran?ois: sorry about the double reply, I forgot to reply to the list. Vince -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 30 10:09:48 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Nov 2016 07:09:48 -0800 Subject: [Python-ideas] Add optional defaults to namedtuple In-Reply-To: References: Message-ID: <583EEBBC.2050206@stoneleaf.us> On 11/30/2016 02:32 AM, Jelte Fennema wrote: > It would be nice to have a supported way to add defaults to namedtuple, > so the slightly hacky solution here does not have to be used: > http://stackoverflow.com/a/18348004/2570866 Actually, the solution right below it is better [1]: --> from collections import namedtuple --> class Node(namedtuple('Node', ['value', 'left', 'right'])): --> __slots__ = () --> def __new__(cls, value, left=None, right=None): --> return super(Node, cls).__new__(cls, value, left, right) But even more readable than that is using the NamedTuple class from my aenum [3] library (and on SO as [3]): --> from aenum import NamedTuple --> class Node(NamedTuple): --> val = 0 --> left = 1, 'previous Node', None --> right = 2, 'next Node', None shamelessly-plugging-my-own-solutions'ly yrs, -- ~Ethan~ [1] http://stackoverflow.com/a/16721002/208880 [2] https://pypi.python.org/pypi/aenum [3] http://stackoverflow.com/a/40891597/208880 From guido at python.org Wed Nov 30 11:11:52 2016 From: guido at python.org (Guido van Rossum) Date: Wed, 30 Nov 2016 08:11:52 -0800 Subject: [Python-ideas] Add optional defaults to namedtuple In-Reply-To: <583EEBBC.2050206@stoneleaf.us> References: <583EEBBC.2050206@stoneleaf.us> Message-ID: On Wed, Nov 30, 2016 at 7:09 AM, Ethan Furman wrote: > On 11/30/2016 02:32 AM, Jelte Fennema wrote: > > It would be nice to have a supported way to add defaults to namedtuple, >> so the slightly hacky solution here does not have to be used: >> http://stackoverflow.com/a/18348004/2570866 >> > > Actually, the solution right below it is better [1]: > > --> from collections import namedtuple > --> class Node(namedtuple('Node', ['value', 'left', 'right'])): > --> __slots__ = () > --> def __new__(cls, value, left=None, right=None): > --> return super(Node, cls).__new__(cls, value, left, right) > > But even more readable than that is using the NamedTuple class from my > aenum [3] library (and on SO as [3]): > > --> from aenum import NamedTuple > --> class Node(NamedTuple): > --> val = 0 > --> left = 1, 'previous Node', None > --> right = 2, 'next Node', None > > shamelessly-plugging-my-own-solutions'ly yrs, > Ditto: with PEP 526 and the latest typing.py (in 3.6) you will be able to do this: class Employee(NamedTuple): name: str id: int We should make it so that the initial value in the class is used as the default value, too. (Sorry, this syntax still has no room for a docstring per attribute.) -- --Guido van Rossum (python.org/~guido ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Nov 30 12:57:22 2016 From: brett at python.org (Brett Cannon) Date: Wed, 30 Nov 2016 17:57:22 +0000 Subject: [Python-ideas] (no subject) In-Reply-To: References: <583DDAF6.9040104@brenbarn.net> Message-ID: On Tue, 29 Nov 2016 at 12:32 MRAB wrote: > On 2016-11-29 19:45, Brendan Barnwell wrote: > > On 2016-11-29 09:43, Brett Cannon wrote: > >> One way to make this cheap is to have a reasonable default message and > >> use attributes on the exceptions trigger the use of the default message. > >> Nearly a year ago I filed a bunch of issues for ideas on providing > >> attributes on exceptions where it made sense, e.g. an index attribute on > >> IndexError (http://bugs.python.org/issue18162). If we did this then for > >> classes like IndexError there constructor could be `IndexError(index=10, > >> start=0, end=3)` and then __str__() can lazily construct the string > >> representation using a default message, e.g. `"index {} is out of range > >> of{} to {}".format(index, start, end)`. Make the arguments keyword-only > >> and they become backwards-compatible and so the only overhead you pay > >> for these richer messages are keyword-based construction if you simply > >> never access the repr for the exception. > > > > I absolutely think this is the way to go. Having the relevant > > information (the list that was too short, the index that was too big, > > the key that wasn't there, etc.) is useful in many situations, and it's > > much better to have that information in a programmatic form than just > > squashed into an error message. This then makes it relatively easy to > > write wrappers that take bubbling-up exceptions and try to construct > > more detailed messages for a less experienced audience. Right now this > > is difficult or impossible because the exception objects don't record > > the information that would be needed for these expanded messages. > > > Couldn't that result in objects being held for longer, taking up memory, > not being collected as promptly, and not releasing resources as quickly? > Sure, just like passing any other object into BaseException's initializer so it gets stored in the args attribute. Notice how my example only used ints, and that was on purpose. If you want to only pass in the repr of the type for the message, then simply enforce that or at least encourage it to prevent/discourage people from using whole objects instead of ints and strings which are cheaper than the whole string message that is currently constructed eagerly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Nov 30 14:25:59 2016 From: random832 at fastmail.com (Random832) Date: Wed, 30 Nov 2016 14:25:59 -0500 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators Message-ID: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> Currently these functions fail if the supplied object has no len(). There are algorithms for this task that can work on any finite iterator (for example, files as a stream of lines), and the functions could fall back to these if there is no len(). From jsbueno at python.org.br Wed Nov 30 14:27:38 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 30 Nov 2016 17:27:38 -0200 Subject: [Python-ideas] Add optional defaults to namedtuple In-Reply-To: <583EEBBC.2050206@stoneleaf.us> References: <583EEBBC.2050206@stoneleaf.us> Message-ID: On 30 November 2016 at 13:09, Ethan Furman wrote: > But even more readable than that is using the NamedTuple class from my aenum > [3] library (and on SO as [3]): > > --> from aenum import NamedTuple > --> class Node(NamedTuple): > --> val = 0 > --> left = 1, 'previous Node', None > --> right = 2, 'next Node', None > > shamelessly-plugging-my-own-solutions'ly yrs, Sorry - taking the boat to even-more-shamelessly anounce extradict.extratuple.defaultnamedtuple - in the newly released extradict v. 0.2.5 https://pypi.python.org/pypi/extradict/0.2.5 It allows one to build an default-paremetrized namedtuple by passing a sequence of 2-tuples with key, values, or, on Python 3.6, pass in the default values as keywords to the defaultnamedtuple factory. (The "extradict" package, with a faster reimplementation of namedtuple already existed, of course - maybe someone can pick some other weird idea I have into there to put it into more day-to-day use) > -- > ~Ethan~ From mafagafogigante at gmail.com Wed Nov 30 14:42:01 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Wed, 30 Nov 2016 17:42:01 -0200 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> Message-ID: <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> On 2016-11-30 17:25, Random832 wrote: > Currently these functions fail if the supplied object has no len(). > There are algorithms for this task that can work on any finite iterator > (for example, files as a stream of lines), and the functions could fall > back to these if there is no len(). I like the idea, as long as it does not add too much overhead to currently existing code. It could be a special code path for reservoir sampling (I assume) for both functions (the first taking only one sample from the stream). -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From ckaynor at zindagigames.com Wed Nov 30 14:52:51 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 30 Nov 2016 11:52:51 -0800 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> Message-ID: This was also brought up back in April: https://mail.python.org/pipermail//python-ideas/2016-April/039707.html It got a few replies from Guido (https://mail.python.org/pipermail//python-ideas/2016-April/039713.html for one of them). It seems the idea got dropped due to problems with making it properly random (practically, the sampling has issues in cases of very large sequences, let along infinite) and performance (either large numbers of calls to random, or copying the sequence, each of which has its own problems). There are also issues with how it should behave on iterables that cannot be re-iterated (eg, random.choice will consume the iterator, and could only be called once safely). Chris On Wed, Nov 30, 2016 at 11:42 AM, Bernardo Sulzbach wrote: > On 2016-11-30 17:25, Random832 wrote: >> >> Currently these functions fail if the supplied object has no len(). >> There are algorithms for this task that can work on any finite iterator >> (for example, files as a stream of lines), and the functions could fall >> back to these if there is no len(). > > > I like the idea, as long as it does not add too much overhead to currently > existing code. > > It could be a special code path for reservoir sampling (I assume) for both > functions (the first taking only one sample from the stream). > > > -- > Bernardo Sulzbach > http://www.mafagafogigante.org/ > mafagafogigante at gmail.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ckaynor at zindagigames.com Wed Nov 30 14:57:46 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 30 Nov 2016 11:57:46 -0800 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> Message-ID: On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor wrote: > There are also issues with how it should behave on iterables that > cannot be re-iterated (eg, random.choice will consume the iterator, > and could only be called once safely). I meant to include a sample in my previous e-mail: Consider that this code will not produce the "correct" results (for a reasonable definition of correct): a = (i for i in range(100)) # Pretend this does something more interesting, and isn't a trivial generator - maybe a file object reading by line. randomEntries = [random.choice(a) for i in range(10)] # May not be quite as obvious, such as the choices could be chosen in a more complex loop. randomEntries may not contain 10 items (or the generation may error due to having insufficent items, depending on implementation), it will not contain duplicates, and will be sorted. This occurs because the first call to random.choice will consume elements from a until it picks one. The next call then consumes from there until it picks one, and so forth. Chris From mafagafogigante at gmail.com Wed Nov 30 15:21:16 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Wed, 30 Nov 2016 18:21:16 -0200 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> Message-ID: <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> On 2016-11-30 17:57, Chris Kaynor wrote: > On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor wrote: >> There are also issues with how it should behave on iterables that >> cannot be re-iterated (eg, random.choice will consume the iterator, >> and could only be called once safely). > > I meant to include a sample in my previous e-mail: > > Consider that this code will not produce the "correct" results (for a > reasonable definition of correct): > > a = (i for i in range(100)) # Pretend this does something more > interesting, and isn't a trivial generator - maybe a file object > reading by line. > randomEntries = [random.choice(a) for i in range(10)] In such a case you should explicitly use a sample. I see your example as the caller's fault, which ignored the fact that the iterator would change after calls to choice. Hold the first 10 (in this case). For every subsequent element, randomly choose to replace one of the "held" ones by it (with a diminishing probability). Assume this does not offer the same performance as loading everything into memory. But it isn't meant to do so, as if you need / can / want, you could just shove it all into a list and use what we currently have. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From prometheus235 at gmail.com Wed Nov 30 15:32:54 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Wed, 30 Nov 2016 14:32:54 -0600 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> Message-ID: Is the goal to allow them to consume a finite generator of *unknown* length (requires reservoir sampling https://en.wikipedia.org/wiki/Reservoir_sampling with N random calls, which seemed to be the rub before?) or just consume a generator with known length that's not indexable (a rare beast?). Consuming iterables if they have a length like below wouldn't be so bad, but might be too niche. class X: def __init__(self, ele): self.ele = ele def __len__(self): return len(self.ele) def __iter__(self): return iter(self.ele) x = X([1, 2, 3, 4, 5]) random.choice(x) # TypeError: 'X' object does not support indexing Would allowing an optional 'len' argument alongside the iterator to sample/choice be too narrow to be useful? On Wed, Nov 30, 2016 at 2:21 PM, Bernardo Sulzbach < mafagafogigante at gmail.com> wrote: > On 2016-11-30 17:57, Chris Kaynor wrote: > >> On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor >> wrote: >> >>> There are also issues with how it should behave on iterables that >>> cannot be re-iterated (eg, random.choice will consume the iterator, >>> and could only be called once safely). >>> >> >> I meant to include a sample in my previous e-mail: >> >> Consider that this code will not produce the "correct" results (for a >> reasonable definition of correct): >> >> a = (i for i in range(100)) # Pretend this does something more >> interesting, and isn't a trivial generator - maybe a file object >> reading by line. >> randomEntries = [random.choice(a) for i in range(10)] >> > > In such a case you should explicitly use a sample. > > I see your example as the caller's fault, which ignored the fact that the > iterator would change after calls to choice. > > Hold the first 10 (in this case). For every subsequent element, randomly > choose to replace one of the "held" ones by it (with a diminishing > probability). > > Assume this does not offer the same performance as loading everything into > memory. But it isn't meant to do so, as if you need / can / want, you could > just shove it all into a list and use what we currently have. > > -- > Bernardo Sulzbach > http://www.mafagafogigante.org/ > mafagafogigante at gmail.com > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckaynor at zindagigames.com Wed Nov 30 16:11:56 2016 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Wed, 30 Nov 2016 13:11:56 -0800 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> Message-ID: On Wed, Nov 30, 2016 at 12:21 PM, Bernardo Sulzbach wrote: > On 2016-11-30 17:57, Chris Kaynor wrote: >> >> On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor >> wrote: >>> >>> There are also issues with how it should behave on iterables that >>> cannot be re-iterated (eg, random.choice will consume the iterator, >>> and could only be called once safely). >> >> >> I meant to include a sample in my previous e-mail: >> >> Consider that this code will not produce the "correct" results (for a >> reasonable definition of correct): >> >> a = (i for i in range(100)) # Pretend this does something more >> interesting, and isn't a trivial generator - maybe a file object >> reading by line. >> randomEntries = [random.choice(a) for i in range(10)] > > > In such a case you should explicitly use a sample. > > I see your example as the caller's fault, which ignored the fact that the > iterator would change after calls to choice. > > Hold the first 10 (in this case). For every subsequent element, randomly > choose to replace one of the "held" ones by it (with a diminishing > probability). > > Assume this does not offer the same performance as loading everything into > memory. But it isn't meant to do so, as if you need / can / want, you could > just shove it all into a list and use what we currently have. It would be the caller's fault: they failed to follow the documentation. That said, it would be a fairly subtle difference, and in many cases may go unnoticed in simple testing. There is a good probability that if you needed 10 samples out of 1000 entries, my code would appear to work correctly (I don't care to do the math to figure out the actual chance). Only on deeper testing or analysis of the results, would you notice that the probabilities are messed up. Most likely, my exact example would be fairly obvious, but imagine if the code were more complex, with the random.choice call inside a loop, or even another function. If random.choice were updated to use a reservoir sampling as a fallback, it also means that it is more likely for performance to degrade with some types. Giving a list (and maybe other sequences) would be O(1), but if a generator gets passed in, it falls back to O(n), and with lots of random.random calls that are generally naturally slow anyways. All that said, I would not be opposed to Python including a random.reservoir_choice (probably not the best name) function *in addition* to random.choice. The algorithm has its uses, but enough drawbacks and gotchas that it likely is not a good candidate for a fallback. From mafagafogigante at gmail.com Wed Nov 30 16:51:08 2016 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Wed, 30 Nov 2016 19:51:08 -0200 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> <73afd24e-a5c1-5646-2431-04a79a4937b9@gmail.com> Message-ID: On 2016-11-30 19:11, Chris Kaynor wrote: > > All that said, I would not be opposed to Python including a > random.reservoir_choice (probably not the best name) function *in > addition* to random.choice. The algorithm has its uses, but enough > drawbacks and gotchas that it likely is not a good candidate for a > fallback. I think this may be the path of least resistance for this. Even if this does imply one or two new functions in random, it may be better than changing random.choice. If these functions would be used enough by enough end users to justify this change is debatable, however. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From steve at pearwood.info Wed Nov 30 18:14:35 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Dec 2016 10:14:35 +1100 Subject: [Python-ideas] Allow random.choice, random.sample to work on iterators In-Reply-To: References: <1480533959.3740598.804127281.6028420C@webmail.messagingengine.com> <1406db4f-8b71-bbd2-de81-4b8328f4b143@gmail.com> Message-ID: <20161130231433.GP3365@ando.pearwood.info> On Wed, Nov 30, 2016 at 11:57:46AM -0800, Chris Kaynor wrote: > Consider that this code will not produce the "correct" results (for a > reasonable definition of correct): > > a = (i for i in range(100)) # Pretend this does something more > interesting, and isn't a trivial generator - maybe a file object > reading by line. > randomEntries = [random.choice(a) for i in range(10)] # May not be > quite as obvious, such as the choices could be chosen in a more > complex loop. > > randomEntries may not contain 10 items (or the generation may error > due to having insufficent items, depending on implementation), Indeed. Given a iterator with 100 items, there's a 9% chance that the first choice will be in the last nine items, which implies that failures here will be common. > it will > not contain duplicates, and will be sorted. Right. In other words, its a highly biased, non-random "random sample". It may be worth adding a "reservoir sample" function, but I think it is a mistake to try modifying the existing functions to deal with iterators. Its too hard to get the same characteristics when sampling from a sequence and an iterator. Better to keep them as separate functions so that the caller knows what they're getting. Here's my first attempt at implementing Algorithm R from Wikipedia: https://en.wikipedia.org/wiki/Reservoir_sampling#Algorithm_R from random import randrange, shuffle import itertools def reservoir_sample(iterable, count=1): """Return a list of count items sampled without replacement from iterable, using reservoir sampling "Algorithm R". This will exhaust the iterable, which must be finite. """ it = iter(iterable) reservoir = list(itertools.islice(it, count)) if len(reservoir) < count: raise ValueError('iterator is too short to sample %d items' % count) shuffle(reservoir) for i, value in enumerate(it, count+1): j = randrange(0, i) if j < count: reservoir[j] = value return reservoir -- Steve