Mailman 3 March 2017 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

dict(default=int)
by Steven Piantadosi May 30, 2017

May 30, 2017

Hi All, I find importing defaultdict from collections to be clunky and it seems like having a default should just be an optional keyword to dict. Thus, something like, d = dict(default=int) would be the same as from collections import defaultdict d = defaultdict(int) Any thoughts? Thanks, ++Steve

15 19

Proposal: making __str__ count in time's class
by Francesco Franchina April 9, 2017

April 9, 2017

Hello everyone, I'm shortly writing to you about a reflection I lately made upon the current functioning of __str__ for the time's class. Before expressing my thought and proposal, I want to make sure we all agree on a simple and clear fact: the __str__ magic method is used to give a literal and human-readable representation to the object (unlike __repr__). Generally this is true across the python panorama. It's not true for the time class, for example. *>>> import time>>> a = time.localtime()>>> a.__str__()'time.struct_time(tm_year=2017, tm_mon=3, tm_mday=8, tm_hour=16, tm_min=6, tm_sec=16, tm_wday=2, tm_yday=67, tm_isdst=0)'* Well, don't get me wrong: the main aim of the __str__ method has been accomplished but, imho, not in the most pythonic way. I just wanted to ask you: what do you think about re-writing the __str__ of the time class so it would return something like ISO 8601 [https://en.wikipedia.org/wiki/ISO_8601] format? Wouldn't it be more meaningful? Especially in the JS-everywhere-era it could be more more productive. *TL;DR* __str__ for dates should return a human-readable date format (eg: https://en.wikipedia.org/wiki/ISO_8601) I'm waiting for your opinions. Thank you for your time and ideas! Francesco Franchina

4 3

What about regexp string litterals : re".*" ?
by Simon D. April 3, 2017

April 3, 2017

Hello, After some french discussions about this idea, I subscribed here to suggest adding a new string litteral, for regexp, inspired by other types like : u"", r"", b"", br"", f""… The regexp string litteral could be represented by : re"" It would ease the use of regexps in Python, allowing to have some regexp litterals, like in Perl or JavaScript. We may end up with an integration like : >>> import re >>> if re".k" in 'ok': ... print "ok" ok >>> Regexps are part of the language in Perl, and the rather complicated integration of regexp in other languages, especially in Python, is something that comes up easily in language comparing discussion. I've always felt JavaScript integration being half the way it should, and new string litterals types in Python (like f"") looked like a good compromise to have a tight integration of regexps without asking to make them part of the language (as I imagine it has already been discussed years ago, and obviously denied…). As per XKCD illustration, using a regexp may be a problem on its own, but really, the "each-language a new and complicated approach" is another difficulty, of the level of writing regexps I think. And then, when you get the trick for Python, it feels to me still to much letters to type regarding the numerous problems one can solve using regexps. I know regexps are slower than string-based workflow (like .startswith) but regexps can do the most and the least, so they are rapide to come up with, once you started to think with them. As Python philosophy is to spare brain-cycles, sacrificing CPU-cycles, allowing to easily use regexps is a brain-cycle savior trick. What do you think ? -- Simon Descarpentries +336 769 702 53 http://acoeuro.com

14 24

Construct a matrix from a list: matrix multiplication
by Chris Angelico April 1, 2017

April 1, 2017

This keeps on coming up in one form or another - either someone multiplies a list of lists and ends up surprised that they're all the same, or is frustrated with the verbosity of the alternatives. Can we use the matmul operator for this? class List(list): def __matmul__(self, other): return [copy.copy(x) for x in self for _ in range(other)] >>> x = List([[0]*4]) @ 2 >>> x [[0, 0, 0, 0], [0, 0, 0, 0]] >>> x[0][0] = 1 >>> x [[1, 0, 0, 0], [0, 0, 0, 0]] If this were supported by the built-in list type, it would be either of these: >>> x = [[0] * 4] @ 2 >>> x = [[0] @ 4] @ 4 (identical functionality, as copying an integer has no effect). The semantics could be either as shown above (copy.copy()), or something very simple and narrow like "lists get shallow-copied, other objects get referenced". Thoughts? ChrisA

5 5

Re: [Python-ideas] unify usage of mutable and immutable objects
by 语言破碎处 April 1, 2017

April 1, 2017

> To me, 'pop' implies mutation. Tuples do not have a pop method, and it > is not obvious to me that either tuples or frozensets should. What are > the use cases that are not better served by converting to list or set? > -- > Terry Jan Reedy 1) coverting to set or list is O(n) in time 2) if I have to keep the old copy, standard set solution will be O(n) both in time and space! working examples: 1) priority queue: insert and pop occur 2) share immutable data to difference subroutines: each one can modify local copy safely and concurrency.

3 3

Way to repeat other than "for _ in range(x)"
by Markus Meskanen March 31, 2017

March 31, 2017

Hi Pythonistas, yet again today I ended up writing: d = [[0] * 5 for _ in range(10)] And wondered, why don't we have a way to repeat other than looping over range() and using a dummy variable? This seems like a rather common thing to do, and while the benefit doesn't seem much, something like this would be much prettier and more pythonic than using underscore variable: d = [[0] * 5 repeat_for 10] And could obviously be used outside of comprehensions too: repeat_for 3: print('Attempting to reconnect...') if reconnect(): break else: print('Unable to reconnect :(') sys.exit(0) I chose to use repeat_for instead of repeat because it's way less likely to be used as a variable name, but obviously other names could be used like loop_for or repeat_times etc. Thoughts?

16 33

Re: [Python-ideas] What about regexp string litterals : re".*" ?
by Ryan Gonzalez March 30, 2017

March 30, 2017

I feel like that borders on a bit too wordy... Personally, I'd like to see something like Felix's regular definitions: http://felix-lang.org/share/src/web/tut/regexp_01.fdoc#Regular_definitions.… -- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com On Mar 29, 2017 3:30 PM, "Abe Dillon" <abedillon(a)gmail.com> wrote: My 2 cents is that regular expressions are pretty un-pythonic because of their horrible readability. I would much rather see Python adopt something like Verbal Expressions ( https://github.com/VerbalExpressions/ PythonVerbalExpressions ) into the standard library than add special syntax support for normal REs. On Tue, Mar 28, 2017 at 3:31 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > On 28 March 2017 at 08:54, Simon D. <simon(a)acoeuro.com> wrote: > > I believe that the u"" notation in Python 2.7 is defined by while > > importing the unicode_litterals module. > > That's not true. The u"..." syntax is part of the language. from > future import unicode_literals is something completely different. > > > Each regexp lib could provide its instanciation of regexp litteral > > notation. > > The Python language has no way of doing that - user (or library) > defined literals are not possible. > > > And if only the default one does, it would still be won for the > > beginers, and the majority of persons using the stdlib. > > How? You've yet to prove that having a regex literal form is an > improvement over re.compile(r'put your regex here'). You've asserted > it, but that's a matter of opinion. We'd need evidence of real-life > code that was clearly improved by the existence of your proposed > construct. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

3 2

Add pathlib.Path.write_json and pathlib.Path.read_json
by Ram Rachum March 29, 2017

March 29, 2017

Hi guys, What do you think about adding methods pathlib.Path.write_json and pathlib.Path.read_json , similar to write_text, write_bytes, read_text, read_bytes? This would make writing / reading JSON to a file a one liner instead of a two-line with clause. Thanks, Ram.

18 30

Proposal: Query language extension to Python (PythonQL)
by Pavel Velikhov March 28, 2017

March 28, 2017

Hi folks! We started a project to extend Python with a full-blown query language about a year ago. The project is call PythonQL, the links are given below in the references section. We have implemented what is kind of an alpha version now, and gained some experience and insights about why and where this is really useful. So I’d like to share those with you and gather some opinions whether you think we should try to include these extensions in the Python core. Intro What we have done is (mostly) extended Python’s comprehensions with group by, order by, let and window clauses, which can come in any order, thus comprehensions become a query language a bit cleaner and more powerful than SQL. And we added a couple small convenience extensions, like a We have identified three top motivations for folks to use these extensions: Our Motivations 1. This can become a standard for running queries against database systems. Instead of learning a large number of different SQL dialects (the pain point here are libraries of functions and operators that are different for each vendor), the Python developer needs only to learn PythonQL and he can query any SQL and NoSQL database. 2. A single PythonQL expression can integrate a number of databases/files/memory structures seamlessly, with the PythonQL optimizer figuring out which pieces of plans to ship to which databases. This is a cool virtual database integration story that can be very convenient, especially now, when a lot of data scientists use Python to wrangle the data all day long. 3. Querying data structures inside Python with the full power of SQL (and a bit more) is also really convenient on its own. Usually folks that are well-versed in SQL have to resort to completely different means when they need to run a query in Python on top of some data structures. Current Status We have PythonQL running, its installed via pip and an encoding hack, that runs our preprocessor. We currently compile PythonQL into Python using our executor functions and execute Python subexpressions via eval. We don’t do any optimization / rewriting of queries into languages of underlying systems. And the query processor is basic too, with naive implementations of operators. But we’ve build DBMS systems before, so if there is a good amount of support for this project, we’ll be able to build a real system here. Your take on this Extending Python’s grammar is surely a painful thing for the community. We’re now convinced that it is well worth it, because of all the wonderful functionality and convenience this extension offers. We’d like to get your feedback on this and maybe you’ll suggest some next steps for us. References PythonQL GitHub page: https://github.com/pythonql/pythonql <https://github.com/pythonql/pythonql> PythonQL Intro and Tutorial (this is all User Documentation we have right now): https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial <https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial> A use-case of querying Event Logs and doing Process Mining with PythonQL: https://github.com/pythonql/pythonql/wiki/Event-Log-Querying-and-Process-Mi… <https://github.com/pythonql/pythonql/wiki/Event-Log-Querying-and-Process-Mi…> PythonQL demo site: www.pythonql.org <http://www.pythonql.org/> Best regards, PythonQL Team

11 24

Adding an 'errors' argument to print
by Ryan Gonzalez March 28, 2017

March 28, 2017

Recently, I was working on a Windows GUI application that ends up running ffmpeg, and I wanted to see the command that was being run. However, the file name had a Unicode character in it (it's a Sawano song), and when I tried to print it to the console, it crashed during the encode/decode. (The encoding used in cmd doesn't support Unicode characters.) The workaround was to do: print(mystring.encode(sys.stdout.encoding, errors='replace).decode(sys.stdout.encoding)) Not fun, especially since this was *just* a debug print. The proposal: why not add an 'errors' argument to print? That way, I could've just done: print(mystring, errors='replace') without having to worry about it crashing. -- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com

8 8

Re: [Python-ideas] Adding an 'errors' argument to print
by Victor Stinner March 27, 2017

March 27, 2017

print(msg) calls sys.stdout.write(msg): write() expects text, not bytes. I dislike the idea of putting encoding options in print. It's too specific. What if tomorrow you replace print() with file.write()? Do you want to add errors there too? No, it's better to write own formatter function as shown in a previous email. Victor Le 25 mars 2017 8:50 PM, "Michel Desmoulin" <desmoulinmichel(a)gmail.com> a écrit : Le 24/03/2017 à 17:37, Victor Stinner a écrit : > *If* we change something, I would prefer to modify sys.stdout. The > following issue proposes to add > sys.stdout.set_encoding(errors='replace'): > http://bugs.python.org/issue15216 > > You can already set the PYTHONIOENCODING environment variable to > ":replace" to use "replace" on sys.stdout (and sys.stderr). > > Victor This is not the same. You may want to locally apply "errors=replace" and not the whole program. Indeed, this can silence encoding problems. So I would probably never set in to errors at dev time except for the few places where I know I can explicitly silence errors. I quite like this print(errors="replace|ignore"). This is not going to cause any trouble, and can only help. _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

6 7

Re: [Python-ideas] Proposal: Query language extension to Python (PythonQL)
by Gerald Britton March 27, 2017

March 27, 2017

> > On 25 March 2017 at 11:24, Pavel Velikhov <pavel.velikhov at gmail.com> > wrote: > > No, the current solution is temporary because we just don’t have the > > manpower to > > implement the full thing: a real system that will rewrite parts of > PythonQL > > queries and > > ship them to underlying databases. We need a real query optimizer and > smart > > wrappers > > for this purpose. But we’ll build one of these for demo purposes soon > > (either a Spark > > wrapper or a PostgreSQL wrapper). > One thought, if you're lacking in manpower now, then proposing > inclusion into core Python means that the core dev team will be taking > on an additional chunk of code that is already under-resourced. That > rings alarm bells for me - how would you imagine the work needed to > merge PythonQL into the core Python grammar would be resourced? > I should say that in practice, I think that the solution is relatively > niche, and overlaps quite significantly with existing Python features, > so I don't really see a compelling case for inclusion. The parallel > with C# and LINQ is interesting here - LINQ is a pretty cool > technology, but I don't see it in widespread use in general-purpose C# > projects (disclaimer: I don't get to see much C# code, so my experience is limited). I see lots of C# code, but (thankfully) not so much LINQ to SQL. Yes, it is a cool technology. But I sometimes have a problem with the SQL it generates. Since I'm also a SQL developer, I'm sensitive to how queries are constructed, for performance reasons, as well as how they look, for readability and aesthetic reasons. LINQ queries can generate poorly-performing SQL, since LINQ is a basically a translator, but not an AI. As far as appearances go, LINQ queries can look pretty gnarly, especially if they include sub queries or a few joins. That makes it hard for the SQL dev (me!) to read and understand if there are performance problems (which there often are, in my experience) So, I would tend to code the SQL separately and put it in a SQL view, function or stored procedure. I can still parse the results with LINQ (not LINQ to SQL), which is fine. For similar reasons, I'm not a huge fan of ORMs either. Probably my bias towards designing the database first and building up queries to meet the business goals before writing a line of Python, C#, or the language de jour. -- Gerald Britton, MCSE-DP, MVP LinkedIn Profile: http://ca.linkedin.com/in/geraldbritton

4 7

(no subject)
by Gerald Britton March 27, 2017

March 27, 2017

>* On 25 Mar 2017, at 15:51, Gerald Britton <gerald.britton at gmail.com <https://mail.python.org/mailman/listinfo/python-ideas>> wrote: > *> >* On 25 March 2017 at 11:24, Pavel Velikhov <pavel.velikhov at gmail.com <http://gmail.com> <http://gmail.com/ <http://gmail.com/>>> wrote: > *>* > No, the current solution is temporary because we just don’t have the > *>* > manpower to > *>* > implement the full thing: a real system that will rewrite parts of PythonQL > *>* > queries and > *>* > ship them to underlying databases. We need a real query optimizer and smart > *>* > wrappers > *>* > for this purpose. But we’ll build one of these for demo purposes soon > *>* > (either a Spark > *>* > wrapper or a PostgreSQL wrapper). > *>* One thought, if you're lacking in manpower now, then proposing > *>* inclusion into core Python means that the core dev team will be taking > *>* on an additional chunk of code that is already under-resourced. That > *>* rings alarm bells for me - how would you imagine the work needed to > *>* merge PythonQL into the core Python grammar would be resourced? > *>* I should say that in practice, I think that the solution is relatively > *>* niche, and overlaps quite significantly with existing Python features, > *>* so I don't really see a compelling case for inclusion. The parallel > *>* with C# and LINQ is interesting here - LINQ is a pretty cool > *>* technology, but I don't see it in widespread use in general-purpose C# > *>* projects (disclaimer: I don't get to see much C# code, so my > *>* experience is limited). > *> >* I see lots of C# code, but (thankfully) not so much LINQ to SQL. Yes, it is a cool technology. But I sometimes have a problem with the SQL it generates. Since I'm also a SQL developer, I'm sensitive to how queries are constructed, for performance reasons, as well as how they look, for readability and aesthetic reasons. > *> >* LINQ queries can generate poorly-performing SQL, since LINQ is a basically a translator, but not an AI. As far as appearances go, LINQ queries can look pretty gnarly, especially if they include sub queries or a few joins. That makes it hard for the SQL dev (me!) to read and understand if there are performance problems (which there often are, in my experience) > *> We want to go beyond being a basic translator. Especially if the common use-case will be integrating multiple databases. We can also introduce decent-looking hints (maybe not always decent looking) to generate better plans. Not sure about asethetics though... >* So, I would tend to code the SQL separately and put it in a SQL view, function or stored procedure. I can still parse the results with LINQ (not LINQ to SQL), which is fine. > *> >* For similar reasons, I'm not a huge fan of ORMs either. Probably my bias towards designing the database first and building up queries to meet the business goals before writing a line of Python, C#, or the language de jour. > * This sounds completely reasonable, but this means you’re tied to a specific DBMS (especially if you’re using a lot of built-in functions that are usually very specific to a database). PythonQL (when it has enough functionality) should give you independence. True though not always needed. e.g. at present I'm working for a large company with thousands of db servers in all the popular flavors. The probability of changing even one of them to a different vendor is essentially zero. The costs and dependencies far outweigh any hoped-for advantage. At the same time, I'm happy to optimize the SQL for different target environments. If I lack the specific expertise, I know where to go to find it. The Adapter pattern helps here. It's actually more important for me to build queries that can be used in multiple client languages. We're using Java, C++, C#, F#, VB, ... and Python, of course (and probably others that I don't know we use). I can optimize the query once and not worry about the clients messing it up. -- Gerald Britton, MCSE-DP, MVP LinkedIn Profile: http://ca.linkedin.com/in/geraldbritton

1 0

IDEA
by Faaiz Asim March 27, 2017

March 27, 2017

Hi, I am currently enrolled in computer science program and am currently in 6th semester. I am comfortable using c++,java,python in general. I know this is a little late for proposing an idea but i was busy in exams. Besides i wanted to get into something where i was determined to contribute to the society. It took a little while but i found this ultimately. I think i have a good idea for a project that might further facilitate the use of python. My experience with python suggests that python is one the best all rounders in market. Its a good and easy head start for newbies and a powerful tool in good hands. Although i haven't used any IDE for python but i have serious complaints against lack of good code editors and python ides in general. I was suggested to use atom from my friends but i preferred rolling back to plain text editor for the trouble it cost me to shift to atom. Not degrading atom or python ides but lack of popularity of any pyhton ide (at least best to my knowledge) suggests a need for improvement in current ides. I am aware of the default IDE (IDLE) which ships with the default python package but I find it unsatisfactory in terms of debugging, aesthetics and usability etc. The main idea is to improve it or rebase it from the grounds up, and making it capable enough to ship with the python package. (PS. i would love to develop a new IDE of my own even if that is besides GSoc )

4 3

PEP: Python Documentation Translations
by Julien Palard March 23, 2017

March 23, 2017

Hi, Here is the follow-up to the "Translated Python documentation" thread on python-dev [1]_, as a PEP describing the steps to make Python Documentation Translation official on accessible on docs.python.org. .. [1] [Python-Dev] Translated Python documentation (https://mail.python.org/pipermail/python-dev/2017-February/147416.html) You can also read it online at https://github.com/JulienPalard/peps/blob/master/pep-0xxx.rst ============================== PEP: XXX Title: Python Documentation Translations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <vstinner(a)redhat.com>, Inada Naoki <songofacandy(a)gmail.com>, Julien Palard <julien(a)palard.fr> Status: Draft Type: Process Created: 04-Mar-2017 Post-History: Abstract ======== The intent of this PEP is to make existing translations of the Python Documentation more accessible and discoverable. By doing so, attracting and motivating new translators and new translations. Translated documentation will be hosted on python.org. Examples of two active translation teams: * http://docs.python.org/fr/: French * http://docs.python.org/jp/: Japanese http://docs.python.org/en/ will redirect to http://docs.python.org/. Sources of translated documentation will be hosted in the Python Documentation organization on GitHub: https://github.com/python-docs/. Contributors will have to sign the Python Contributor Agreement (CLA) and the license will be the PSF License. Motivation ========== On the french ``#python-fr`` IRC channel on freenode, it's not rare to meet people who don't speak english and so are unable to read the Python official documentation. Python wants to be widely available, to all users, in any language: that's also why Python 3 now allows any non-ASCII identifiers: https://www.python.org/dev/peps/pep-3131/#rationale There are a least 3 groups of people who are translating the Python documentation in their mother language (french [16]_ [17]_ [18]_, japanese [19]_ [20]_, spanish [21]_), even though their translation are not visible on d.p.o. Other less visible and less organized groups are also translating in their mother language, we heard of Russian, Chinese, Korean, maybe some others we didn't found yet. This PEP defines rules to move translations on docs.python.org so they can easily be found by developers, newcomers and potential translators. The Japanese team currently (March 2017) translated ~80% of the documentation, french team ~20%. French translation went from 6% to 23% in 2016 [13]_ with 7 contributors [14]_, proving a translation team can be faster than documentation mutates. Quoting Xiang Zhang about Chinese translations: I have seen several groups trying to translate part of our official doc. But their efforts are disperse and quickly become lost because they are not organized to work towards a single common result and their results are hold anywhere on the Web and hard to find. An official one could help ease the pain. Rationale ========= Translation ----------- Issue tracker ''''''''''''' Considering that issues opened about translations may be written in the translation language, which can be considered noise but at least is inconsistent, issues should be placed outside `bugs.python.org <https://bugs.python.org/>`_ (b.p.o). As all translation must have their own github project (see `Repository for Po Files`_), they must use the associated github issue tracker. Considering the noise induced by translation issues redacted in any languages which may beyond every warnings land in b.p.o, triage will have to be done. Considering that translations already exist and are not actually a source of noise in b.p.o, an unmanageable amount of work is not to be expected. Considering that Xiang Zhang and Victor Stinner are already triaging, and Julien Palard is willing to help on this task, noise on b.p.o is not to be expected. Also, language team coordinators (see `Language Team`_) should help triaging b.p.o by properly indicating, in the issue author language if needed, the right issue tracker. Branches '''''''' Translation teams should focus on last stable versions, and use tools (scripts, translation memory, …) to automatically translate what is done in one branch to other branches. .. note:: Translation memories are a kind of database of previously translated paragraphs, even removed ones. See also `Sphinx Internationalization <http://www.sphinx-doc.org/en/stable/intl.html>`_. The three stable branches will be translated [12]_: 2.7, 3.5, and 3.6. The scripts to build the documentation of older branches have to be modified to support translation [12]_, whereas these branches now only accept security-only fixes. The development branch (master) should have a lower translation priority than stable branches. But docsbuild-scripts should build it anyway so it is possible for a team to work on it to be ready for the next release. Hosting ------- Domain Name, Content negociation and URL '''''''''''''''''''''''''''''''''''''''' Different translation can be told appart by changing one of: Country Code Top Level Domain (CCTLD), path segment, subdomain, or by content negociation. Buying a CCTLD for each translations is expensive, time-consuming, and sometimes almost impossible when already registered, this solution should be avoided. Using subdomains like "es.docs.python.org" or "docs.es.python.org" is possible but confusing ("is it `es.docs.python.org` or `docs.es.python.org`?"). Hyphens in subdomains like `pt-br.doc.python.org` in uncommon and SEOMoz [23]_ correlated the presence of hyphens as a negative factor. Usage of underscores in subdomain is prohibited by the RFC1123 [24]_, section 2.1. Finally using subdomains means creating TLS certificates for each languages, which is more maintenance, and will probably causes us troubles in language pickers if, like for version picker, we want a preflight to check if the translation exists in the given version: preflight will probably be blocked by same-origin-policy. Wildcard TLS certificates are very expensive. Using content negociation (HTTP headers ``Accept-Language`` in the request and ``Vary: Accept-Language``) leads to a bad user experience where they can't easily change the language. According to Mozilla: "This header is a hint to be used when the server has no way of determining the language via another way, like a specific URL, that is controlled by an explicit user decision." [25]_. As we want to be able to easily change the language, we should not use the content negociation as a main language determination, so we need somthing else. Last solution is to use the URL path, which looks readable, allows for an easy switch from a language to another, and nicely accepts hyphens. Typically something like: "docs.python.org/de/". Example with a hyphen: "docs.python.org/pt-BR/" As for version, sphinx-doc does not support compiling for multiple languages, so we'll have full builds rooted under a path, exactly like we're already doing with versions. So we can have "docs.python.org/de/3.6/" or "docs.python.org/3.6/de/". Question is "Does the language contains multiple version or does version contains multiple languages?" As versions exists in any cases, and translations for a given version may or may not exists, we may prefer "docs.python.org/3.6/de/", but doing so scatter languages everywhere. Having "/de/3.6/" is clearer about "everything under /de/ is written in deutch". Having the version at the end is also an habit taken by readers of the documentation: they like to easily change the version by changing the end of the path. So we should use the following pattern: "docs.python.org/LANGUAGE_TAG/VERSION/". Current documentation is not moved to "/en/", but "docs.python.org/en/" will redirect to "docs.python.org/en/". Language Tag '''''''''''' A common notation for language tags is the IETF Language Tag [3]_ [4]_ based on ISO 639, alghough gettext uses ISO 639 tags with underscores (ex: ``pt_BR``) instead of dashes to join tags [5]_ (ex: ``pt-BR``). Examples of IETF Language Tags: ``fr`` (French), ``jp`` (Japanese), ``pt-BR`` (Orthographic formulation of 1943 - Official in Brazil). It is more common to see dashes instead of underscores in URLs [6]_, so we should use IETF language tags, even if sphinx uses gettext internally: URLs are not meant to leak the underlying implementation. It's uncommon to see capitalized letters in URLs, and docs.python.org don't use any, so it may hurt readability by attracting the eye on it, like in: "https://docs.python.org/pt-BR/3.6/library/stdtypes.html". RFC 5646 (Tags for Identifying Languages (IETF)) section-2.1 [7]_ tells the tags are not case sensitive. As the RFC allows lower case, and it enhances readability, we should use lowercased tags like ``pt-br``. It's redundant to display both language and country code if they're the same, typically "de-DE", "fr-FR", although it make sense, respectively "Deutch as spoken in Germany" and "French as spoken in France", it's not a usefull information for the reader. So we may drop those redundencies. We should obviously keep the country part when it make sense like "pt-BR" for "Portuguese as spoken in Brazil". So we should use IETF language tags, lowercased, like ``/fr/``, ``/pt-br/``, ``/de/`` and so on. Fetching And Building Translations '''''''''''''''''''''''''''''''''' Currently docsbuild-scripts are building the documentation [8]_. These scripts should be modified to fetch and build translations. Building new translations is like building new versions, so we're adding complexity, but not that much. Two steps should be configurable distinctively: Build a new language, and add it to the language picker. This allows a transition step between "we accepted the language" and "it is translated enough to be made public", during this step, translators can review their modifications on d.p.o without having to build the documentation locally. From the translations repositories, only the ``.po`` files should be opened by the docsbuild-script to keep the attack surface and probable bugs sources at a minimum. This mean no translation can patch sphinx to advertise their translation tool. (This specific feature should be handled by sphinx anyway [9]_). Community --------- Mailing List '''''''''''' The `doc-sig`_ mailing list will be used to discuss cross-language changes on translated documentations. There is also the i18n-sig list but it's more oriented towards i18n APIs [1]_, than translation the Python documentation. .. _i18n-sig: https://mail.python.org/mailman/listinfo/i18n-sig .. _doc-sig: https://mail.python.org/mailman/listinfo/doc-sig Chat '''' Python community being highly active on IRC, we should create a new IRC channel on freenode, typically #python-doc for consistency with the mailing list name. Each language coordinator can organize its own team, even by choosing another chat system if the local usage asks for it. As local teams will write in their native languages, we don't want each team in a single channel, and it's also natural for the local teams to reuse their local channels like "#python-fr" for french translators. Repository for PO Files ''''''''''''''''''''''' Considering that each translation teams may want to use different translation tools, and that those tools should easily be synchronized with git, all translations should expose their ``.po`` files via a git repository. Considering that each translation will be exposed via git repositories, and that Python has migrated to GitHub, translations will be hosted on github. For consistency and discoverability, all translations should be in the same github organization and named according to a common pattern. Considering that we want translations to be official, and that Python already have a github organization, translations should be hosted as projects of the `Python documentation GitHub organization`_. For consistency, translations repositories should be called ``python-docs-LANGUAGE_TAG`` [22]_. The docsbuild-scripts may enforce this rule by refusing to fetch outside of the Python organization or a wrongly named repository. The CLA bot may be used on the translation repositories, but with a limited effect as local coordinators may synchronize themselves translations from an external tool like transifex, loosing in the process who translated what. Version can be hosted on different repositories, different directories or different branches. Storing them on different repositories will probably pollute the Python documentation github organization. As it is typical and natural to use branches to separate versions, branches should be used to do so. .. _Python documentation GitHub organization: https://github.com/python-docs/ Translation tools ''''''''''''''''' Most of the translation work is actually done on Transifex [15]_. Other tools may be used later https://pontoon.mozilla.org/ and http://zanata.org/ Contributor Agreement ''''''''''''''''''''' Contributions to translated documentation will be requested to sign the Python Contributor Agreement (CLA): https://www.python.org/psf/contrib/contrib-form/ Language Team ''''''''''''' Each language team should have one coordinator responsible to: - Manage the team - Choose and manage the tools its team will use (chat, mailing list, …) - Ensure contributors understand and agree with the CLA - Ensure quality (grammar, vocabulary, consistency, filtering spam, ads, …) - Do redirect to GitHub issue tracker issues related to its language on bugs.python.org The license will be the `PSF License <https://docs.python.org/3/license.html>`_, and copyright should be transferable to PSF later. Alternatives ------------ Simplified English '''''''''''''''''' It would be possible to introduce a "simplified english" version like wikipedia did [10]_, as discussed on python-dev [11]_, targetting english learners and childrens. Pros: It yields a single other translation, theorically readable by everyone, and reviewable by current maintainers. Cons: Subtle details may be lost, and translators from english to english may be hard to find as stated by Wikipedia: > The main English Wikipedia has 5 million articles, written by nearly 140K active users; the Swedish Wikipedia is almost as big, 3M articles from only 3K active users; but the Simple English Wikipedia has just 123K articles and 871 active users. That's fewer articles than Esperanto! Changes ======= Migrate GitHub Repositories --------------------------- We (authors of this PEP) already own french and japanese Git repositories, so moving them to the Python documentation organization will not be a problem. We'll however follow the `New Translation Procedure`_. Patch docsbuild-scripts to Compile Translations ----------------------------------------------- Docsbuild-script must be patched to: - List the languages tags to build along with the branches to build. - List the languages tags to display in the language picker. - Find translation repositories by formatting ``github.com:python-docs/python-docs-{language_tag}.git`` (See `Repository for Po Files`_) - Build translations for each branches and each languages Patched docsbuild-scripts must only open ``.po`` files from translation repositories. List coordinators in the devguide --------------------------------- Add a page or a section with an empty list of coordinators to the devguide, each new coordinators will be added to this list. Create sphinx-doc Language Picker --------------------------------- Highly similar to the version picker, a language picker must be implemented. This language picker must be configurable to hide or show a given language. Enhance rendering of untranslated fuzzy translations ---------------------------------------------------- It's an opened sphinx issue [9]_, but we'll need it so we'll have to work on it. Translated, fuzzy, and untranslated paragraphs should be differentiated. (Fuzzy paragraphs have to warn the reader what he's reading may be out of date.) New Translation Procedure ========================= Designate a Coordinator ----------------------- The first step is to designate a coordinator, see `Language Team`_, The coordinator must sign the CLA. The coordinator should be added to a list of translation coordinator on the devguide. Create github repository ------------------------ Create a repository named "python-docs-{LANGUAGE_TAG}" on the Python documentation github organization (See `Repository For Po Files`_.), and grant the language coordinator push rights to this repository. Add support for translations in docsbuild-scripts ------------------------------------------------- As soon as the translation hits its firsts commits, update the docsbuild-scripts configuration to build the translation (but not displaying it in the language picker). Add translation to the language picker -------------------------------------- As soon as the translation hits: - 100% of bugs.html with proper links to the language repository issue tracker. - 100% of tutorial - 100% of library/functions (builtins) the translation can be added to the language picker. Previous discussions ==================== - `[Python-ideas] Cross link documentation translations (January, 2016)`_ - `[Python-ideas] Cross link documentation translations (January, 2016)`_ - `[Python-ideas] https://docs.python.org/fr/ ? (March 2016)`_ .. _[Python-ideas] Cross link documentation translations (January, 2016): https://mail.python.org/pipermail/python-ideas/2016-January/038010.html .. _[Python-Dev] Translated Python documentation (Febrary 2016): https://mail.python.org/pipermail/python-dev/2017-February/147416.html .. _[Python-ideas] https://docs.python.org/fr/ ? (March 2016): https://mail.python.org/pipermail/python-ideas/2016-March/038879.html References ========== .. [1] [I18n-sig] Hello Python members, Do you have any idea about Python documents? (https://mail.python.org/pipermail/i18n-sig/2013-September/002130.html) .. [2] [Doc-SIG] Localization of Python docs (https://mail.python.org/pipermail/doc-sig/2013-September/003948.html) .. [3] Tags for Identifying Languages (http://tools.ietf.org/html/rfc5646) .. [4] IETF language tag (https://en.wikipedia.org/wiki/IETF_language_tag) .. [5] GNU Gettext manual, section 2.3.1: Locale Names (https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html) .. [6] Semantic URL: Slug (https://en.wikipedia.org/wiki/Semantic_URL#Slug) .. [7] Tags for Identifying Languages: Formatting of Language Tags (https://tools.ietf.org/html/rfc5646#section-2.1.1) .. [8] Docsbuild-scripts github repository (https://github.com/python/docsbuild-scripts/) .. [9] i18n: Highlight untranslated paragraphs (https://github.com/sphinx-doc/sphinx/issues/1246) .. [10] Wikipedia: Simple English (https://simple.wikipedia.org/wiki/Main_Page) .. [11] Python-dev discussion about simplified english (https://mail.python.org/pipermail/python-dev/2017-February/147446.html) .. [12] Passing options to sphinx from Doc/Makefile (https://github.com/python/cpython/commit/57acb82d275ace9d9d854b156611e641f6…) .. [13] French translation progression (https://mdk.fr/pycon2016/#/11) .. [14] French translation contributors (https://github.com/AFPy/python_doc_fr/graphs/contributors?from=2016-01-01&t…) .. [15] Python-doc on Transifex (https://www.transifex.com/python-doc/) .. [16] French translation (https://www.afpy.org/doc/python/) .. [17] French translation github (https://github.com/AFPy/python_doc_fr) .. [18] French mailing list (http://lists.afpy.org/mailman/listinfo/traductions) .. [19] Japanese translation (http://docs.python.jp/3/) .. [20] Japanese github (https://github.com/python-doc-ja/python-doc-ja) .. [21] Spanish translation (http://docs.python.org.ar/tutorial/3/index.html) .. [22] [Python-Dev] Translated Python documentation: doc vs docs (https://mail.python.org/pipermail/python-dev/2017-February/147472.html) .. [23] Domains - SEO Best Practices | Moz (https://moz.com/learn/seo/domain) .. [24] Requirements for Internet Hosts -- Application and Support (https://www.ietf.org/rfc/rfc1123.txt) .. [25] Accept-Language (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Julien Palard https://mdk.fr

7 11

Third party module in standard library
by George Fischhof March 23, 2017

March 23, 2017

Hi Guys, I would like to ask You: What is the process to propose a module to be part of the standard library? I would like to propose the following modules: requests https://pypi.python.org/pypi/requests and xmltodict https://pypi.python.org/pypi/xmltodict Both of them makes the life easier, and more simple ;-) Of course there are several similar libraries -- I know -- and actually several good modules could be part of standard library. ... Kind regards, George

3 2

Multi threaded computation, thread local storage reducing false sharing, read only access to globals and a paused GIL to allow parallel scaling reads
by Josh Ring March 20, 2017

March 20, 2017

Hi My name is Josh Ring, I am interested in raw compute performance from C modules called from python. Most of which are single threaded (eg Numpy, Scipy etc) Some things are sensible with many threads: 1. Read only global state can avoid locks entirely in multithreaded code, this is to avoid cache line invalidations killing scaling >2-4 threads. 2. Incr/decr can be paused when entering the parallel region to avoid invalidating caches of objects, readonly access makes this safe. 2. Locality of memory, so using thread local stack by default and heap allocations bound per thread this is essential to scale >4 threads and with NUMA server systems. 3. Leaving the GIL intact for single threaded code to do the "cleanup stage" of temporaries after parallel computation has finished. - I liked the approach of a "parallel region", where data does not need to be pickled, and can directly read-only access shared memory. - If global state is unchangeable from a threaded region we can avoid many gotchas and races, leaving the GIL alone, almost. - If reference counting can be "paused" during the parallel region we can avoid cache invalidation from multiple threads, due to incr/decr, which limits scaling with more threads, this is evident even with 2 threads. - Thread local storage is the default and only option, avoiding clunky "threading.local()" storage classes, "thread bound" heap allocations would also be a good thing to increase efficiency and reduce "false sharing". https://en.wikipedia.org/wiki/False_sharing - Implement the parallel region using a function with a decorator akin to openMP? The function then defines the scope for the local variables and the start and end of parallel region, when to return etc in a straightforward manner. - By default, the objects returned from the parallel region return into separate objects (avoiding GIL contention), these temporary objects are then merged into a list once control is returned to just a single thread. - Objects marked @thread_shared have their state merged from the thread local copies once execution from all the threads has finished. This could be made intelligent if some index is provided to put the entries in the right place in a list/dict etc. Thoughts? This proposal is borrowing several ideas from Pyparallel, but without the focus on windows only and IO. It is more focused on accelerating raw compute performance and is tailored for high performance computing for instance at CERN etc.

1 0

Additions to collections.Counter and a Counter derived class
by Marco Cognetta March 16, 2017

March 16, 2017

Hi all, I have been using the Counter class recently and came across several things that I was hoping to get feedback on. (This is my first time mailing this list, so any advice is greatly appreciated) 1) Addition of a Counter.least_common method: This would add a method to Counter that is basically the opposite of the pre-existing Counter.most_common method. In this case, the least common elements are considered the elements in c with the lowest (non-zero) frequency. This was addressed in https://bugs.python.org/issue16994, but it was never resolved and is still open (since Jan. 2013). This is a small change, but I think that it is useful to include in the stdlib. I have written a patch for this, but have not submitted a PR yet. It can be found at https://github.com/mcognetta/cpython/tree/collections_counter_least_common 2) Undefined behavior when using Counter.most_common: Consider the case c = Counter([1, 1, 2, 2, 3, 3, 'a', 'a', 'b', 'b', 'c', 'c']), when calling c.most_common(3), there are more than 3 "most common" elements in c and c.most_common(3) will not always return the same list, since there is no defined total order on the elements in c. Should this be mentioned in the documentation? Additionally, perhaps there is room for a method that produces all of the elements with the n highest frequencies in order of their frequencies. For example, in the case of c = Counter([1, 1, 1, 2, 2, 3, 3, 4, 4, 5]) c.aforementioned_method(2) would return [(1, 3), (2, 2), (3, 2), (4, 2)] since the two highest frequencies are 3 and 2. 3) Addition of a collections.Frequency or collections.Proportion class derived from collections.Counter: This is sort of discussed in https://bugs.python.org/issue25478. The idea behind this would be a dictionary that, instead of returning the integer frequency of an element, would return it's proportional representation in the iterable. So, for example f = Frequency('aabbcc'), f would hold Frequency({'a': 0.3333333333333333, 'b': 0.3333333333333333, 'c': 0.3333333333333333}). To address >The pitfall I imagine here is that if you continue adding elements after normalize() is called, the >results will be nonsensical. from the issue, this would not be a problem because we could just build it entirely on top of a Counter, keep a count of the total number of elements in the Counter, and just divide by that every time we output or return the object or any of its elements. I think that this would be a pretty useful addition especially for code related to discrete probability distributions (which is what motivated this in the first place). Thanks in advance, -Marco

4 8

Optional parameters without default value
by Serhiy Storchaka March 10, 2017

March 10, 2017

Function implemented in Python can have optional parameters with default value. It also can accept arbitrary number of positional and keyword arguments if use var-positional or var-keyword parameters (*args and **kwargs). But there is no way to declare an optional parameter that don't have default value. Currently you need to use the sentinel idiom for implementing this: _sentinel = object() def get(store, key, default=_sentinel): if store.exists(key): return store.retrieve(key) if default is _sentinel: raise LookupError else: return default There are drawback of this: * Module's namespace is polluted with sentinel's variables. * You need to check for the sentinel before passing it to other function by accident. * Possible name conflicts between sentinels for different functions of the same module. * Since the sentinel is accessible outside of the function, it possible to pass it to the function. * help() of the function shows reprs of default values. "foo(bar=<object object at 0xb713c698>)" looks ugly. I propose to add a new syntax for optional parameters. If the argument corresponding to the optional parameter without default value is not specified, the parameter takes no value. As well as the "*" prefix means "arbitrary number of positional parameters", the prefix "?" can mean "single optional parameter". Example: def get(store, key, ?default): if store.exists(key): return store.retrieve(key) try: return default except NameError: raise LookupError Alternative syntaxes: * "=" not followed by an expression: "def get(store, key, default=)". * The "del" keyword: "def get(store, key, del default)". This feature is orthogonal to supporting positional-only parameters. Optional parameters without default value can be positional-or-keyword, keyword-only or positional-only (if the latter is implemented).

20 41

Exploiting type-homogeneity in list.sort() (again!)
by Elliot Gorokhovsky March 10, 2017

March 10, 2017

(Summary of results: my patch at https://bugs.python.org/issue28685 makes list.sort() 30-50% faster in common cases, and at most 1.5% slower in the uncommon worst case.) Hello all, You may remember seeing some messages on here about optimizing list.sort() by exploiting type-homogeneity: since comparing apples and oranges is uncommon (though possible, i.e. float to int), it pays off to check if the list is type-homogeneous (as well as homogeneous with respect to some other properties), and, if it is, to replace calls to PyObject_RichCompareBool with calls to ob_type->tp_richcompare (or, in common special cases, to optimized compare functions). The entire patch is less than 250 lines of code, most of which is pretty boilerplate (i.e. a lot of assertions in #ifdef Py_DEBUG blocks, etc). I originally wrote that patch back in November. I've learned a lot since then, both about CPython and about mailing list etiquette :). Previous discussion about this can be found at https://mail.python.org/pipermail/python-dev/2016-October/146648.html and https://mail.python.org/pipermail/python-ideas/2016-October/042807.html. Anyway, I recently redid the benchmarks much more rigorously (in preparation for presenting this project at my school's science fair), achieving a standard deviation of less than 0.5% of the mean for all measurements. The exact benchmark script used can be found at https://github.com/embg/python-fastsort-benchmark (it's just sorting random lists of/lists of tuples of [type]. While listsort.txt talks about benchmarking different kinds of structured lists, instead of just random lists, the results here would hold in those cases just as well, because this makes individual comparisons cheaper, instead of reducing the number of comparisons based on structure). I also made a poster describing the optimization and including a pretty graph displaying the benchmark data: https://github.com/embg/python-fastsort-benchmark/blob/master/poster.pdf. For those who would rather read the results here (though it is a *really* pretty graph): *** Percent improvement for sorting random lists of [type] (1-patched/unpatched): float: 48% bounded int (magnitude smaller than 2^32): 48.4% latin string (all characters in [0,255]): 32.7% general int (reasonably uncommon?): 17.2% general string (reasonably uncommon?): 9.2% tuples of float: 63.2% tuples of bounded int: 64.8% tuples of latin string: 55.8% tuples of general int: 50.3% tuples of general string: 44.1% tuples of heterogeneous: 41.5% heterogeneous (lots of float with an int at the end; worst-case): -1.5% *** Essentially, it's a gamble where the payoff is 20-30 times greater than the cost, and the odds of losing are very small. Sorting is perhaps not a bottleneck in most applications, but considering how much work has gone into Python's sort (Timsort, etc; half of listobject.c is sort code), I think it's interesting that list.sort() can be made essentially twice faster by a relatively simple optimization. I would also add that Python dictionaries already implement this optimization: they start out optimizing based on the assumption that they'll only be seeing string keys, checking to make sure that assumption holds as they go. If they see a non-string key, they permanently switch over to the general implementation. So it's really the same idea, except here it doesn't matter as much what type we're dealing with, which is important, because lists are commonly used with lots of different types, as opposed to dictionaries, which overwhelmingly commonly use string keys, especially internally. (Correct me if I'm wrong in any of the above). I got a lot of great feedback/input on this patch as I was writing it, but after submitting it, I didn't hear much from anybody. (The reason I took so long to post was because I wanted to wait until I had the chance to do the benchmarks *right*). What do you all think? Thanks, Elliot

17 63

For/in/as syntax
by Brice PARENT March 9, 2017

March 9, 2017

Object: Creation of a ForLoopIterationObject object (name open to suggestions) using a "For/in/as" syntax. Disclaimer: I've read PEP-3136 which was refused. The present proposal includes a solution to what it tried to solve (multi-level breaking out of loops), but is a more general proposal and it offers many unrelated advantages and simplifications. It also gives another clean way of making a kind of for/except/else syntax of another active conversation. What it tries to solve: During the iteration process, we often need to create temporary variables (like counters or states) or tests that don't correspond to the logic of the algo, but are required to make it work. This proposal aims at removing these unnecessary and hard to understand parts and help write simpler, cleaner, more maintainable and more logical code. So, more PEP-8 compliant code. How does it solve it: By instanciating a new object when starting a loop, which contains some data about the iteration process and some methods to act on it. The list of available properties and methods are yet to be defined, but here are a subset of what could be really helpful. A word about compatibility and understandability before: "as" is already a keyword, so it is already reserved and easy to parse. It couldn't be taken for its other uses (context managers, import statements and exceptions) as they all start with a specific and different keyword. It would have a really similar meaning, so be easy to understand. It doesn't imply any incompatibility with previous code, as current syntax would of course still be supported, and it doesn't change anything anywhere else (no change in indentation levels for example, no deprecation). Syntax: for element in elements as elements_loop: assert type(elements_loop) is ForLoopIterationObject Properties and methods (non exhaustive list, really open to discussion and suggestions): for element in elements as elements_loop: elements_loop.length: int # len ? count ? elements_loop.counter: int # Add a counter0, as in Django? a counter1 ? elements_loop.completed: bool elements_loop.break() elements_loop.continue() elements_loop.skip(count=1) Examples of every presented element (I didn't execute the code, it's just to explain the proposal): ################################################## # forloop.counter and forloop.length for num, dog in enumerate(dogs): print(num, dog) print("That's a total of {} dogs !".format(len(dogs))) # would be equivalent to for dog in dogs as dogs_loop: print(dogs_loop.counter, dog) print("That's a total of {} dogs !".format(dogs_loop.length)) # -> cleaner, and probably faster as it won't have to call len() or enumerate() #(but I could easily be wrong on that) ################################################## # forloop.length when we broke out of the list small_and_medium_dogs_count = 0 for dog in generate_dogs_list_by_size(): if dog.size >= medium_dog_size: break small_and_medium_dogs_count += 1 print("That's a total of {} small to medium dogs !".format( small_and_medium_dogs_count)) # would be equivalent to for dog in generate_dogs_list_by_size() as dogs_loop: if dog.size >= medium_dog_size: break print("That's a total of {} small to medium dogs !".format( dogs_loop.length - 1)) # -> easier to read, less temporary variables ################################################## # forloop.break(), to break out of nested loops (or explicitly out of current #loop) - a little like pep-3136's first proposal has_dog_named_rex = False for owner in owners: for dog in dogs: if dog.name == "Rex": has_dog_named_rex = True break if has_dog_named_rex: break # would be equivalent to for owner in owners as owners_loop: for dog in dogs: # syntax without "as" is off course still supported if dog.name == "Rex": owners_loop.break() # -> way easier to read and understand, less temporary variables ################################################## # forloop.continue(), to call "continue" from any nested loops (or explicitly #in active loop) has_dog_named_rex = False for owner in owners: for dog in owner.dogs: if dog.name == "Rex": has_dog_named_rex = True break if has_dog_named_rex: continue # do something # would be equivalent to for owner in owners as owners_loop: for dog in owner.dogs: if dog.name == "Rex": owners_loop.continue() # do something # -> way easier to read and understand, less temporary variables ################################################## # forloop.completed, to know if the list was entirely consumed or "break" was #called (or exception caught) - might help with the for/except/elseproposal broken_out = False for dog in dogs: if dog.name == "Rex": broken_out = True break if broken_out: print("We didn't consume all the dogs list") # would be equivalent to for dog in dogs as dogs_loop: if dog.name == "Rex": break if not dogs_loop.completed: print("We didn't consume all the dogs list") # -> less temporary variables, every line of code is relevant, easy to #understand ################################################## # forloop.skip # In the example, we want to skip 2 elements starting from item #2 skip = 0 for num, dog in enumerate(dogs): if skip: skip -= 1 continue if num == 2: skip = 2 # would be equivalent to for dog in dogs as dogs_loop: if dogs_loop.counter == 2: dogs_loop.skip(2) # -> way easier to read and understand, less temporary variables # Notes : # - Does a call to forloop.skip() implies a forloop.continue() call or does # the code continue its execution until the end of the loop, which will # then be skipped? Implying the .continue() call seems less ambiguous to # me. Or the method should be called skip_next_iteration, or something # like that. # - Does a call to forloop.skip(2) adds 2 to forloop.length or not? # -> kwargs may be added to allow both behaviours for both questions. # We could allow the argument to be a function that accepts a single argument #and return a boolean, like dogs_loop.skip(lambda k: k % 3 == 0) # Execute the code on multiples of 3 only ################################################## Thoughts : - It would allow to pass forloop.break and forloop.continue as callback to other functions. Not sure yet if it's a good or a bad thing (readability against what it could offer). - I haven't yet used much the asynchronous functionalities, so I couldn't yet think about the implications of such a new syntax to this (and what about a lazy keyword in here?) - I suppose it's a huge work to create such a syntax. And I have no idea how complicated it could be to create methods (like break() and continue()) doing what keywords were doing until now. - I'm not sure if that would make sense in list comprehensions, despite being confusing. - Would enable to support callback events like forloop.on_break. But would there be a need for that? - Would this have a major impact on the loops execution times? - would a "while condition as condition_loop:" be of any use too? Sorry for the very long message, I hope it will get your interest. And I also hope my English was clear enough. Brice Parent

7 14

Smoothing transition: 'unicode' and 'basestring' as aliases for 'str'?
by Thomas Güttler March 7, 2017

March 7, 2017

I found this in an old post: > Maybe too late now but there should have been 'unicode', > 'basestring' as aliases for 'str'. I guess it is too late to think about it again ... Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

4 5

Wrapper for ctypes
by George Fischhof March 7, 2017

March 7, 2017

Hi Guys, right now I had to call functions from a dll, and I started using ctypes. I found this library too https://pypi.python.org/pypi/pywrap/0.1.0 which says (qutation): Replace this: prototype = ctypes.WINFUNCTYPE(wintypes.HANDLE, wintypes.UINT, wintypes.HANDLE)paramflags = (1, "uFormat"), (1, "hMem")SetClipboardData = prototype(("SetClipboardData", user32), paramflags)SetClipboardData.errcheck = null_errcheck With this: SetClipboardData = pywrap.wrap_winapi(name="SetClipboardData", library=user32, restype=wintypes.BOOL, params=[ Parameter("uFormat", wintypes.UINT), Parameter("hMem", wintypes.HANDLE) ], errcheck=null_errcheck) (end qutation) My idea: something like this library should be implemented in Python to make it more simple to use ctypes. BR, George

3 2

Re: [Python-ideas] Exploiting type-homogeneity in list.sort() (again!)
by Elliot Gorokhovsky March 7, 2017

March 7, 2017

On Mon, Mar 6, 2017, 4:42 PM Jim J. Jewett <jimjjewett(a)gmail.com> wrote: (1) Good Job. Thanks! (3) Ideally, your graph would have the desired-to-be lines after the as-is lines; for English writing, that would mean putting your (short red) lines to the right of the (tall blue) lines. (4) I suspect colors other than red and blue would be helpful as well, but I am explicitly not a design wiz. My best suggestion (other than "ask someone who isn't me") would be to use blue for as-is and green for to-be. Ya, I know the colors are terrible, I just have no graphic design experience so I figured I'd make the graph the same colors as the diagram (5) I don't know that all-ASCII (or at least all-Latin1) is true for most applications, but it is certainly true for most datasets run in countries where latin1 is sufficient, which includes most places outside of Asia, Africa, and perhaps Eastern Europe. If anything, that strengthens your case, since you can win on plenty of datasets even for applications where it isn't always safe, and for those datasets that do require a wider charset, you're likely to discover this quickly. True. I think a much bigger part of the consideration is also that a lot of software (e.g. file systems) *don't* support unicode, at least by default, so if you're dealing with text you got from another program or from the OS, it's usually ASCII. And usually, our shell scripts our getting their text from other programs (e.g. file names). (6) When I saw the flowchart around f(v, w), at first I was thinking about the key function used in some sorting... I suppose that isn't relevant, since those sorts (If I Recall Correctly) already create a parallel array to avoid recomputing the keys, but ... it might be worth clarifying, if you can find a way to do it easily without adding too much complexity. Maybe just change "this f" to "the compare function f"? You are correct -- key sorts create a parallel array. In fact, *all* sorts create a parallel array, for safety: the only way to make sure the objects aren't getting mutated as you sort is to keep them safe! If the objects are getting mutated during the sort, something is clearly going horribly wrong, but at least you won't segfault. (This is not part of my patch -- it's part of the original implementation). (7) I hope you have put in a pull request to get this added to python 3.7. Thanks! I don't think I can make pull requests, but I am going to submit a fixed version to the bug tracker (Tim pointed out that my current code isn't thread-safe or adversary-safe because it stores compare_function in a global. I have to modify the code keep it in local scope and pass in to every function that needs it. This will make the diff hairier, but not increase code complexity. -jJ

1 0

get() method for list and tuples
by Michel Desmoulin March 6, 2017

March 6, 2017

dict.get() is a very useful method, but for lists and tuples, we need to rely on try/except instead. Can we get list.get and tuple.get as well? Also, for list, a list.setdefault like the dict.setdefault would be logical.

18 69

for/except/else
by Wolfgang Maier March 5, 2017

March 5, 2017

I know what the regulars among you will be thinking (time machine, high bar for language syntax changes, etc.) so let me start by assuring you that I'm well aware of all of this, that I did research the topic before posting and that this is not the same as a previous suggestion using almost the same subject line. Now here's the proposal: allow an except (or except break) clause to follow for/while loops that will be executed if the loop was terminated by a break statement. The idea is certainly not new. In fact, Nick Coghlan, in his blog post http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_e…, uses it to provide a mental model for the meaning of the else following for/while, but, as far as I'm aware, he never suggested to make it legal Python syntax. Now while it's possible that Nick had a good reason not to do so, I think there would be three advantages to this: - as explained by Nick, the existence of "except break" would strengthen the analogy with try/except/else and help people understand what the existing else clause after a loop is good for. There has been much debate over the else clause in the past, most prominently, a long discussion on this list back in 2009 (I recommend interested people to start with Steven D'Aprano's Summary of it at https://mail.python.org/pipermail/python-ideas/2009-October/006155.html) that shows that for/else is misunderstood by/unknown to many Python programmers. - in some situations for/except/else would make code more readable by bringing logical alternatives closer together and to the same indentation level in the code. Consider a simple example (taken from the docs.python Tutorial: for n in range(2, 10): for x in range(2, n): if n % x == 0: print(n, 'equals', x, '*', n//x) break else: # loop fell through without finding a factor print(n, 'is a prime number') There are two logical outcomes of the inner for loop here - a given number can be either prime or not. However, the two code branches dealing with them end up at different levels of indentation and in different places, one inside and one outside the loop block. This second issue can become much more annoying in more complex code where the loop may contain additional code after the break statement. Now compare this to: for n in range(2, 10): for x in range(2, n): if n % x == 0: break except break: print(n, 'equals', x, '*', n//x) else: # loop fell through without finding a factor print(n, 'is a prime number') IMO, this reflects the logic better. - it could provide an elegant solution for the How to break out of two loops issue. This is another topic that comes up rather regularly (python-list, stackoverflow) and there is again a very good blog post about it, this time from Ned Batchelder at https://nedbatchelder.com/blog/201608/breaking_out_of_two_loops.html. Stealing his example, here's code (at least) a newcomer may come up with before realizing it can't work: s = "a string to examine" for i in range(len(s)): for j in range(i+1, len(s)): if s[i] == s[j]: answer = (i, j) break # How to break twice??? with for/except/else this could be written as: s = "a string to examine" for i in range(len(s)): for j in range(i+1, len(s)): if s[i] == s[j]: break except break: answer = (i, j) break So much for the pros. Of course there are cons, too. The classical one for any syntax change, of course, is: - burden on developers who have to implement and maintain the new syntax. Specifically, this proposal would make parsing/compiling of loops more complicated. Others include: - using except will make people think of exceptions and that may cause new confusion; while that's true, I would argue that, in fact, break and exceptions are rather similar features in that they are gotos in disguise, so except will still be used to catch an interruption in normal control flow. - the new syntax will not help people understand for/else if except is not used; importantly, I'm *not* proposing to disallow the use of for/else without except (if that would ever happen it would be in the *very* distant future) so that would indeed mean that people would encounter for/else, not only in legacy, but also in newly written code. However, I would expect that they would also start seeing for/except increasingly (not least because it solves the "break out of two loops" issue) so they would be nudged towards thinking of the else after for/while more like the else in try/except/else just as Nick proposes it. Interestingly, there has been another proposal on this list several years ago about allowing try/else without except, which I liked at the time and which would have made try/except/]else work exactly as my proposed for/except/else. Here it is: https://mail.python.org/pipermail/python-ideas/2011-November/012875.html - as a result of previous discussions about for/else a section was added to PEP3099 saying: "The else clause in while and for loops will not change semantics, or be removed." However, the proposal here is not to change the else clause semantics, but add an additional except clause. So that's it and while I'm well aware of the slim chances of this getting legal syntax, I would still be happy to get feedback from you :) Best, Wolfgang

11 17

Add socket utilities for IPv4/6 dual-stack servers
by Giampaolo Rodola' March 5, 2017

March 5, 2017

Some years ago I started working on a patch for the socket module which added a couple of utility functions for being able to easily create a server socket, with the addition of being able to accept both IPv4 and IPv6 connections (as a single socket): https://bugs.python.org/issue17561 Given that not all OSes (Windows, many UNIXes) support this natively, I later submitted a recipe adding a "multiple socket listener" class. https://code.activestate.com/recipes/578504-server-supporting-ipv4-and-ipv6/ >From the user perspective, the whole thing can be summarized as follows: >>> sock = create_server_sock(("", 8000)) >>> if not has_dual_stack(sock): ... sock.close() ... sock = MultipleSocketsListener([("0.0.0.0", 8000), ("::", 8000)]) >>> Part of this work ended up being included into Tulip internals: https://github.com/python/cpython/blob/70d28a184c42d107cc8c69a95aa52a4469e7… ...and after that I basically forgot about the original patch. The other day I bumped into a case where I needed exactly this (on Windows), so here I am, trying to revamp the original proposal. To be clear, the proposal is to add 3 new APIs in order to avoid the low-level cruft needed when creating a server socket (SO_REUSEADDR, getaddrinfo() etc.) and being able to support IPv4/6 dual-stack socket servers in a cross-platform fashion: - socket.has_dual_stack() - socket.create_server_sock() - socket.MultipleSocketsListener Whereas the first two functions are relatively straightforward, MultipleSocketsListener is more debatable because, for instance, it's not clear what methods like getsockname() should return (because there are 2 sockets involved). One possible solution is to *not* expose such (all get*?) methods and simply expose the underlying socket objects as in: >>> socket.MultipleSocketsListener(...).socks[0].getsockname() On the other hand, all set* / write methods (setblocking(), setsockopt(), shutdown(), ...) can be exposed and internally they can simply operate against both sockets. Thoughts? -- Giampaolo - http://grodola.blogspot.com

2 1

Suggestion: Collection type argument for argparse where nargs != None
by David Mayo March 4, 2017

March 4, 2017

A friend of mine (@bcjbcjbcj on twitter) came up with an idea for an argparse improvement that I'd like to propose for inclusion. Currently, argparse with nargs=<anything but None> collects arguments into a list (or a list of lists in the case of action="append"). I would like to propose adding a "collection type" argument to the store and append actions and to add_argument, consisting of a callable that would be applied to the list of type-converted args before adding them to the Namespace. This would allow for alternate constructors (e.g. set), for modifying the list (e.g. with sorted), or to do checking of properties expected across all components of the argument at parse time. I've worked up a set of examples in this gist: https://gist.github.com/pobocks/bff0bea494f2b7ec7eba1e8ae281b888 And a rough implementation here: https://github.com/python/cpython/compare/master...pobocks:argparse_colltype I think this would be genuinely useful, and would require very little change to argparse, which should be backwards compatible provided that the default for the collection type is list, or None with list specified if None. Thank you all for your time in considering this, - Dave Mayo @pobocks on twitter, github, various others

1 0

add __contains__ into the "type" object
by 语言破碎处 March 4, 2017

March 4, 2017

where we use types? almost: isinstance(obj, T); # issubclass(S, T); Note that TYPE is SET; if we add __contains__ and __le__ into "type", then things become: obj in T; # S <= T; # if only not misleading to a total ordering example: def __getitems__(self, i): if i in Integral: ... elif i in slice: ... # Save "(,)". Really, I prefer to type "lambda:;" than "()". # I fail to modify the "type" object, since it is a C-object.

12 13

Thank's for your confirmation.
by Yander Martinez March 4, 2017

March 4, 2017

1 0

Delayed Execution via Keyword
by Joseph Hackman March 3, 2017

March 3, 2017

Howdy All! This suggestion is inspired by the question on "Efficient debug logging". I propose a keyword to mark an expression for delayed/lazy execution, for the purposes of standardizing such behavior across the language. The proposed format is: delayed: <expr> i.e. log.info("info is %s", delayed: expensiveFunction()) Unlike 'lambda' which returns a function (so the receiver must be lambda-aware), delayed execution blocks are for all purposes values. The first time the value (rather than location) is read, or any method on the delayed object is called, the expression is executed and the delayed expression is replaced with the result. (Thus, the delayed expression is only every evaluated once). Ideally: a = delayed: 1+2 b = a print(a) #adds 1 and 2, prints 3 # a and b are now both just 3 print(b) #just prints 3 Mechanically, this would be similar to the following: class Delayed(): def __init__(self, func): self.__func = func self.__executed = False self.__value = None def __str__(self): if self.__executed: return self.__value.__str__() self.__value = self.__func() self.__executed = True return self.__value.__str__() def function_print(value): print('function_print') print(value) def function_return_stuff(value): print('function_return_stuff') return value function_print(function_return_stuff('no_delay')) function_print(Delayed(lambda: function_return_stuff('delayed'))) delayed = Delayed(lambda: function_return_stuff('delayed_object')) function_print(delayed) function_print(delayed) Unfortunately, due to https://docs.python.org/3/reference/datamodel.html#special-lookup , this magic delayed class would need to implement many magic methods, as __getattribute__ is not _always_ called.

16 80

Re: [Python-ideas] Optional parameters without default value
by Victor Stinner March 3, 2017

March 3, 2017

In my code, I commonly use a NOT_SET singleton used as default value. I like this name for the test: if arg is NOT_SET: ... ;-) I use that when I want to behave differently when None is passed. And yes, I have such code. Victor Le 2 mars 2017 9:36 AM, "M.-A. Lemburg" <mal(a)egenix.com> a écrit : On 02.03.2017 09:03, Serhiy Storchaka wrote: > Function implemented in Python can have optional parameters with default > value. It also can accept arbitrary number of positional and keyword > arguments if use var-positional or var-keyword parameters (*args and > **kwargs). But there is no way to declare an optional parameter that > don't have default value. Currently you need to use the sentinel idiom > for implementing this: > > _sentinel = object() > def get(store, key, default=_sentinel): > if store.exists(key): > return store.retrieve(key) > if default is _sentinel: > raise LookupError > else: > return default > > There are drawback of this: > > * Module's namespace is polluted with sentinel's variables. > > * You need to check for the sentinel before passing it to other function > by accident. > > * Possible name conflicts between sentinels for different functions of > the same module. > > * Since the sentinel is accessible outside of the function, it possible > to pass it to the function. > > * help() of the function shows reprs of default values. "foo(bar=<object > object at 0xb713c698>)" looks ugly. > > > I propose to add a new syntax for optional parameters. If the argument > corresponding to the optional parameter without default value is not > specified, the parameter takes no value. As well as the "*" prefix means > "arbitrary number of positional parameters", the prefix "?" can mean > "single optional parameter". > > Example: > > def get(store, key, ?default): > if store.exists(key): > return store.retrieve(key) > try: > return default > except NameError: > raise LookupError Why a new syntax ? Can't we just have a pre-defined sentinel singleton NoDefault and use that throughout the code (and also special case it in argument parsing/handling)? def get(store, key, default=NoDefault): if store.exists(key): return store.retrieve(key) ... I added a special singleton NotGiven to our mxTools long ago for this purpose. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 02 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: [Python-ideas] Optional parameters without default value
by Victor Stinner March 3, 2017

March 3, 2017

I dislike the try/except NameError test to chevk if the parameter is set. Catching NameError is slow. What is the root issue? Function signature in help(func)? If yes, the solution can be a special value understood by inspect.signature(). Should it be possible to pass explicitly the special value? func(optional=special_value)? I think it's ok to allow it. The expected behaviour is to behave as func(). Victor Le 2 mars 2017 9:04 AM, "Serhiy Storchaka" <storchaka(a)gmail.com> a écrit : Function implemented in Python can have optional parameters with default value. It also can accept arbitrary number of positional and keyword arguments if use var-positional or var-keyword parameters (*args and **kwargs). But there is no way to declare an optional parameter that don't have default value. Currently you need to use the sentinel idiom for implementing this: _sentinel = object() def get(store, key, default=_sentinel): if store.exists(key): return store.retrieve(key) if default is _sentinel: raise LookupError else: return default There are drawback of this: * Module's namespace is polluted with sentinel's variables. * You need to check for the sentinel before passing it to other function by accident. * Possible name conflicts between sentinels for different functions of the same module. * Since the sentinel is accessible outside of the function, it possible to pass it to the function. * help() of the function shows reprs of default values. "foo(bar=<object object at 0xb713c698>)" looks ugly. I propose to add a new syntax for optional parameters. If the argument corresponding to the optional parameter without default value is not specified, the parameter takes no value. As well as the "*" prefix means "arbitrary number of positional parameters", the prefix "?" can mean "single optional parameter". Example: def get(store, key, ?default): if store.exists(key): return store.retrieve(key) try: return default except NameError: raise LookupError Alternative syntaxes: * "=" not followed by an expression: "def get(store, key, default=)". * The "del" keyword: "def get(store, key, del default)". This feature is orthogonal to supporting positional-only parameters. Optional parameters without default value can be positional-or-keyword, keyword-only or positional-only (if the latter is implemented). _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

lazy use for optional import
by Nicolas Cellier March 2, 2017

March 2, 2017

I have seen some interest into lazy functionality implementation. I wondered if it can be linked with optional import. PEP 8 <http://www.python.org/dev/peps/pep-0008/> authoritatively states: Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants. So, if we want to stick to PEP8 with non mandatory import, we have to catch the import errors, or jail the class or function using extra functionnality. Why not using the potential lazy keyword to have a nice way to deal with it? For example: lazy import pylab as pl # do nothing for now > > # do stuff > > def plot(*args): > pl.figure() # Will raise an ImportError at this point > pl.plot(...) > That way, our library will raise an ImportError only on plot func usage with an explicit traceback : if matplotlib is not installed, we will have the line where it is used for the first time and we will have the name of the faulty library.

5 4

suggestion about the sort() function of the list instance
by qhlonline March 2, 2017

March 2, 2017

Hi, all I have a suggestion that, the sort() member method of the list instance, should return the 'self' as the result of list.sort() call. Now list.sort() returns nothing, so that I can NOT write code like this: res = {item: func(item) for item in item_list.sort()} It feels bad. Regards!

13 16

PEP 8 coding style included in grammar ?
by Mathieu BEAL March 2, 2017

March 2, 2017

I was wondering why the PEP coding style ( https://www.python.org/dev/peps/pep-0008/) is not natively included in python grammar ? For instance, both, *function definition* and *class definition*, are using the same “NAME” token. ((see, https://docs.python.org/3/reference/grammar.html). classdef: 'class' NAME ['(' [arglist] ')'] ':' suite funcdef: 'def' NAME parameters ['->' test] ':' suite Then, we are using libraries like pyflake, flake8, pylint, to check the coding style. It seems useful to natively make distinction between NAME of classdef and NAME of funcdef, with something like: classdef: 'class' NAME_CLASS ['(' [arglist] ')'] ':' suite funcdef: 'def' NAME_FUNC parameters ['->' test] ':' suite NAME_CLASS: · Class name should normally use the CapWord convention; NAME_FUNC: · Function name should be lowercase, with words separated by underscore as necessary to improve readability, · mixedCase is allowed; STYLE_CAP_WORDS = r’([A-Z]{1}[a-z]+(_[A-Z]{1}[a-z]+)*)’ STYLE_UNDERSCORE_LOW_WORDS = r’([a-z]+(_[a-z]+)*)’ STYLE_MIXED_CASE = r’([a-z]{1,}([A-Z]{1}[a-z]+)*’) NAME_FUNC = STYLE_LOW_WORDS | STYLE_UNDERSCORE_LOW_WORDS | STYLE_MIXED_CASE NAME_CLASS = STYLE_CAP_WORDS | NAME_FUNC # naming convention for functions may be used in cases where the interface is documented and used primarily as a callable (pep8 source). I didn't find any information about this point in a previous post or elsewhere. Is there any reason for such a choice ? or is it a dark old discussion we never talk about ? Mathieu ------------- import re # Testing the first part of the class def, only. style_cap_words = r'^class ([A-Z]{1}[a-z]+(_?[A-Z]{1}[a-z]+)*)$' compiled_style_cap_words = re.compile(style_cap_words) status_samples = {True:['class Hello', 'class Hel_Lo', # ok if Hel and Lo are 2 words 'class HelLo', # same. ], False:['class HellO', 'class _He', 'class HEllo', 'class HE', 'class H', 'class h', 'class Hell_oo', 'class Hell_', 'class _hell', 'class H_E_L_L_O', 'class _H']} def is_matched(sample, pattern=compiled_style_cap_words): matched = pattern.match(sample) return matched is not None for status, samples in status_samples.items(): for sample in samples: is_correct = status == is_matched(sample) assert is_correct, '%s is not correct, required %s' % (sample, status)

6 5

__repr__: to support pprint
by 语言破碎处 March 1, 2017

March 1, 2017

Hi, everyone! Oftenly, __repr__ return "{type}({args}, {kws})" problem: 1) to call .format() myself is tedious I have seen someone do it wrong: "T(1,)"! I write a helper function myself, but to import such tiny function everywhere isnot good. 2) pprint cannot dive into string that from repr() To use pprint, I sometimes have to recursively turn objects into builtin containers: (type_name, args...) solution: allow __repr__ to return str OR tuple: (args, kws) Better, isn't it?

8 10

add a always turn on "assert"
by 语言破碎处 March 1, 2017

March 1, 2017

"assert" is good, but it is used as a guard frequently. We can make such usage legal by adding a new syntax: assert bool_expr, ExceptionType, True suggest reasons: 1) even "__debug__" turn off, assert is working assertion as guard. 2) to avoid boilerplate code I write code like this: if pred(....) or pred(....): raise ValueError('pred(....) or pred(....)') Simplifed: assert pred(...), ValueError, True # the above line will be printed when error. # I need not to copy the condition! 3) future: "assert bool_expr, ET, False" To aid static tool, like Proof System. Better document. For complicate algorithm, I actually add a invariant outline comment above every statement.

2 1

Re: [Python-ideas] for/except/else
by Wolfgang Maier March 1, 2017

March 1, 2017

On 01.03.2017 12:56, Steven D'Aprano wrote: > On Wed, Mar 01, 2017 at 10:37:17AM +0100, Wolfgang Maier wrote: > >> Now here's the proposal: allow an except (or except break) clause to >> follow for/while loops that will be executed if the loop was terminated >> by a break statement. > > Let me see if I understand the proposal in full. You would allow: > > > for i in (1, 2, 3): > print(i) > if i == 2: > break > except break: # or just except > assert i == 2 > print("a break was executed") > else: > print("never reached") # this is never reached > print("for loop is done") > > > as an alternative to something like: > > > broke_out = False > for i in (1, 2, 3): > print(i) > if i == 2: > broke_out = True > break > else: > print("never reached") # this is never reached > if broke_out: > assert i == 2 > print("a break was executed") > print("for loop is done") > > correct. > I must admit the suggestion seems a little bit neater than having to > manage a flag myself, but on the other hand I can't remember the last > time I've needed to manage a flag like that. > > And on the gripping hand, this is even simpler than both alternatives: > > for i in (1, 2, 3): > print(i) > if i == 2: > assert i == 2 > print("a break was executed") > break > else: > print("never reached") # this is never reached > print("for loop is done") > Right, that's how you'd likely implement the behavior today, but see my argument about the two alternative code branches not ending up together at the same level of indentation. > > > There are some significant unanswered questions: > > - Does it matter which order the for...except...else are in? > Obviously the for block must come first, but apart from that? > Just like in try/except/else, the order would be for (or while)/except/else with the difference that both except and else would be optional. > - How is this implemented? Currently "break" is a simple > unconditional GOTO which jumps past the for block. This will > need to change to something significantly more complex. > Yeah, I know that's why I listed this under cons. > - There are other ways to exit a for-loop than just break. Which > of them, if any, will also run the except block? > None of them (though, honestly, I cannot think of anything but exceptions here; what do you have in mind?) > >

1 0

a bad feature in Python syntax
by 语言破碎处 March 1, 2017

March 1, 2017

I'm bited once: >>> '' in {} == False False >>> ('' in {}) == False True # '' in {} == False ==>> ('' in {}) and ({} == False) ==>> False! I think only compare operations should be chained.

4 3

Expose a child factory using MappingProxyType in builtins
by Michel Desmoulin March 1, 2017

March 1, 2017

We have the immutable frozenset for sets and and tuples for lists. But we also have something to manipulate dict as immutable datastructures: >>> from types import MappingProxyType as idict >>> d = idict({'a':1, 'b':2, 'c':3}) >>> d['a'] = 4 Traceback (most recent call last): File "<ipython-input-3-c6f93d6278af>", line 1, in <module> d['a'] = 4 TypeError: 'mappingproxy' object does not support item assignment We could expose this as a built type to allow the last of the most important data structure in Python to be easily immutable.

7 7

add variable "__this_func__" inside all functions' locals
by 语言破碎处 March 1, 2017

March 1, 2017

How a function refer itself? def f(): f() # fine... really??? consider these situations: 1) decorator @xxx def f():... # how can I access the wrapped version? 2) replace by other def f(): f() # call second!! a = T(f) # to call first def f():... 3) refactor: rename function # first version def f():f() ------------------- # second version, wrap it def f(): log(); try: # to avoid indent _f_impl() except... def _f_impl(): _f_impl() # hadnot we rename it! solution: # add __this_func__ class C: @xxxx def f(self): self.f() __class__.f(self) __this_func__(self)

3 3

Re: [Python-ideas] unify usage of mutable and immutable objects
by 语言破碎处 March 1, 2017

March 1, 2017

> We are not going to change the implementation of sets so fundamentally, > mostly because having O(1) behaviors is so nice (add, remove, membership > check, etc). What I want are such interfaces (no matter with standard set): class IDynSet: # "Dyn" means it have methods like ipop_both def ipop_both(self): if hasattr(self, 'pop_mutable'): elem = self.pop_mutable() elif hasattr(self, 'ipop_immutable'): elem, self = self.ipop_immutable() else: raise return elem, self class IMutableDynSet(IDynSet): def pop_mutable(self): return self.pop() class IImutableDynSet(IDynSet): def ipop_immutable(self): elem, new = self.ipop() # ipop not pop return elem, new

1 0

unify usage of mutable and immutable objects
by 语言破碎处 March 1, 2017

March 1, 2017

Hi! I write a '<'-based immutable set class. But it is quit different from the standard set class. I wish collections.abc.Set be more friendly to immutable tree sets or Python add new syntax to unify such difference. good example: a = [] a += a # "a" is the original list a = () a += a # "a" is a new tuple bad example: a = set() a.add(1) # return None; "a" changed e = a.pop(); a = frozen_tree_set() a = a.add(1) # return another set; e, a = a.ipop() # return two objects instead of one! solution I used: a <<= 1 # <==> a = a.add(1) but "a.ipop()" .... my current solution is to write a wrapper class to turn immutable set into mutable set, but it is so noisy to box and unbox. solution that I wish: a :=.add(1) # "=." mimic "+="; return the result object e ~ a :=.pop() d[key] :=.add(1) # in dict if only python add following features: 1) modify the original object 1-0) 1) define: def .method(self):... # "." means "intend to modify self" # return any thing # to avoid immutable method # which was intended to return new object # leave self unchanged 2) invoke: r = a..method(); 1-1) ignore result # no matter what returned, discard it a.-.method(); # <==> [a..method(), None][-1] 1-2) return self # no matter what returned, return self a.>.method().>.method();# <==> [a..method(), a..method(), a][-1] 2) create new object 2-0) 1) define # intend to return (result, new object) def ^method():... 2)invoke: r, a' = a.^method(); 2-1) return other, discard result a.-^method().-^method();# <==> a.^method()[1].^method()[1]; 2-2) assign other to original variable a=^method(); # <==> a = a.^method()[1]; 3) unify both: a :=.method(); # if defined ".method" then "a..method();" # elif defined "^method" then "a = a.^method()[1];" # elif defined "method" then "a.method();" # else error r ~ a :=.method(); # if defined ".method" then "r = a..method();" # elif defined "^method" then "r, a = a.^method();" # elif defined "method" then "r = a.method();" # else error

4 7

Re: [Python-ideas] add __contains__ into the "type" object
by 语言破碎处 March 1, 2017

March 1, 2017

TYPE is not a collection! But conceptually it is a SET, set of all its possible instances. quot: "Types and Programming Languages (2002)(Benjamin C. Pierce)" [page 92] Chapter 8 Typed Arithmetic Expressions "a term t has type T" (or "t belongs to T," or "t is an element of T") An object obj has a type T or obj is a type T <==> obj belongs to T So, "obj in T" is fine. e.g. "1 in int" means 1 is in the whole number set. > > Note that TYPE is SET; > What does that mean? I don't understand. > > obj in T; > But obj is **not** in T, since T is a type, not a container. > "is-a" tests are not the same as "in" tests. They are completely > unrelated comparisons.

1 0