Mailman 3 March 2016 - Python-ideas

Control Flow - Never Executed Loop Body
by Sven R. Kunze 06 Apr '16

06 Apr '16

Hi Python-ideas, this is a follow up from an thread of Python-List: https://mail.python.org/pipermail/python-list/2016-March/705205.html What do you think about the following proposal? *Abstract* Control structures such as "for", "while" and "try" feature some various additional clauses being executed under certain conditions. This proposal adds another extra clause to "for" and "while" in order to provide an easy way to discover whether the body of a loop as never been executed at all. *Motivation** *Coming from the Web application development, we periodically can see a benefit of a hypothetical "empty" clause of Python's for loop like this: for item in my_iterator: # do per item empty: # do something else The same goes for "while" although there the choice of the name "empty" might be questionable. "empty" refers to the involved iterable which is not really existing in case of "while". However, there is not need of introducing inconsistencies among the loop constructs as both feature the "else" clause as well. Better suggestions are welcome; also see the keyword section. *Keyword* Keywords under consideration have been in (an attempt of dispassionate) order of preference: 1) empty -> most obvious, most people responded with a solution to solve the intended problem 2) else -> already taken but to some would be the preferred one 3) or -> as alternative if a new keyword is too much 4) except -> as alternative if a new keyword is too much 5) instead -> less obvious and new keyword *Use-Cases* 1) As humans mostly have a hard time to discover the difference between *nothing**as error* and *nothing**as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case. 2) Single way and no boilerplate of handling empty loops. As the discussion on Python-List showed are there quite some number of different ways of achieving this kind of functionality. 3) Less errors in development and maintenance. Some of the proposed solutions would not work in some circumstances (either during development or depending on the type of the items) which can hide errors. *Implementation* To be done. *Maintainer* Me. *Issues** *People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. *Related Work** *This proposal is partly inspired by template engines like Django or jinja2 which both already provide such facility. Django: {% for item in my_iterator %} ... {% empty %} ... {% endfor %} jinja2: {% for item in my_iterator %} ... {% else %} ... {% endfor %} Best, Sven

29 144

`to_file()` method for strings
by Nick Eubank 04 Apr '16

04 Apr '16

As a social scientists trying to help other social scientists move from language like R, Stata, and Matlab into Python, one of the behaviors I've found unnecessarily difficult to explain is the "file.open()/file.close()" idiom (or, alternatively, context managers). In normal operating systems, and many high level languages, saving is a one-step operation. I understand there are situations where an open file handle is useful, but it seems a simple `to_file` method on strings (essentially wrapping a context-manager) would be really nice, as it would save users from learning this idiom. Apparently there's something like this in the Path library ( https://docs.python.org/dev/library/pathlib.html#pathlib.Path.write_text) , but I suspect most people have no idea about that method (I've been doing python for years and this has always been a personal frustration, and I've asked several others for better options and no one had one to offer), and it seems like it would make much more sense as a string method. If someone has a string they want to save to disk, I can't imagine anyone looking in the Path library. I respect the desire to avoid bloat -- the context manager or open/close idiom has just felt unnecessarily complicated (dare I say unpythonic?) for a common task. Thanks! Nick

24 73

Fwd: Make parenthesis optional in parameterless functions definitions
by Mahan Marwat 03 Apr '16

03 Apr '16

---------- Forwarded message ---------- From: Mahan Marwat <mahanmarwat(a)gmail.com> Date: Thu, Mar 31, 2016 at 9:09 PM Subject: Make parenthesis optional in parameterless functions definitions To: python-ideas(a)python.org Hi, I have an idea of making parenthesis optional for functions having no parameters. i.e def greet: # note the missing parenthesis print('hello') The less awkward characters we have, the more readable our code will be (Beautiful is better then ugly). Some people argued that function definition with parenthesis seems to them natural. But actually it seems to us natural, because we have been used to it a-lot. IMHO parenthesisless functions definitions are natural and readable. In Python we have already adopted this. i.e parenthesis are optional in `if` statements, in class definition, in tuple etc if x == 1: # parenthesis are optional here pass class Greet: # now we don't explicitly inherit from `object` pass Tuple = 1, 2, 3 # parenthesis are optional here too Please, give your opinion. Thanks, Adnan Khan

16 34

Working with Path objects: p-strings?
by Koos Zevenhoven 02 Apr '16

02 Apr '16

So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path. On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze <srkunze(a)mail.de> wrote: >> On 24.03.2016 22:06, Koos Zevenhoven wrote: >> >> Or even better, that you could do p"filename.txt", which would give you a >> Path string object. Has this been discussed? >> > > Interesting. I had this thought half a year ago. Quite recently I mentioned > this to Andrew in a private conversation. > Yeah, I first thought about this at some point last summer when I had written the little np package for creating numpy arrays as np[3,4,5] (which I am planning to polish and maybe advertise), probably also inspired by the f-string discussions on this list. And now that you asked me about dealing with Paths... Anyway, as you say, if different people have thought about this independently, maybe its worth discussing. > p'/etc/hosts' would make a perfect path which subclassed from str. > Yeah, and regarding the "`to_file()` method for strings" thread, one could simply do p"targetfile.txt".write_text(string_to_write) using the write_text method introduced to Path objects in 3.5. [And of course there's also read_text(), write_bytes(...) and read_bytes()] One downside to p-strings like this is that they don't help you when you have the path in a plain str variable (i.e. not a literal), which has led me to think Path perhaps should be in builtins. But one thing that just occured to me now is that, for instance in interactive sessions, you could do p"" / path_as_str instead of `import pathlib; pathlib.Path(path_as_str)`. Not sure if this is the most beautiful thing ever seen, but it would work for quick scripting at least. > The p-string idea has not been discussed in the PEP I think. The subclassing > thing however was and I think its resolution was a mistake. The explanation, > at least from my point of view, is a bit weird. > Yeah, having Path subclass str would immediately solve a lot of compatibility issues. One question is how large a part of those issues would be solved by just making the builtin open(...) and a few places here and there in the stdlib accept Path objects. Maybe there are implementation issues too, although having them lead to bad decisions would be sad. Anyway, I think the question of whether Path should subclass str is partly separate from the p-string concept. After all, b'byte strings' are not subclassed from str either. - Koos

26 141

Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?]
by Paul Moore 31 Mar '16

31 Mar '16

On 31 March 2016 at 10:46, Sven R. Kunze <srkunze(a)mail.de> wrote: > On 31.03.2016 11:15, Paul Moore wrote: >> >> Any suggestions for a better design? It might be nice for someone to >> write a blog entry or posting somewhere on "how to upgrade your 3rd >> party library to support pathlib" and one of the harder questions will >> be "how do I decide whether to return a string or a path?" If we had a >> good recommendation for this, it might help adoption. > > Good point. This discussion has been long already so it might make sense to > write something down in a concise manner. > > I am not sure if I can make a good suggestion here because I am still > trapped at the point where I need to sort out if a path more like a dict or > another structured datatype, or if it is more a monolithic object. That will > be the next blog post topic for me. I'm only talking here about supporting the *existing* pathlib object. Discussing alternative path object options doesn't help promote adoption of the stdlib path object, which is in all honesty the only thing I'm interested in. > "how do I decide whether to return a string or a path?" << very good > question btw; my initial thought would be; make two functions (maybe the > same name in two different namespaces), so it's crystal clear what this > function takes a arguments and what it returns. But I need to think more > about this. So, coming at this from the perspective of a library developer: 1. Which interface (the str or the path one) gets the "obvious" name? Backward compatibility pretty much dictates it'd be the str version. 2. How do I support Python 2.7? And Python 3.3/3.4 (where the ".path" attribute doesn't exist)? 3. What's my transition plan for the future when I deprecate the str interface and only support path objects? 4. What will my users have to do if they want to use Path objects in their code? And how do I minimise their pain in doing so? So it's a pretty hard issue, to think about adding pathlib support if you *return* paths. (It's essentially trivial if you only *consume* paths - just add "patharg = getattr(patharg, 'path', patharg)" at the top of your APIs, and you're done - apart from testing, documentation, etc, of course :-)) But without library adoption, *users* bear the pain of adapting everything to Path objects. And that's a real barrier to adoption. Maybe once the stdlib uses path objects more consistently (there's a *lot* of modules that could do with accepting Path or str arguments equally, and one or two like tempfile that create path names), it will be easier to argue for wider pathlib support in 3rd party libraries. Paul

7 10

Add a developer mode to Python: -X dev command line option
by Victor Stinner 30 Mar '16

30 Mar '16

Hi, When I develop on CPython, I'm always building Python in debug mode using ./configure --with-pydebug. This mode enables a *lot* of extra checks which helps me to detect bugs earlier. The debug mode makes Python much slower and so is not the default. "python3" in Linux distributions is a binary compiled in release mode. When they also provide a binary compiled in debug mode, you will probably have issues to use your existing C extensions, since they all of them are compiled in release mode which is not compatible (you must recompile C extensions in debug mode). I propose to add a "development mode" to Python, to get a few checks to detect bugs earlier: a new -X dev command line option. Example: python3.6 -X dev script.py Checks enabled by this flag must: * Not flood the stdout/stderr (ex: don't write one message per second) * Have a low overhead in term of CPU and memory (ex: max 10%) I propose to enable: * Show DeprecationWarning and ResourceWarning warnings: python -Wd * Show BytesWarning warning: python -b * Enable Python assertions (assert) and set __debug__ to True: remove (or just ignore) -O or -OO command line arguments * faulthandler to get a Python traceback on segfault and fatal errors: python -X faulthandler * Debug hooks on Python memory allocators: PYTHONMALLOC=debug For example, I consider that enabling tracemalloc is too expensive (CPU & memory) and must not be enabled in -X dev. I wrote a proof-of-concept: if -X dev is used, executable Python once again with more parameters. Basically, replace: python3.6 -X dev ... with PYTHONMALLOC=debug python3.6 -Wd -b -X faulthandler ... The list of checks can be extended later. For example, we may enable the debug mode of asyncio: PYTHONASYNCIODEBUG=1. The scope of the "developer mode" is unclear to me. Many modules (ex: asyncio) already have a debug mode. Would it be a bad idea to enable *all* debug modes of *all* modules? For example, http.client has a debug level: do you expect debug traces of the HTTP client when you develop your application? IMHO the scope must be well defined: don't modify modules of the stdlib, only enable more debug flags in the Python core (warnings, unicode, memory allocators, etc.). Maybe we even a need -X dev=strict which would be stricter: * use -Werror instead of -Wd: raise an exception when a warning is emitted * use -bb instead of -b: get BytesWarning exceptions * Replace "inconsistent use of tabs and spaces in indentation" warning with an error in the Python parser * etc. Again, this list can be extended later. Or maybe we need multiple level to control the quantity of debug traces, warnings, ... written into stdout/stderr? In my experience, the problem with converting warnings to errors is that you don't control the code of your whole application. It's common that third party modules raise DeprecationWarning, ResourceWarning, etc. Even some modules of the Python stdlib raise DeprecatingWarning! For example, I recall that distutils raises a DeprecationWarning on Python 3.4 when importing the imp module. "Similar" idea in other programming languages: * Perl: http://perldoc.perl.org/strict.html * PHP: https://secure.php.net/manual/fr/function.error-reporting.php Victor

3 3

URLs/URIs + pathlib.Path + literal syntax = ?
by Koos Zevenhoven 30 Mar '16

30 Mar '16

The 'Working with Path objects: p-strings?' thread spawned a discussion about how URLs (or more generally URIs) and Paths should work together. I suggest we move that discussion to this new thread. The concept is 'explained' below in this email and quotes, but a little bit of discussion happened in the other thread too. While I think that the decisions about p-strings (or a-strings for addresses or whatever they should be) should keep URIs in mind, it is premature to add the Path+URI fusion into the stdlib. I agree with Paul Moore that this URL stuff should be on PyPI first. It could even be library that monkey patches pathlib to accept URIs. Or a URI library that instantiates Path objects when appropriate. Then there could be a smooth transition into the stdlib some day. See all the stuff below: On Tue, Mar 29, 2016 at 11:44 AM, Sven R. Kunze <srkunze(a)mail.de> wrote: > > On 27.03.2016 00:51, Koos Zevenhoven wrote: >> >> OT: >> >> To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths. The difference between "http://domain.xyz/" and "C:\\" is not huge. I also think there should be a Python type (stdlib or builtin), which handles JSON objects nicer than dicts do and has its own literal >> > > That even occurred to me after we talked about the p-string (mainly because I am working in this field, so I basically need both file paths and URIs). > Again, that you say you thought about it too perhaps means it's worth discussing :). > > Just for the record: "Path" might not be the most correct wording. There is a "file://" scheme which identifies locally located files. So, paths are basically a subset of URLs speaking functionality-wise. Thus, a better/more generic name would be "URL", "URI", "Link" or the like in order to avoid confusing of later generations. However, I think I could live with Path. > Yes, these are concerns that should be considered if/when deciding whether to make URI/URLs a subclass of Path or the other way around, or something else. Anyway, since Path(...) already instantiates different subclasses based on the situation, having it instantiate a URI in some cases would not be completely unnatural. As suggested by Stephen, I've been looking into RFC 3986 as a whole, and it seems that making instantiating both URIs and fs paths from p-strings does not seem completely impossible. Some points below (you can skip past them if you have to, there's more general discussion at the end): - Only some URIs (or even URLs) can be reliably distinguished from file paths. However, those that contain '://' could be automatically turned into URI objects by p-strings [or Path(...)]. I suspect that would cover the majority of use cases. (The unambiguous cases would be exactly those URIs that contain an 'authority' component -- these always begin with 'scheme://' while other's don't) - If we want allow URIs without an 'authority' component, like mailto:someone@domain.com', they should be explicitly instantiated as URI objects. - Some terminology: There are indeed 'URI's and 'relative references'. Relative references are essentially the URI-equivalent of relative paths. Then there are 'URI references' which can be either 'URIs' or 'relative references' (kinda like if you consider general paths that can be absolute or relative paths, as is done in pathlib). - Instantiating relative URI references with Path(...) or p-strings may cause issues, because they might get turned into Windows paths and the like. It does seem like this could be worked around by for instance making another class like "RelativePath" or "RelativeRef", but there are some questions about when/how these should be instantiated. This may lead to a need slight backwards incompatibilities if implemented within pathlib. - "Queries" like '?this=that' after the path component have a special role in URIs, but in file system paths they can be parts of the file (or even directory) name. This might again be ambiguous when using relative paths / references. This could perhaps be dealt with by requiring more explicit handling when joining relative paths / references together. - "Fragments" like '#what'. This is essentially the same issue as with queries above and should be solved the same way. Anyway, both may be present at the same time. - '..' and '.' in relative paths / references. In URIs, there's a difference between 'scheme://foo/bar/' and 'scheme://foo/bar'. Merging the relative reference './baz' to the former gives 'scheme://foo/baz' while merging it to the latter gives 'scheme://foo/bar/baz'. I kinda wish the same thing was the standard with filesystem paths too. - Percent encoding of URIs: quite obvious -- should not be done before it is unambiguous that we deal with an URI. Perhaps it should be done only when the resource is accessed or when the URI is exported to a plain str or bytes etc. I suppose this is matter of what we would want in the repr. - I may still have missed or forgotten something. So, also with paths, especially relative ones, a library should "resist the temptation to guess", and carry around all the information until the context becomes unambiguous. For instance, when merging a relative reference with an explicit URI, the ambiguities about ?query and #fragment and about resolving the merged path disappear. > Another thought: requesting URLs. Basically the same as p'/etc/hosts'.write_text(secret). It's really important to have a dead simple library which is able to work with URLs. So, if I could do: > > p'https://mysite.com/{page}'.get() > Good idea. When I suggested extending Paths (and p-strings) to work with URLs, I indeed meant that it would be an instance of (a subclass of) Path, so that you do the same as with filesystem path objects: p'https://mysite.com/somepage.html'.read_text() or (p'https://mysite.com' / page).read_text() But who knows what we might end up with if we go down this path. An I mean a metaphorical path here, not necessarily Path :). Whatever it is, it probably can't be added to the stdlib right away. Still, we could take some measures regarding the language and stdlib now, to prepare for the future. -Koos

10 20

Re: [Python-ideas] Working with Path objects: p-strings?
by Nathan Schneider 30 Mar '16

30 Mar '16

On Wed, Mar 30, 2016 at 4:25 AM, Random832 <random832(a)fastmail.com> wrote: > On Tue, Mar 29, 2016, at 17:39, Ethan Furman wrote: > > On 03/29/2016 02:30 PM, Greg Ewing wrote: > > > Paul Moore wrote: > > >> > > >> P(wherever)/youwant/togo > > > > > > If P were a suitable seed object, this could be > > > > > > P/wherever/youwant/togo > > > > An interesting idea, but how would you differentiate between relative > > and absolute paths? > > Path.ROOT/absolute/path > Path.DOT/relative/path > How about: Path/absolute/path Path%relative/path [think: % as the Unix shell prompt] I also like the idea of being able to do Path/~homesubdir/path as an alias for Path/'~'/homesubdir/path Apart from the above, the user would have to know to quote special path characters like ~ and . and trailing / to avoid a syntax error: Path/'~' Path/'..'/relative/path Path/absolute/path/'/' [in rare cases where the trailing slash is necessary] Nathan >

1 0

Re: [Python-ideas] Integrate some itertools into the Python syntax
by Chris Barker 29 Mar '16

29 Mar '16

I haven't seen enough in this conversation to persuade me that we need more built ins, but: I recently taught Intro to Python using py3 for the first time recently, and really noticed that while the most obvious difference (at the beginning level) is that print is a function now, that is only annoying, but didn't confuse anyone at all. (and Unicode because a non-issue -- yeah!) However, where I found things changed in a conceptual way is the movement from sequences to iterables in many places: zip() dict.keys(), etc.... I found that my approach of focusing a lot on sequences -- you can iterate over them, you can slice them, you can.... seems out of date now, and I think lead to confusion as folks got a bit more advanced. Py2 was all about sequences -- py3 is all about iterables, and I think I need to start teaching it that way -- start with iterables, and introduce sequences as a special case of iterables, rather than starting from sequences, and introducing iterables as a more advanced topic. I understand the focus on iterables -- for performance reasons if nothing else, but I think it does, in fact, make the language more complex. and I think we should think about making the language more iterable-focused. This certainly comes up with this discussion -- if Python is all about iterables, then some of itertools should be more discoverable and obvious. And it came up in another recent thred about a mechanism for doing something after a for loop that didn't loop -- in that case, for sequences, the idiom is obvious: if seq: do_the_for_loop else: do_somethign else since the loop wont have run. But when you plug in an arbitrary iterable into that, it doesn't work, and there is no easy, obvious, and robust idiom to replace that with. I don't know that that particular issue needs to be solved, but it makes my point -- there is much to be done to make python really about iterables, rather than sequences. For my part, I would like to see iterables look more like sequences -- I know it's going to be inefficient in many cases to index an iterable, but it i really less efficient that wrapping list() around it? One tiny example: I often need to parse text files, which I used to do with code like: for line in the_file.readlines(): .... then, particulary when debugging, I could do: for line in the_file.readlines()[:10]: .... and just work with the first ten lines.... but now that file objects ar iterable, I can do: for line in the_file: .... much nicer! but it breaks when I want to debug and try to do: for line in the_file[:10]: ... arrgg! files are not indexable!. (and by the way, not only for testing, but also when you really do want the next ten lines from the file -- maybe it's a header, or...) Sure, I have plenty of ways to work around this, but frankly, they are all a bit ugly, even if trivial. So maybe there should be some ability to index / slice iterables? But aside from that -- just the idea that looking at how to make iterable a more "natural" part of the language is a good thing. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

10 22

custom predicate for all() & any()
by titouan dk 29 Mar '16

29 Mar '16

Hi, I was wondering if the builtins functions all() & any() could ever accept a custom predicate: all(iterable, predicate=bool) any(iterable, predicate=bool) Something like Lodash.js functions: every(), ... ## Example 1 ages = [21, 13, 18] major = 18..__le__ allowed = all(ages, major) ## Example 2 people = [ {"name": "Mike", "age": 42}, {"name": "Josh", "age": 17} ] major = 18..__le__ allowed = all(people, {"age": major}) ## Consistency drawbacks The filter() function takes its arguments in the reverse order: filter(predicate_or_none, iterable) I would have write this function like this... filter(iterable, predicate=None) ... but this is not the case, anyway. ## Your opinion? What do you think about this? Good idea, could be a quick-win? Or requiring a lot of efforts for nothing? What about consistency drawbacks? Thanks for reading, Titouan de Kerautem

9 10