Mailman 3 August 2015 - Python-ideas

PEP-499: "python -m foo" should bind to both "__main__" and "foo" in sys.modules
by Cameron Simpson 10 Aug '15

10 Aug '15

I was recently bitten by the fact that the command: python -m foo pulls in the module and attaches it as sys.modules['__main__'], but not to sys.modules['foo']. Should the program also: import foo it pulls in the same module code, but binds a completely independent separate instance of it to sys.modules['foo']. This is counter intuitive; it is a natural expectation that "python -m foo" imports "foo" in a normal fashion. If the program modifies items in "foo", those modifications are not effected in "__main__", since these are two distinct modules. I propose that "python -m foo" imports foo as normal, binding it to sys.modules["__main__"] as at present, but that it also binds the module to sys.modules["foo"]. This will remove the disconnect between "python -m foo" and a program's internal "import foo". For people who are concerned that the modules .__name__ is "__main__", note that the module's resolved "offical" name is present in .__spec__.name as described in PEP 451. There are two recent discussion threads on this in python-list at: https://mail.python.org/pipermail/python-list/2015-August/694905.html and in python-ideas at: https://mail.python.org/pipermail/python-ideas/2015-August/034947.html Please give them a read and give this PEP your thoughts. The raw text of the PEP is below. It feels uncontroversial to me, but then it would:-) It is visible on the web here: https://www.python.org/dev/peps/pep-0499/ and I've made a public repository to track the text as it evolves here: https://bitbucket.org/cameron_simpson/pep-0499/ Cheers, Cameron Simpson <cs(a)zip.com.au> PEP: 499 Title: ``python -m foo`` should bind ``sys.modules['foo']`` in addition to ``sys.modules['__main__']`` Version: $Revision$ Last-Modified: $Date$ Author: Cameron Simpson <cs(a)zip.com.au> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 07-Aug-2015 Python-Version: 3.6 Abstract ======== When a module is used as a main program on the Python command line, such as by: python -m module.name ... it is easy to accidentally end up with two independent instances of the module if that module is again imported within the program. This PEP proposes a way to fix this problem. When a module is invoked via Python's -m option the module is bound to ``sys.modules['__main__']`` and its ``.__name__`` attribute is set to ``'__main__'``. This enables the standard "main program" boilerplate code at the bottom of many modules, such as:: if __name__ == '__main__': sys.exit(main(sys.argv)) However, when the above command line invocation is used it is a natural inference to presume that the module is actually imported under its official name ``module.name``, and therefore that if the program again imports that name then it will obtain the same module instance. That actuality is that the module was imported only as ``'__main__'``. Another import will obtain a distinct module instance, which can lead to confusing bugs. Proposal ======== It is suggested that to fix this situation all that is needed is a simple change to the way the ``-m`` option is implemented: in addition to binding the module object to ``sys.modules['__main__']``, it is also bound to ``sys.modules['module.name']``. Nick Coghlan has suggested that this is as simple as modifying the ``runpy`` module's ``_run_module_as_main`` function as follows:: main_globals = sys.modules["__main__"].__dict__ to instead be:: main_module = sys.modules["__main__"] sys.modules[mod_spec.name] = main_module main_globals = main_module.__dict__ Considerations and Prerequisites ================================ Pickling Modules ---------------- Nick has mentioned `issue 19702`_ which proposes (quoted from the issue): - runpy will ensure that when __main__ is executed via the import system, it will also be aliased in sys.modules as __spec__.name - if __main__.__spec__ is set, pickle will use __spec__.name rather than __name__ to pickle classes, functions and methods defined in __main__ - multiprocessing is updated appropriately to skip creating __mp_main__ in child processes when __main__.__spec__ is set in the parent process The first point above covers this PEP's specific proposal. Background ========== `I tripped over this issue`_ while debugging a main program via a module which tried to monkey patch a named module, that being the main program module. Naturally, the monkey patching was ineffective as it imported the main module by name and thus patched the second module instance, not the running module instance. However, the problem has been around as long as the ``-m`` command line option and is encountered regularly, if infrequently, by others. In addition to `issue 19702`_, the discrepancy around `__main__` is alluded to in PEP 451 and a similar proposal (predating PEP 451) is described in PEP 395 under `Fixing dual imports of the main module`_. References ========== .. _issue 19702: http://bugs.python.org/issue19702 .. _I tripped over this issue: https://mail.python.org/pipermail/python-list/2015-August/694905.html .. _Fixing dual imports of the main module: https://www.python.org/dev/peps/pep-0395/#fixing-dual-imports-of-the-main-m… Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

4 9

Briefer string format
by Mike Miller 09 Aug '15

09 Aug '15

Have long wished python could format strings easily like bash or perl do, ... and then it hit me: csstext += f'{nl}{selector}{space}{{{nl}' (This script included whitespace vars to provide a minification option.) I've seen others make similar suggestions, but to my knowledge they didn't include this pleasing brevity aspect. -Mike

35 163

String interpolation for all literal strings
by Eric V. Smith 08 Aug '15

08 Aug '15

In the "Briefer string format" thread, Guido suggested [1] in passing that it would have been nice if all literal strings had always supported string interpolation. I've come around to this idea as well, and I'm going to propose it for inclusion in 3.6. Once I'm done with my f-string PEP, I'll consider either modifying it or creating a new (and very similar) PEP. The concept would be that all strings are scanned for \{ and } pairs. If any are found, then they'd be interpreted in the same was as the other discussion on "f-strings". That is, the expression between the \{ and } would be extracted and searched for conversion characters and format specifiers. The expression would be evaluated, converted if needed, have its __format__ method called, and the resulting string inserted back in to the original string. Because strings containing \{ are currently valid, we'd have to introduce this feature with a __future__ import statement. How we transition to having this be the default interpretation of strings is up in the air. Guido privately suggested that it might be nice to also support the 'f' modifier on strings, to give the identical behavior. This way, you could start using the feature without requiring the __future__ import. While I'm not crazy about having two ways to achieve the same thing, I do think it might be nice to support interpolated strings without requiring the __future__ import. Eric. [1] https://mail.python.org/pipermail/python-ideas/2015-August/034928.html

28 118

Re: [Python-ideas] Make non-meaningful backslashes illegal in string literals
by random832＠fastmail.us 08 Aug '15

08 Aug '15

On Fri, Aug 7, 2015, at 23:45, Steven D'Aprano wrote: > > I challenge you to find *one* use in the wild. Just one. > > I'll take that challenge. Here are SEVEN uses for \v in the real world I should have better defined what I meant by "use". It has to be A) The actual \v escape, actually used, in a string literal in source code. I was asking for real-world uses of _the escape_. I'd guess about half of the ones you named actually are embodied in source code in a C-derived language somewhere - but probably not in Python. B) For the actual vertical tab function, rather than some other purpose the byte is being repurposed for. For other functions, the \v spelling obfuscates rather than illuminating. C) Code merely used for parsing string literal formats that themselves define \v, naturally, don't count. Mentioned for completeness, since otherwise these would technically satisfy the other two while accomplishing nothing useful. The vertical tab function is clearly defined as moving the cursor down (or the paper up) one or more lines to a predetermined position - e.g. a multiple of six lines, just as a tab conventionally takes you to the next multiple of eight columns, or a position that has been programmed into the device by use of other control mechanisms: VT - LINE TABULATION Notation: (C0) Representation: 00/11 VT causes the active presentation position to be moved in the presentation component to the corresponding character position on the line at which the following line tabulation stop is set.

1 0

Re: [Python-ideas] fork
by Sven R. Kunze 06 Aug '15

06 Aug '15

On 06.08.2015 17:52, Xavier Combelle wrote: > >> One quick comment: from my experience (mostly with other >> languages that are very different from Python, so I can't promise >> how well it applies here...), implicit futures without implicit >> laziness or even an explicit delay mechanism are not as useful as >> they look at first glance. Code that forks off 8 Fibonacci calls, >> but waits for each one's result before forking off the next one, >> might as well have just stayed sequential. And if you're going to >> use the result by forking off another job, then it's actually >> more convenient to use explicit futures like the ones in the stdlib. >> >> One slightly bigger idea: If you really want to pursue your >> implicit-as-possible design further, you might want to consider >> making the decorators replace the function with an object whose >> __call__ method just implicitly submits it to the pool. > > I added two new decorators for this. But they don't work with the > @ syntax. It seems like a well-known issue of Python: > > _pickle.PicklingError: Can't pickle <function fib_fork at > 0x7f8eaeb09730>: it's not the same object as __main__.fib_fork > > Would be great if somebody could fix that. > > > Sorry but I don't follow you have you any example that fail ? I fixed that, well, halfhearted: https://github.com/srkunze/fork/blob/2359265/fork.py#L47 and the following 3 lines. Remove that lines, and the tests using @cpu_bound_fork will fail. The reason for this is that the Pickle module is only capable of pickling module-level-named objects. Do you have a better fix? I would rather see that fixed in the Python internal decorator implementation than by me. Cheers, Sven

1 0

Re: [Python-ideas] Concurrency Modules
by Sven R. Kunze 05 Aug '15

05 Aug '15

Hi. I am back. First of all thanks for your eager participation. I would like to catch on on Steve's and Mark's examples as they seem to be very good illustrations of what issue I still have. Steve explained why asyncio is great and Mark explained why threading+multiprocessing is great. Each from his own perspective and focusing on the internal implementation details. To me, all approaches can now be fit into this sort of table. Please, correct me if it's wrong (that is very important): # | code lives in | managed by --+---------------+------------- 1 | processes | os scheduler 2 | threads | os scheduler 3 | tasks | event loop But the original question still stands: Which one to use? Ignoring little details like 'shared state', 'custom prioritization', etc., they all look the same to me and to what it all comes down are these little nasty details people try to explain so eagerly. Not saying that is a bad thing but it has some implications on production code I do not like and in the following I am going to explain that. Say, we have decided for approach N because of some requirements (examples from here and there, guidelines given by smart people, customer needs etc.) and wrote hundred thousand lines of code. What if these requirements change 6 years in the future? What if the maintainer of approach N decided to change it in such a way that is not compatible with our requirements anymore? From what I can see there is no easy way 'back' to use another approach. They all have different APIs, basically for: 'executing a function and returning its precious result (the cake)'. asyncio gives us the flexibility to choose a prioritization mechanism. Nice to have, because we are now independent on the os scheduler. But do we really ever need that? What is wrong with the os scheduler? Would that not mean that Mark better switches to asyncio? We don't know if we ever would need that in project A and project B. What now? Use asyncio just in case? Preemptively? @Steve Thanks for that great explanation of how asyncio works and its relationship to threads/processes. But I still have a question: why can't we use threads for the cakes? (1 cake = 1 thread). Not saying that asyncio would be a bad idea to use here, but couldn't we accomplish the same functionality by using threads? I think, after we've settled the above questions, we should change the focus from How do they work internally and what are the tiny differences? (answered greatly by Mark) to When do I use which one? The latter question actually is what counts for production code. It actually is quite interesting to know and to ponder over all the differences, dependencies, corner cases etc. However, when it actually comes down to 'executing a piece of code and returning its result', you end up deciding which approach to choose. You won't implement all 3 different ways just because it is great to see all the nasty little details to click in. On Thursday, July 9, 2015 at 11:54:11 PM UTC+1, Sven R. Kunze wrote: > > In order to make a sound decision for the question: "Which one(s) do I > use?", at least the following items should be somehow defined clearly > for these modules: > > 1) relationship between the modules > 2) NON-overlapping usage scenarios > 3) future development intentions > 4) ease of usage of the modules => future syntax > 5) examples

10 24

Reprs of classes and functions
by Serhiy Storchaka 05 Aug '15

05 Aug '15

This idea is already casually mentioned, but sank deep into the threads of the discussion. Raise it up. Currently reprs of classes and functions look as: >>> int <class 'int'> >>> int.from_bytes <built-in method from_bytes of type object at 0x826cf60> >>> open <built-in function open> >>> import collections >>> collections.Counter <class 'collections.Counter'> >>> collections.Counter.fromkeys <bound method Counter.fromkeys of <class 'collections.Counter'>> >>> collections.namedtuple <function namedtuple at 0xb6fc4adc> What if change default reprs of classes and functions to just full qualified name __module__ + '.' + __qualname__ (or just __qualname__ if __module__ is builtins)? This will look more neatly. And such reprs are evaluable.

7 9

Use the plus operator to concatenate iterators
by Grayson, Samuel Andrew 05 Aug '15

05 Aug '15

Concatenation is the most fundamental operation that can be done on iterators. In fact, we already do that with lists. [1, 2, 3] + [4, 5, 6] # evaluates to [1, 2, 3, 4, 5, 6] I propose: iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6]) There is some python2 code where: a = dict(zip('abcd', range(4))) isinstance(a.values(), list) alphabet = a.keys() + a.values() In python2, this `alphabet` becomes a list of all values and keys In current python3, this raises: TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values' But in my proposal, it works just fine. `alphabet` becomes an iterator over all values and keys (similar to the python2 case). Sincerely, Sam G

13 31

Re: [Python-ideas] fork - other approaches
by Sven R. Kunze 04 Aug '15

04 Aug '15

Thanks everybody for inspiring me with alternative ways of working with pools. I am very certain that any them will work as intended. However, they do not zero in 100% on my main intentions: 1) easy to understand 2) exchangeable (seq par) A) pmap It origates from map and allows easy exchangeability back and forth sequential and concurrent/parallel execution. However, I have to admit that I have difficulties to change all the 'for loops' to map (mentally as well as for real). The 'for loop' IS the most used loop construct in business applications and I do not see it going away because of something else (such as map). B) with Pool() It removes the need to close and join the pool which removes the visual clutter from the source code. That as such is great. However, exchangeability is clearly not given and the same issue concerning understandability like pmap arises. C) apply Nick's approach of providing a 'call_in_background' solution comes almost close to what would solve the issues at hand. However, it reminds me of apply (deprecated built-in function for calling other functions). So, a better name for it would be 'bg_apply'. All of these approaches basically rip the function call out of the programmer's view. It is no longer function(arg) but apply(function, arg) # or bg_apply(function, arg) # or bg_apply_many(function, args) I don't see this going well in production and in code reviews. So, an expression keyword like 'fork' would still be better at least from my perspective. It would tell me: 'it's not my responsibility anymore; delegate this to someone else and get me a handle of the future result'. Best, Sven ------------------------------------------------------------------------------------------------- FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSITÄT UND KOMFORT

4 6

A different format for PI?
by Abe Dillon 31 Jul '15

31 Jul '15

Is there a forum or something similar related to python-ideas? If there isn't, I think there should be. The mailing list format is restrictive. There's no good way to search past discussions and the digests I get are disorganized and difficult to follow. I'd like to contribute, but I don't know if my ideas are topics that have already been discussed in depth or if they're actually new. -Abe Dillon

6 7