Mailman 3 October 2014 - Python-ideas

metamodules (was: Re: Idea to support lazy loaded names.)
by Nathaniel Smith 10 Oct '14

10 Oct '14

On Tue, Oct 7, 2014 at 5:12 AM, Andrew Barnert <abarnert(a)yahoo.com.dmarc.invalid> wrote: > On Oct 6, 2014, at 20:51, Terry Reedy <tjreedy(a)udel.edu> wrote: > >> So a 'solution' might be to make modules be instances (but with no __new__ or __init__) of a module metaclass, so that module dicts could act like class dicts with respect to descriptors. I have no idea how much code this would break ;-). > > Didn't we just have this discussion a few weeks ago, in the context of making lazy loading of subpackages easier to implement? Yeah, and having had some time to think about that discussion and do some prototyping, I'm going to argue below that allowing assignment to module instances' __class__ really is the best path forward. (For those who find the below TLDR, check this out instead: https://github.com/njsmith/metamodule) > IIRC, the not-obviously-unreasonable options suggested were: Great list! I've rearranged a bit to make my argument clearer. > 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. This is very similar to the current approach of having __init__.py reassign sys.modules[__name__]. The advantages are: - It gives a way to enforce the rule that you have to do this assignment as the very first thing inside your module, before allowing arbitrary code to run (e.g. by importing other modules which might recursively import your module in turn, and access sys.modules[__name__] before you've modified it). - Because __new__.py would run *before* the It avoids the headache of having to juggle two module objects, one of whose __dict__'s is already being used as the execution environment for the code that is trying to do the switcheroo. But: - It's a pretty complicated way to accomplish the stated goals. - The restriction to packages is unfortunate. - The backcompat story is terrible -- faking __new__.py support in old versions of python would be really difficult, and the main reason I care about this stuff in the first place is because I want to be able to e.g. deprecate module attributes that are in the public API of old, widely-used packages. It will be many years before such packages can require 3.5. So I think we should probably try to make the existing sys.modules[__name__] = ... strategy work before considering this. > 2) Analogy with metaclass= (or with 2.x __metaclass__): If a module (or a package's __init__.py) does some new syntax or magic comment before any non-comment code, it can specify a custom type in place of ModuleType (not sure how that type gets imported and made available). I don't see any way to solve this import problem you refer to at the end -- in most cases the code implementing the metamodule type will be defined *inside* the module/package which wants to use the metamodule, so we have a chicken-and-egg problem. > 4) Make it easier to write import hooks for this special purpose. This has the same problem as the previous -- who imports the importer? > 5) Make it easier for a module to construct a ModuleType-subclass instance with a copy of the same dict and replace itself in sys.modules. So, trying to *copy* the dict is just not going to work. Consider the package foo, with a foo/__init__.py that looks like: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 from .submod import b c = 3 sys.modules[__name__].__dict__.update(orig_dict) and where foo/submod.py looks like: import foo b = foo.a + 1 def c(): return foo.a + 2 This won't work, because at the time we import .submod, sys.modules["foo"].__dict__ does not contain an entry for "a" -- only the original module's dict has that. There are a bunch of different ways we could try writing our __init__.py. We might try putting the sys.module assignment at the end: a = 1 from .submod import b, c d = 4 orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__.update(orig_dict) Now when .submod re-imports the top-level module, it ends up with a reference to the original module object, which has an "a" entry, so the definition of "b" works. But, now .submod.foo will continue to refer to the original module object, even after we substitute in the metamodule object. If we do 'foo.a = 5' later on, then foo.c() will continue to use the original binding of 'a'; this mutation will be invisible to it. I guess the only way to make it work in this case is to do multiple copies, one before every nested-import: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 sys.modules[__name__].__dict__.update(orig_dict) from .submod import b, c d = 4 sys.modules[__name__].__dict__.update(orig_dict) ...but this is incredibly ugly and error-prone. What we really want to do instead is to make our new metamodule object refer directly to the original module's __dict__: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__ = orig_dict a = 1 from .submod import b, c d = 4 That way they will always be in sync. This looks like it should work great! But it has a few problems: - Trying to assign to a module's __dict__ attribute raises "TypeError: readonly attribute". - So I actually implemented a fix for that, and ran into a new problem: module's take jealous ownership of their __dict__. In particular, they assume that they are when they are deallocated they should wipe their dict clean (https://www.python.org/doc/essays/cleanup/). Obviously this is bad for us because we are still using that dict! - Also, in modern Python module objects contain more state besides __dict__ -- in particular, PEP 3121-related state. There's no public API to get at this. - Possibly future version of Python will add more state fields again, who knows. The easiest way to solve all these problems is to *swap* all of the internal fields between the old module object and the new metamodule object. This can be done hackishly using ctypes; this requires knowing about CPython's struct layouts, but that's okay for prototyping and for backwards compatibility hacks (which only have to support specific known versions). To do it non-hackishly, I was at first thinking that we should provide an official API for swapping module object states. But then I realized that at that point, we're basically back to... > 3) Analogy with re-classing object instances: Just allow modules to set __class__ during execution (or after, if you want). Variations on this include allowing that for all non-heap types, or even getting rid of the concept of non-heap types. ...this proposal after all. And allowing __class__ assignment on modules strikes me as more asthetic than having a sys.swap_module_contents function. I implemented a prototype of this functionality here: https://github.com/njsmith/metamodule The implementation is here: https://github.com/njsmith/metamodule/blob/master/metamodule.py That file has 3 parts: - A fancy metamodule class that handles implicit-imports and warn-on-attribute-access. - A utility function for setting up a metamodule; it tries __class__ assignment, and if that doesn't work falls back on ctypes hackery. - The aforementioned ctypes hackery. It's pretty ugly, but if we add __class__ assignment then it will become unnecessary on future Python versions, woohoo! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

4 6

issubclass(collections.OrderedDict, collections.Sequence)
by Ram Rachum 09 Oct '14

09 Oct '14

Hi guys, I was just going to test whether a mapping is dict-like or an OrderedDict-like object by doing this: isinstance(x, collections.Sequence) But then I checked and saw that this is False: issubclass(collections.OrderedDict, collections.Sequence) is False Why isn't an OrderedDict a Sequence?

13 43

Better integration of multiprocessing with asyncio
by Dan O'Reilly 09 Oct '14

09 Oct '14

I think it would be helpful for folks using the asyncio module to be able to make non-blocking calls to objects in the multiprocessing module more easily. While some use-cases for using multiprocessing can be replaced with ProcessPoolExecutor/run_in_executor, there are others that cannot; more advanced usages of multiprocessing.Pool aren't supported by ProcessPoolExecutor (initializer/initargs, contexts, etc.), and other multiprocessing classes like Lock and Queue have blocking methods that could be made into coroutines. Consider this (extremely contrived, but use your imagination) example of a asyncio-friendly Queue: import asyncio import time def do_proc_work(q, val, val2): time.sleep(3) # Imagine this is some expensive CPU work. ok = val + val2 print("Passing {} to parent".format(ok)) q.put(ok) # The Queue can be used with the normal blocking API, too. item = q.get() print("got {} back from parent".format(item)) def do_some_async_io_task(): # Imagine there's some kind of asynchronous I/O # going on here that utilizes asyncio. asyncio.sleep(5) @asyncio.coroutine def do_work(q): loop.run_in_executor(ProcessPoolExecutor(), do_proc_work, q, 1, 2) do_some_async_io_task() item = yield from q.coro_get() # Non-blocking get that won't affect our io_task print("Got {} from worker".format(item)) item = item + 25 yield from q.coro_put(item) if __name__ == "__main__": q = AsyncProcessQueue() # This is our new asyncio-friendly version of multiprocessing.Queue loop = asyncio.get_event_loop() loop.run_until_complete(do_work(q)) I have seen some rumblings about a desire to do this kind of integration on the bug tracker (http://bugs.python.org/issue10037#msg162497 and http://bugs.python.org/issue9248#msg221963) though that discussion is specifically tied to merging the enhancements from the Billiard library into multiprocessing.Pool. Are there still plans to do that? If so, should asyncio integration with multiprocessing be rolled into those plans, or does it make sense to pursue it separately? Even more generally, do people think this kind of integration is a good idea to begin with? I know using asyncio is primarily about *avoiding* the headaches of concurrent threads/processes, but there are always going to be cases where CPU-intensive work is going to be required in a primarily I/O-bound application. The easier it is to for developers to handle those use-cases, the better, IMO. Note that the same sort of integration could be done with the threading module, though I think there's a fairly limited use-case for that; most times you'd want to use threads over processes, you could probably just use non-blocking I/O instead. Thanks, Dan

4 10

Enhanced stat_result objects (was: Better stdlib support for Path objects)
by Paul Moore 08 Oct '14

08 Oct '14

On 6 October 2014 18:47, Barry Warsaw <barry(a)python.org> wrote: > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. The other thing that bugs me whenever I try to use pathlib is the non-caching behaviour of the stat-related methods. I know why this is as it is, and I agree with the reasons, but for some reason, I still find that I expect code like p = Path(...) if not p.exists(): print("Nonexistent") elif p.is_dir(): print("Directory") elif p.is_file(): print("File") else: print("Something else") to only call stat once - even though I *don't* expect that of the equivalent os.path based code. I would find it really useful if stat_result objects supported the various is_XXX methods from pathlib. Then I could write code like: p = Path(...) st = p.stat() if st.exists(): ... elif st.is_dir(): ... which explicitly shows that stat is only called once, but doesn't involve verbose and ugly code like if stat.S_ISDIR(st.st_mode): ... Would this be a worthwhile addition? Paul

3 2

Better stdlib support for Path objects
by Barry Warsaw 07 Oct '14

07 Oct '14

Over in issue 22570, I lament the fact that while pathlib is awesome, its wider support in the stdlib is pretty sparse. I've tried to convert parts of a medium sized Python 3 application from os.path to pathlib and found this lack of support rather demotivating. Yes, it's fairly easy to wrap Path objects in str() to pass them to stdlib methods that expect only strings, but it's a lot of work in user code and I find that the resulting str()s are distracting. It's a disincentive. Antoine provided a link to a previous discussion[*] but that didn't go very far. One simple solution would be to sprinkle str() calls in various stdlib methods, but I'm not sure if that would fail miserably in the face of bytes paths (if the original stdlib API even accepts bytes paths). The suggestion in the issue is to add a "path protocol" and the referenced article suggests .strpath and .bytespath. OTOH, isn't str() and bytes() enough? I don't have any other brilliant ideas, but I opened the issue and am posting here to see if we can jump start another discussion for Python 3.5. I'd *like* to use more Paths, but not at the expense of my own code's readability. Yes, I'd sacrifice a bit of readability in the stdlib, especially if that would cover more use cases. Cheers, -Barry [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html

23 59

Idea to support lazy loaded names.
by Brian Allen Vanderburg II 07 Oct '14

07 Oct '14

When developing Python libraries (and any other language for that matter), items are separated into modules for organization and other purpose. The following is an example fake framework to illustrate this: mini-framework/ __init__.py app.py class BaseApplication class Application class ProxyApplication config.py class Config class ConfigLoader template.py class Template class TemplateCompiler class TemplateManager In order to be used externally, one must reference the module: from miniframework.app import Application from miniframework.template import TemplateManager This can be solved by importing these values directly in __init__.py. However this requires fully loading those imported modules: __init__.py: from .app import BaseApplication, Application, ProxyApplication from .config import Config, ConfigLoader from .template import Template, TemplateCompiler, TemplateManager One idea I had was to support lazy imported names. I've seen some frameworks implement this in various means, and figured the idea is good to be part Python. The idea is that for a new module attribute to exist: __lazyload__. During the access of any attribute for a module, the following would happen: * If the named attribute already exists, use it * If the named attribute does not already exist: * If a lazy load of the name has already been attempted, result in a NameError * If a lazy load of the name has not yet been attempted: * Check the __lazyload__ module attribute for the name, perform the loading operation and assign the module attribute the value if found or result in a NameError * Even if not found, set a flag that lazy load has been attempted so it will not be attempted again for the same name The __lazyload__ attribute is intended to be a dictionary. The key of the dictionary is the name of the attribute that would be set/tested for in the module. The value of the dictionary is a string that represents either the module name to load or the module name and attribute to load. If the value starts with a dot, then it is treated as a relative import relative to the module/package containing the __lazyload__ value. With this idea, the packages __init__.py file could look like this: __lazyload__ = { 'BaseApplication': '.app.BaseApplication', 'Application': '.app.Application', ... } The end use of the package (and even the developer) can then perform an import as follows: from miniframework import Application instead of: from miniframework.app import Application This allows the public api be be cleaner, while still being efficient by not loading all modules in __init__.py until the value is actually accessed. Brian Allen Vanderburg II

5 5

Add __parent__ to all classes, functions, and modules
by Neil Girdhar 07 Oct '14

07 Oct '14

Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations. It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors. I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in. One concrete use case for __parent__ is allowing a method decorator to call super, which is currently impossible because the class in which the method has been defined is not available to the decorator. This way, one could write: def ensure_finished(iterator): try: next(iterator) except StopIteration: return else: raise RuntimeError def derived_generator(method): def new_method(self, *args, **kwargs): x = method(self, *args, **kwargs) y = getattr(super(method.__parent__, self), method.__name__)\ (*args, **kwargs) for a, b in zip(x, y): assert a is None and b is None yield ensure_finished(x) ensure_finished(y) This is currently impossible to implement without restating the class name every time the decorator is used as far as I know. Best, Neil

7 16

Classes returned from functions
by Juancarlo Añez 06 Oct '14

06 Oct '14

Grako is a PEG parser builder in Python: https://bitbucket.org/apalala/grako In grako, there's code to synthesize AST node types according to annotations in the grammar whenever a type is not previously provided. This is the code: def _get_nodetype(self, typename): typename = str(typename) if typename in self.nodetypes: return self.nodetypes[typename] # synthethize a new type nodetype = type(typename, (self.baseType,), {}) self._register_nodetype(nodetype) return nodetype It works, but the classes thus created are "weird". I don't remember the details right now, but I think they don't have a proper module or qualname. This particular feature (exploiting this Python capability) is very useful in the early stages of developing a parser, while most of the work goes into debugging the grammar. Having a nice AST produced cheaply is of great value. Cheers, -- Juancarlo *Añez*

1 0

collections.abc.BytesLike or ...? [was: [Python-Dev] bytes-like objects]
by Jan Kaliszewski 06 Oct '14

06 Oct '14

2014-10-05 Victor Stinner <victor.stinner(a)gmail.com> dixit: > I prefer "bytes-like" than "buffer protocol". By the way, is there a > documentation in Python doc which explains "bytes-like" and maybe > list most compatible types? > > I'm not sure that the term has an unique definition. In some parts of > Python, I saw explicit checks on the type: bytes or bytearray, > sometimes memoryview is accepted. The behaviour is different in C > functions using PyArg API. It probably depends if the function relies > on the content (read bytes) or on methods (ex: call .find). Maybe an ABC would be useful for isinstance checks? Or some kind of almost-an-ABC-which-does-not-allow-to-register- -arbitrary-virtual-subclasses? Cheers. *j

2 1

O(1) containment testing with Py3 range objects (was Re: Make len() usable on a generator)
by Nick Coghlan 05 Oct '14

05 Oct '14

On 4 October 2014 22:39, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Oct 4, 2014, at 13:48, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> On 4 October 2014 02:45, Ian Cordasco <graffatcolmingov(a)gmail.com> wrote: >>> On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky >>> <alexander.belopolsky(a)gmail.com> wrote: >>>> You are correct about python-ideas, but range is not a generator in python 3 >>>> either: >>>> >>>> Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>>>>> type(range(10)) # not a generator >>>> <class 'range'> >>>>>>> type(x for x in range(10)) # a generator >>>> <class 'generator'> >>> >>> Ah yeah. All too often I conflate lazy iterators with generators. Nevermind >> >> Python 3 range isn't a lazy iterator either - it's a full sequence >> type (specifically, a calculated tuple: >> https://docs.python.org/3/library/stdtypes.html#ranges). The only >> immutable sequence operations it doesn't support are concatenation and >> repetition (since concatenating or repeating a range can't be >> represented using the simple "start + n*step" calculation that range >> uses internally). >> >> We'd have gotten a *lot* more complaints if we just dropped xrange in >> as a substitute for the list objects returned by Python 2's range >> builtin :) > > For some reason, most of the 3.x early adopters understood this, but later switchers seem to often believe that this is exactly what Python 3 did. I've seen a few transition guides that oversimplify the situation as "Python 3 range is the same as Python 2 xrange". Those transition guides are wrong (xrange has many limitations compared to the more full featured Python 3 range implementation), but we don't have the capacity to chase down every occurrence and ask people to fix it. One nice trick with Py3 range for example, is to work with ranges that Py2 could never hope to handle: $ python3 -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" 1000000 loops, best of 3: 0.295 usec per loop $ python -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" Traceback (most recent call last): File "/usr/lib64/python2.7/timeit.py", line 300, in main x = t.timeit(number) File "/usr/lib64/python2.7/timeit.py", line 195, in timeit timing = self.inner(it, self.timer) File "<timeit-src>", line 3, in inner seq = range(-(10**24), 10**24) OverflowError: range() result has too many items There's still an unfortunate technical limitation in Py3 where *len* is restricted to 64 bit integers, but start/stop/step are exposed on range objects these days, so you can do the calculation directly if you really need to. (That workaround severely reduces the motivation for anyone to put together a backwards compatible proposal for an alternative C level length API that uses a Python object rather than a C sssize_t value) > For example, at least half a dozen times, I've written an answer on StackOverflow showing someone that `n in range(start, stop)` is what they were trying to write, and explaining that it means the same thing as `start <= n < stop` (with various caveats about Python 2, and `step`, etc.) only for someone to edit or comment on my answer claiming that range.__contents__ has to be linear and should never be used. I explain that it's a sequence, and can be indexed, and show them a quick timing test, and they still insist that's just an accidental optimization that shouldn't be relied on, because it's really just meant to be Python 2's xrange. Once someone insisted that if you want this functionality you should use a multi-interval class off PyPI, even though the docs for that class explicitly say that a single interval is just an empty subclass of range in Py 3. People who stick with 2.x just really want to believe you got this wrong, I guess. Containment tests on range() are guaranteed to be O(1) when suing CPython to test against true int and bool objects. Other implementations may not offer the same guarantee (although it's a simple enough optimisation that I believe they should). CPython itself will currently fall back to O(n) behaviour for int subclasses and other objects to handle backwards compatibility issues in relation to custom __eq__ methods (see http://bugs.python.org/issue1766304 for history and details - note that this *was* under consideration for 2.7 xrange as well, and merely didn't make the release deadline). For those that haven't seen the kind of timing examples Andrew mentioned, one easy way to see the difference is by comparing integer lookup performance with a numeric type that isn't an integer at all: $ python3 -m timeit -s "seq = range(10**6)" "10e5 in seq" 10 loops, best of 3: 118 msec per loop $ python3 -m timeit -s "seq = range(10**6)" "int(10e5) in seq" 1000000 loops, best of 3: 0.348 usec per loop You cop a small constant-time hit for the float->int conversion, but gain back 6 orders of magnitude (in this particular example) through the switch to an O(1) containment check. This optimisation is particularly useful in cases that can't easily be converted to comparison operations, such as when the range has a step magnitude other than 1. I thought I had an open issue proposing to expand the O(1) optimisation to any type that implements index (as well as making it a documented guarantee all implementations are expected to provide), but it looks like I may have only thought about filing that rather than actually doing it. (Or perhaps it's hiding in the python-ideas archives somewhere) > I've seen similar things with, e.g., people not believing that dict.keys() returns a set that's a view into the dict's storage rather than some kind of magical iterator that knows how to be used multiple times. (I'm not even sure how that would work; how could __iter__ know when you were trying to get self and when you were trying to start over?) That at least has an authoritative reference in PEP 3106 (along with the standard library docs). Unfortunately, the more convinced someone is that they already understand a change, the less likely they are to read the references to the relevant design discussions and decisions :P Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0