Mailman 3 October 2014 - Python-ideas

Idea to support lazy loaded names.
by Brian Allen Vanderburg II Oct. 7, 2014

Oct. 7, 2014

When developing Python libraries (and any other language for that matter), items are separated into modules for organization and other purpose. The following is an example fake framework to illustrate this: mini-framework/ __init__.py app.py class BaseApplication class Application class ProxyApplication config.py class Config class ConfigLoader template.py class Template class TemplateCompiler class TemplateManager In order to … [View More]be used externally, one must reference the module: from miniframework.app import Application from miniframework.template import TemplateManager This can be solved by importing these values directly in __init__.py. However this requires fully loading those imported modules: __init__.py: from .app import BaseApplication, Application, ProxyApplication from .config import Config, ConfigLoader from .template import Template, TemplateCompiler, TemplateManager One idea I had was to support lazy imported names. I've seen some frameworks implement this in various means, and figured the idea is good to be part Python. The idea is that for a new module attribute to exist: __lazyload__. During the access of any attribute for a module, the following would happen: * If the named attribute already exists, use it * If the named attribute does not already exist: * If a lazy load of the name has already been attempted, result in a NameError * If a lazy load of the name has not yet been attempted: * Check the __lazyload__ module attribute for the name, perform the loading operation and assign the module attribute the value if found or result in a NameError * Even if not found, set a flag that lazy load has been attempted so it will not be attempted again for the same name The __lazyload__ attribute is intended to be a dictionary. The key of the dictionary is the name of the attribute that would be set/tested for in the module. The value of the dictionary is a string that represents either the module name to load or the module name and attribute to load. If the value starts with a dot, then it is treated as a relative import relative to the module/package containing the __lazyload__ value. With this idea, the packages __init__.py file could look like this: __lazyload__ = { 'BaseApplication': '.app.BaseApplication', 'Application': '.app.Application', ... } The end use of the package (and even the developer) can then perform an import as follows: from miniframework import Application instead of: from miniframework.app import Application This allows the public api be be cleaner, while still being efficient by not loading all modules in __init__.py until the value is actually accessed. Brian Allen Vanderburg II [View Less]

5 5

Add __parent__ to all classes, functions, and modules
by Neil Girdhar Oct. 7, 2014

Oct. 7, 2014

Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations. It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors. I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in. One concrete use case for __parent__ is allowing a method decorator to call super, which is currently impossible because the … [View More]

7 16

Classes returned from functions
by Juancarlo Añez Oct. 6, 2014

Oct. 6, 2014

Grako is a PEG parser builder in Python: https://bitbucket.org/apalala/grako In grako, there's code to synthesize AST node types according to annotations in the grammar whenever a type is not previously provided. This is the code: def _get_nodetype(self, typename): typename = str(typename) if typename in self.nodetypes: return self.nodetypes[typename] # synthethize a new type nodetype = type(typename, (self.baseType,), {}) self.… [View More]

1 0

collections.abc.BytesLike or ...? [was: [Python-Dev] bytes-like objects]
by Jan Kaliszewski Oct. 6, 2014

Oct. 6, 2014

2014-10-05 Victor Stinner <victor.stinner(a)gmail.com> dixit: > I prefer "bytes-like" than "buffer protocol". By the way, is there a > documentation in Python doc which explains "bytes-like" and maybe > list most compatible types? > > I'm not sure that the term has an unique definition. In some parts of > Python, I saw explicit checks on the type: bytes or bytearray, > sometimes memoryview is accepted. The behaviour is different in C > functions using PyArg API. … [View More]

2 1

O(1) containment testing with Py3 range objects (was Re: Make len() usable on a generator)
by Nick Coghlan Oct. 5, 2014

Oct. 5, 2014

On 4 October 2014 22:39, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Oct 4, 2014, at 13:48, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> On 4 October 2014 02:45, Ian Cordasco <graffatcolmingov(a)gmail.com> wrote: >>> On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky >>> <alexander.belopolsky(a)gmail.com> wrote: >>>> You are correct about python-ideas, but range is not a generator in python 3 >>>> either: … [View More]>>>> >>>> Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>>>>> type(range(10)) # not a generator >>>> <class 'range'> >>>>>>> type(x for x in range(10)) # a generator >>>> <class 'generator'> >>> >>> Ah yeah. All too often I conflate lazy iterators with generators. Nevermind >> >> Python 3 range isn't a lazy iterator either - it's a full sequence >> type (specifically, a calculated tuple: >> https://docs.python.org/3/library/stdtypes.html#ranges). The only >> immutable sequence operations it doesn't support are concatenation and >> repetition (since concatenating or repeating a range can't be >> represented using the simple "start + n*step" calculation that range >> uses internally). >> >> We'd have gotten a *lot* more complaints if we just dropped xrange in >> as a substitute for the list objects returned by Python 2's range >> builtin :) > > For some reason, most of the 3.x early adopters understood this, but later switchers seem to often believe that this is exactly what Python 3 did. I've seen a few transition guides that oversimplify the situation as "Python 3 range is the same as Python 2 xrange". Those transition guides are wrong (xrange has many limitations compared to the more full featured Python 3 range implementation), but we don't have the capacity to chase down every occurrence and ask people to fix it. One nice trick with Py3 range for example, is to work with ranges that Py2 could never hope to handle: $ python3 -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" 1000000 loops, best of 3: 0.295 usec per loop $ python -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" Traceback (most recent call last): File "/usr/lib64/python2.7/timeit.py", line 300, in main x = t.timeit(number) File "/usr/lib64/python2.7/timeit.py", line 195, in timeit timing = self.inner(it, self.timer) File "<timeit-src>", line 3, in inner seq = range(-(10**24), 10**24) OverflowError: range() result has too many items There's still an unfortunate technical limitation in Py3 where *len* is restricted to 64 bit integers, but start/stop/step are exposed on range objects these days, so you can do the calculation directly if you really need to. (That workaround severely reduces the motivation for anyone to put together a backwards compatible proposal for an alternative C level length API that uses a Python object rather than a C sssize_t value) > For example, at least half a dozen times, I've written an answer on StackOverflow showing someone that `n in range(start, stop)` is what they were trying to write, and explaining that it means the same thing as `start <= n < stop` (with various caveats about Python 2, and `step`, etc.) only for someone to edit or comment on my answer claiming that range.__contents__ has to be linear and should never be used. I explain that it's a sequence, and can be indexed, and show them a quick timing test, and they still insist that's just an accidental optimization that shouldn't be relied on, because it's really just meant to be Python 2's xrange. Once someone insisted that if you want this functionality you should use a multi-interval class off PyPI, even though the docs for that class explicitly say that a single interval is just an empty subclass of range in Py 3. People who stick with 2.x just really want to believe you got this wrong, I guess. Containment tests on range() are guaranteed to be O(1) when suing CPython to test against true int and bool objects. Other implementations may not offer the same guarantee (although it's a simple enough optimisation that I believe they should). CPython itself will currently fall back to O(n) behaviour for int subclasses and other objects to handle backwards compatibility issues in relation to custom __eq__ methods (see http://bugs.python.org/issue1766304 for history and details - note that this *was* under consideration for 2.7 xrange as well, and merely didn't make the release deadline). For those that haven't seen the kind of timing examples Andrew mentioned, one easy way to see the difference is by comparing integer lookup performance with a numeric type that isn't an integer at all: $ python3 -m timeit -s "seq = range(10**6)" "10e5 in seq" 10 loops, best of 3: 118 msec per loop $ python3 -m timeit -s "seq = range(10**6)" "int(10e5) in seq" 1000000 loops, best of 3: 0.348 usec per loop You cop a small constant-time hit for the float->int conversion, but gain back 6 orders of magnitude (in this particular example) through the switch to an O(1) containment check. This optimisation is particularly useful in cases that can't easily be converted to comparison operations, such as when the range has a step magnitude other than 1. I thought I had an open issue proposing to expand the O(1) optimisation to any type that implements index (as well as making it a documented guarantee all implementations are expected to provide), but it looks like I may have only thought about filing that rather than actually doing it. (Or perhaps it's hiding in the python-ideas archives somewhere) > I've seen similar things with, e.g., people not believing that dict.keys() returns a set that's a view into the dict's storage rather than some kind of magical iterator that knows how to be used multiple times. (I'm not even sure how that would work; how could __iter__ know when you were trying to get self and when you were trying to start over?) That at least has an authoritative reference in PEP 3106 (along with the standard library docs). Unfortunately, the more convinced someone is that they already understand a change, the less likely they are to read the references to the relevant design discussions and decisions :P Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

1 0

Delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__`
by Ram Rachum Oct. 4, 2014

Oct. 4, 2014

Hi, I suggest delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__`, so instead of doing `bound_arguments.arguments['foo']` we could do `bound_arguments['foo']`. Since there isn't that much more to a `BoundArguments` object than its arguments I think it's nice to save some typing and be able to access the arguments directly. Thanks, Ram.

3 3

possible extension to how the new StopIteration is handled (python >=3.3)
by Stephan Sahm Oct. 3, 2014

Oct. 3, 2014

Hi all, I just tried the new possibilities in writing generators which were included since python3.3 into the python standard. Concretely, using return 1 in a generator is equal to raise StopIteration(1) Thus the following works: >>> def f(): ... yield 0 ... return 1 >>> g = f() >>> next(g) 0 >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 1 however calling next(g) once again … [View More]

3 5

Python 2's re module should take longs
by Ryan Gonzalez Oct. 1, 2014

Oct. 1, 2014

This works: re.search('(abc)', 'abc').group(1) but this doesn't: re.search('(abc)', 'abc').group(1L) The latter raises "IndexError: no such group". Shouldn't that technically work? -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/

9 11