Mailman 3 October 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

including psutil in the standard library?
by Stefano Borini Nov. 11, 2014

Nov. 11, 2014

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed. I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way. Any thoughts?

12 15

Deprecate dunder functions from operator module
by Steven D'Aprano Nov. 4, 2014

Nov. 4, 2014

(For the avoidance of all doubt, the following has nothing to do with dunder methods like int.__add__ etc., but only the functions in the operator module.) The operator module contains various function versions of operators, e.g. operator.add for + , operator.sub for - , etc. There are also dunder versions of each function, and according to the documentation: The function names are those used for special class methods; variants without leading and trailing __ are also provided … [View More]

4 7

"or raise" syntax
by Javier Dehesa Nov. 3, 2014

Nov. 3, 2014

This happens to me with some frequency: result = f(args) if not result: # of "if result is None:" raise Exception(...) What if I could just say? result = f(args) or raise Exception(...) I think that's pretty idiomatic. What's more, this would give you a shorter syntax for things like: if x <= y: raise Exception(...) Which you could write simply as: x > y or raise Exception(...) Effectively covering the use case of this proposal http://article.… [View More]

4 3

Re: [Python-ideas] Improving memory usage in shared-key attribute dicts
by Hill, Bruce Oct. 31, 2014

Oct. 31, 2014

On Thu Oct 30 23:42:21 CET 2014, Steven D'Aprano wrote: > How does this compare to the "Alternate Implementation" described in PEP 412? > > http://python.org/dev/peps/pep-0412/#id17 I feel pretty dumb--I didn't read the alternative implementation section. It would be identical. Thanks for catching that. I'm going to try implementing it to see how it pans out.

1 0

Improving memory usage in shared-key attribute dicts
by Hill, Bruce Oct. 30, 2014

Oct. 30, 2014

Thanks to PEP 412: "Key-Sharing Dictionary", CPython attribute dictionaries can share keys between multiple instances, so the memory cost of new attribute dicts comes primarily from the values array. In the current implementation, the keys array and the values array are always kept to be the same size. This is done so that once the key's location in the key array has been tracked down, the same array offset can be used on the value array to find the value. Rather than storing values in a … [View More]

2 1

operator.call / operator.__call__
by Antony Lee Oct. 30, 2014

Oct. 30, 2014

A simple suggestion: add "operator.call" and "operator.__call__", which would provide a function like Python2's "apply" (but it'd be called with already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). Why? To be able to pass it as an argument to other function, just like the other functions defined in the "operator" module. Antony

6 7

Re: [Python-ideas] Adding iOS/Android support to Python
by Todd Oct. 28, 2014

Oct. 28, 2014

On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" <russell(a)keith-magee.com> wrote: > > > On Sat, Oct 25, 2014 at 3:36 PM, Todd <toddrjen(a)gmail.com> wrote: >> >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" <russell(a)keith-magee.com> wrote: >> > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes … [View More]bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. >> >> I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. > > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. Perhaps no console by default, but it is possible to have a traditional console on android. I have one and many ROMs install one by default. So although it may not be part of the default configuration I think it would be good to support it for the people who do have a console. Further, with rooted users, python could be set up to be used with the built-in adb shell. It is unclear from the discussion where things ultimately came out on this issue. If there still a possibility it might removed, although I understand that consoles are not the primary use-case, I think is still a valid use-case that should supported. >> I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. > > Firstly - I don't know what gave you the impression Apple devices aren't multicore - every Apple processor since the A5 (introduced in the iPhone 4S and iPad 2) has been at least dual core, and the A8X in the newest iPads is triple core. I was referring to the benchmarks where corresponding iOs and android devices generally have better single and multi-core performance, respectively, but you right that isn't that important. > Secondly, if you're assuming "multicore" automatically means "mathematical powerhouse", you're mistaken. If you're planning on doing serious mathematical heavy lifting on a phone... well, you've already made your first mistake :-) No, on the contrary, I was thinking that on devices with limited performance, being able to divide components between processes, such as UI and logic, is all the more important. It probably would not be any use for the sort of calculator I am thinking about. [View Less]

4 4

Adding iOS/Android support to Python
by Russell Keith-Magee Oct. 28, 2014

Oct. 28, 2014

Hi all, tl;dr - I'd like to add iOS and Android to the list of platforms that Python supports out-of-the-box, and I'm looking for guidance on the way forward (and assistance, if anyone is keen). Longer version: For the last 9 months or so, I've been working on tools to enable developers to write native mobile apps using Python. My motivation is to get Toga (http://pybee.org/toga) to the point where it's a viable way to write mobile apps. However, I'm not alone in the "Python on mobile" goal - … [View More]Kivy has been around for a few years, with similar goals (although they've got some different ideas and priorities). To date, I've been using the groundwork established by Kivy, which takes the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, applies a bunch of patches, and wraps those in a set of build scripts to yield Python libraries that can be used on iOS and Android. I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository. In particular, there are four areas where I can see changes required: 1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform) 2) Modifications to the build system to support cross-platform builds for mobile - AFAICT, cross platform builds currently only work on Linux and require readelf; iOS libraries can only be built on Mac, and building Android libraries don't require readelf (AFAICT). 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. 4) Support for building fat libraries for iOS. Xcode uses a tool called lipo to mash multiple platform libraries into a single library file. I don't imagine 1 & 2 are especially controversial. However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of? Any platforms where Python is *only* available as a library? I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory. Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS. Is there any documentation of the build process itself? Or is there anyone who can point me in the right direction? I'm also not clear how to tackle the testing and CI problems. Since you can't run iOS code on a non-iOS machine, it isn't entirely clear to me what an acceptance test would look like for a mobile build. So - am I completely wasting my time? Are patches for mobile platforms likely to be accepted into Python's repository? Or should I stick with the "external patches" approach? Any pointers, hints, guidance or assistance would be gratefully accepted. Yours, Russ Magee %-) [View Less]

12 22

Fwd: Adding iOS/Android support to Python
by Russell Keith-Magee Oct. 27, 2014

Oct. 27, 2014

On Sat, Oct 25, 2014 at 3:36 PM, Todd <toddrjen(a)gmail.com> wrote: > On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" <russell(a)keith-magee.com> > wrote: > > 3) Disabling certain modules on mobile platforms. Supporting modules > like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on > mobile platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like … [View More]multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > > I would definitely be extremely interested in a python shell in android. > One thing I feel are lacking on android are good advanced mathematical > tools and and python shell with appropriate libraries could make a very > powerful open-source tool for that. There have been some attempts at that > already. > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. > I would also differentiate android and iOs more. Android seems to be > betting on multi-core performance while iOs seems to be betting on > single-chore performance. So while multiprocessing may not make much sense > on iOs, I think it may be more sense on Android, especially if they move > from 4 to 8 cores. > Firstly - I don't know what gave you the impression Apple devices aren't multicore - every Apple processor since the A5 (introduced in the iPhone 4S and iPad 2) has been at least dual core, and the A8X in the newest iPads is triple core. Secondly, if you're assuming "multicore" automatically means "mathematical powerhouse", you're mistaken. If you're planning on doing serious mathematical heavy lifting on a phone... well, you've already made your first mistake :-) > Similarly, ultimately android is Linux-based, so things like readline > should work, and it seems others have been able to get it to work, and bz2 > seems to work fine on third-party android file managers. > iOS is Posix under the hood as well. My point was that readline and curses make a whole lot of assumptions about the type of terminal you're using, and if you're writing a halfway decent UI for a mobile platform, you'd probably be better served throwing those tools overboard in preference for something that takes native UI controls into account. As for bz2 - I can't think of any reason it wouldn't work on Android (or iOS either, for that matter); I was more flagging the fact that it has binary dependencies which need to be resolved, and external dependencies complicate the cross-compilation process. More broadly, as a result of both these points, I was trying to gauge the extent to which "complete availability of all Python modules" would be considered a requirement of any official effort to port Python to Android and iOS. Yours, Russ Magee %-) [View Less]

2 2

Fwd: Adding iOS/Android support to Python
by Russell Keith-Magee Oct. 25, 2014

Oct. 25, 2014

On Sat, Oct 25, 2014 at 9:27 PM, Hernan Grecco <hgrecco(a)gmail.com> wrote: > > El oct 24, 2014 11:22 PM, "Russell Keith-Magee" <russell(a)keith-magee.com> > escribió: > > > > Hi all, > > > > tl;dr - I'd like to add iOS and Android to the list of platforms that > Python supports out-of-the-box, and I'm looking for guidance on the way > forward (and assistance, if anyone is keen). > > Have you looked a QPython? They provide Python 2 and 3 … [View More]

1 0

namespace for backported stdlib modules?
by Robert Collins Oct. 21, 2014

Oct. 21, 2014

I want to make the improvements in unittest (which transitively includes mock and traceback and perhaps inspect ...) available for folk that can't use 3.5 yet. We've currently got the unittest2 and mock as modules on pypi, and clearly we can add another e.g. traceback2 as a way to get the traceback changes out there. I'm wondering though if perhaps putting up a namespace package on pypi for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) would make sense. In particular that … [View More]

9 12

Need feedback about exposing an operator overloading in a module
by Julien Palard Oct. 21, 2014

Oct. 21, 2014

Hi everybody, Long long time ago, I wrote thehttps://pypi.python.org/pypi/pipe Python module, TL;DR: a 6 lines of code decorator enabling infix syntax. Long time ago someone came to speak to me on IRC advising me to speak about it here, but I was still blocked in my mind about the exposed/leaked `|` operator overload. For me, a module should never expose an operator overload, because it may lead to unpredicted/unnatural/surprising behavior. But I may be wrong. Recently I was thinking about … [View More]

4 3

Query about proposed Multi-line Anonymous Function syntaxes
by Adam Jorgensen Oct. 14, 2014

Oct. 14, 2014

Hey all, my first time posting here. So, as I understand it, the main reason we don't have multi-line anonymous functions is because there has been no consensus about the syntax. I was just wondering if someone has a document detailing the various syntaxes that have been proposed in the the past. Adam

8 9

Make len() usable on a generator
by Thomas Chaumeny Oct. 12, 2014

Oct. 12, 2014

Hi! I have just come across some code counting a generator comprehension expression by doing len([foo for foo in bar if some_condition]) and I realized it might be better if we could just use len(foo for foo in bar if some_condition) as it would avoid a list allocation in memory. Another possibility is to write sum(1 for foo in bar if some_condition), but that is not optimal either as it generates a lot of intermediate additions which should not be needed. Sure, len(generator) might lead to … [View More]

20 38

Negative limit for traceback.*
by Thomas Allen Oct. 11, 2014

Oct. 11, 2014

I'd like to propose adding the ability to extract last n stack trace entries from a traceback using functions from traceback module. Simply put, passing limit=-n would make functions to return (or print or yield) last abs(n) entries. No-brainer implementation for extract_tb: extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0])) if limit is not None and limit < 0: return extracted[limit:] return extracted The motivation: http://stackoverflow.com/q/25472412/2301450

3 3

reduce(func, seq, initial=0)
by Stephan Sahm Oct. 10, 2014

Oct. 10, 2014

Hi folk, I stumbled upon the builtin reduce function. It has an optional parameter initial, however if trying to set it like import operator as op reduce(op.add, range(10), initial=0) I get TypeError: reduce() takes no keyword arguments I think it would be really straightforward to also add keyword possibilities, at least to the parameter initial. What do you think? best, Stephan

6 8

Extend unicodedata with a name search
by Philipp A. Oct. 10, 2014

Oct. 10, 2014

I noticed that the excellent perl utility unum <http://www.fourmilab.ch/webtools/unum/> uses an obsolete unicode database. Since I’m a Pythonista, i recalled hearing about the stdlib unicodedata module, using which I either wanted to rewrite unum or extend its database. Unfortunately, unicodedata is very limited. Partly rightfully so, since you can convert codepoints and chars with chr() and ord(), and str.upper() and friends are unicode-aware. But the name database is only queryable … [View More]

8 15

metamodules (was: Re: Idea to support lazy loaded names.)
by Nathaniel Smith Oct. 10, 2014

Oct. 10, 2014

On Tue, Oct 7, 2014 at 5:12 AM, Andrew Barnert <abarnert(a)yahoo.com.dmarc.invalid> wrote: > On Oct 6, 2014, at 20:51, Terry Reedy <tjreedy(a)udel.edu> wrote: > >> So a 'solution' might be to make modules be instances (but with no __new__ or __init__) of a module metaclass, so that module dicts could act like class dicts with respect to descriptors. I have no idea how much code this would break ;-). > > Didn't we just have this discussion a few weeks ago, in the … [View More]context of making lazy loading of subpackages easier to implement? Yeah, and having had some time to think about that discussion and do some prototyping, I'm going to argue below that allowing assignment to module instances' __class__ really is the best path forward. (For those who find the below TLDR, check this out instead: https://github.com/njsmith/metamodule) > IIRC, the not-obviously-unreasonable options suggested were: Great list! I've rearranged a bit to make my argument clearer. > 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. This is very similar to the current approach of having __init__.py reassign sys.modules[__name__]. The advantages are: - It gives a way to enforce the rule that you have to do this assignment as the very first thing inside your module, before allowing arbitrary code to run (e.g. by importing other modules which might recursively import your module in turn, and access sys.modules[__name__] before you've modified it). - Because __new__.py would run *before* the It avoids the headache of having to juggle two module objects, one of whose __dict__'s is already being used as the execution environment for the code that is trying to do the switcheroo. But: - It's a pretty complicated way to accomplish the stated goals. - The restriction to packages is unfortunate. - The backcompat story is terrible -- faking __new__.py support in old versions of python would be really difficult, and the main reason I care about this stuff in the first place is because I want to be able to e.g. deprecate module attributes that are in the public API of old, widely-used packages. It will be many years before such packages can require 3.5. So I think we should probably try to make the existing sys.modules[__name__] = ... strategy work before considering this. > 2) Analogy with metaclass= (or with 2.x __metaclass__): If a module (or a package's __init__.py) does some new syntax or magic comment before any non-comment code, it can specify a custom type in place of ModuleType (not sure how that type gets imported and made available). I don't see any way to solve this import problem you refer to at the end -- in most cases the code implementing the metamodule type will be defined *inside* the module/package which wants to use the metamodule, so we have a chicken-and-egg problem. > 4) Make it easier to write import hooks for this special purpose. This has the same problem as the previous -- who imports the importer? > 5) Make it easier for a module to construct a ModuleType-subclass instance with a copy of the same dict and replace itself in sys.modules. So, trying to *copy* the dict is just not going to work. Consider the package foo, with a foo/__init__.py that looks like: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 from .submod import b c = 3 sys.modules[__name__].__dict__.update(orig_dict) and where foo/submod.py looks like: import foo b = foo.a + 1 def c(): return foo.a + 2 This won't work, because at the time we import .submod, sys.modules["foo"].__dict__ does not contain an entry for "a" -- only the original module's dict has that. There are a bunch of different ways we could try writing our __init__.py. We might try putting the sys.module assignment at the end: a = 1 from .submod import b, c d = 4 orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__.update(orig_dict) Now when .submod re-imports the top-level module, it ends up with a reference to the original module object, which has an "a" entry, so the definition of "b" works. But, now .submod.foo will continue to refer to the original module object, even after we substitute in the metamodule object. If we do 'foo.a = 5' later on, then foo.c() will continue to use the original binding of 'a'; this mutation will be invisible to it. I guess the only way to make it work in this case is to do multiple copies, one before every nested-import: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 sys.modules[__name__].__dict__.update(orig_dict) from .submod import b, c d = 4 sys.modules[__name__].__dict__.update(orig_dict) ...but this is incredibly ugly and error-prone. What we really want to do instead is to make our new metamodule object refer directly to the original module's __dict__: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__ = orig_dict a = 1 from .submod import b, c d = 4 That way they will always be in sync. This looks like it should work great! But it has a few problems: - Trying to assign to a module's __dict__ attribute raises "TypeError: readonly attribute". - So I actually implemented a fix for that, and ran into a new problem: module's take jealous ownership of their __dict__. In particular, they assume that they are when they are deallocated they should wipe their dict clean (https://www.python.org/doc/essays/cleanup/). Obviously this is bad for us because we are still using that dict! - Also, in modern Python module objects contain more state besides __dict__ -- in particular, PEP 3121-related state. There's no public API to get at this. - Possibly future version of Python will add more state fields again, who knows. The easiest way to solve all these problems is to *swap* all of the internal fields between the old module object and the new metamodule object. This can be done hackishly using ctypes; this requires knowing about CPython's struct layouts, but that's okay for prototyping and for backwards compatibility hacks (which only have to support specific known versions). To do it non-hackishly, I was at first thinking that we should provide an official API for swapping module object states. But then I realized that at that point, we're basically back to... > 3) Analogy with re-classing object instances: Just allow modules to set __class__ during execution (or after, if you want). Variations on this include allowing that for all non-heap types, or even getting rid of the concept of non-heap types. ...this proposal after all. And allowing __class__ assignment on modules strikes me as more asthetic than having a sys.swap_module_contents function. I implemented a prototype of this functionality here: https://github.com/njsmith/metamodule The implementation is here: https://github.com/njsmith/metamodule/blob/master/metamodule.py That file has 3 parts: - A fancy metamodule class that handles implicit-imports and warn-on-attribute-access. - A utility function for setting up a metamodule; it tries __class__ assignment, and if that doesn't work falls back on ctypes hackery. - The aforementioned ctypes hackery. It's pretty ugly, but if we add __class__ assignment then it will become unnecessary on future Python versions, woohoo! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org [View Less]

4 6

issubclass(collections.OrderedDict, collections.Sequence)
by Ram Rachum Oct. 9, 2014

Oct. 9, 2014

Hi guys, I was just going to test whether a mapping is dict-like or an OrderedDict-like object by doing this: isinstance(x, collections.Sequence) But then I checked and saw that this is False: issubclass(collections.OrderedDict, collections.Sequence) is False Why isn't an OrderedDict a Sequence?

13 43

Better integration of multiprocessing with asyncio
by Dan O'Reilly Oct. 9, 2014

Oct. 9, 2014

I think it would be helpful for folks using the asyncio module to be able to make non-blocking calls to objects in the multiprocessing module more easily. While some use-cases for using multiprocessing can be replaced with ProcessPoolExecutor/run_in_executor, there are others that cannot; more advanced usages of multiprocessing.Pool aren't supported by ProcessPoolExecutor (initializer/initargs, contexts, etc.), and other multiprocessing classes like Lock and Queue have blocking methods that … [View More]could be made into coroutines. Consider this (extremely contrived, but use your imagination) example of a asyncio-friendly Queue: import asyncio import time def do_proc_work(q, val, val2): time.sleep(3) # Imagine this is some expensive CPU work. ok = val + val2 print("Passing {} to parent".format(ok)) q.put(ok) # The Queue can be used with the normal blocking API, too. item = q.get() print("got {} back from parent".format(item)) def do_some_async_io_task(): # Imagine there's some kind of asynchronous I/O # going on here that utilizes asyncio. asyncio.sleep(5) @asyncio.coroutine def do_work(q): loop.run_in_executor(ProcessPoolExecutor(), do_proc_work, q, 1, 2) do_some_async_io_task() item = yield from q.coro_get() # Non-blocking get that won't affect our io_task print("Got {} from worker".format(item)) item = item + 25 yield from q.coro_put(item) if __name__ == "__main__": q = AsyncProcessQueue() # This is our new asyncio-friendly version of multiprocessing.Queue loop = asyncio.get_event_loop() loop.run_until_complete(do_work(q)) I have seen some rumblings about a desire to do this kind of integration on the bug tracker (http://bugs.python.org/issue10037#msg162497 and http://bugs.python.org/issue9248#msg221963) though that discussion is specifically tied to merging the enhancements from the Billiard library into multiprocessing.Pool. Are there still plans to do that? If so, should asyncio integration with multiprocessing be rolled into those plans, or does it make sense to pursue it separately? Even more generally, do people think this kind of integration is a good idea to begin with? I know using asyncio is primarily about *avoiding* the headaches of concurrent threads/processes, but there are always going to be cases where CPU-intensive work is going to be required in a primarily I/O-bound application. The easier it is to for developers to handle those use-cases, the better, IMO. Note that the same sort of integration could be done with the threading module, though I think there's a fairly limited use-case for that; most times you'd want to use threads over processes, you could probably just use non-blocking I/O instead. Thanks, Dan [View Less]

4 10

Enhanced stat_result objects (was: Better stdlib support for Path objects)
by Paul Moore Oct. 8, 2014

Oct. 8, 2014

On 6 October 2014 18:47, Barry Warsaw <barry(a)python.org> wrote: > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. The other thing that bugs me whenever I try to use pathlib is the non-caching behaviour of the stat-related methods. I know why this is as it is, and I agree with the reasons, but for some reason, I still find that I expect code like p = Path(...) if not p.exists(): print("… [View More]

3 2

Better stdlib support for Path objects
by Barry Warsaw Oct. 7, 2014

Oct. 7, 2014

Over in issue 22570, I lament the fact that while pathlib is awesome, its wider support in the stdlib is pretty sparse. I've tried to convert parts of a medium sized Python 3 application from os.path to pathlib and found this lack of support rather demotivating. Yes, it's fairly easy to wrap Path objects in str() to pass them to stdlib methods that expect only strings, but it's a lot of work in user code and I find that the resulting str()s are distracting. It's a disincentive. Antoine … [View More]

23 59